A three-dimensional model construction system and method, device, medium

Abstract

The application relates to the medical field and discloses a three-dimensional model construction system and method, equipment and a medium.The three-dimensional model construction system comprises an image acquisition module, which is used for acquiring color images of a target object at different angles and fluorescence images of a target region in the target object; a control module, which is used for controlling the image acquisition module to normally complete the acquisition of the color images of the target object at different angles and the fluorescence images of the target region in the target object; and an image processing module, which is used for obtaining a three-dimensional model corresponding to the target object based on the color images of the target object at different angles and the fluorescence images of the target region in the target object.The three-dimensional model construction scheme can obtain a three-dimensional image of a target region of a patient when the image of the target region of the patient is acquired, the three-dimensional image can provide more detailed structure information of the target region of the patient, and thus can better assist doctors in obtaining more accurate disease diagnosis results.

Description

TECHNICAL FIELD

[0001] The present application relates to the medical field, and in particular to a three-dimensional model construction system and method, device and medium. BACKGROUND

[0002] In the modern medical field, the demand for precision medicine is growing, especially in the process of disease diagnosis and treatment. Therefore, in order to enable doctors to more intuitively observe the body structure information of patients, so that doctors can obtain more accurate diagnosis results, various imaging systems have been gradually widely used in the medical field to provide image information of the body structure of patients for doctors to diagnose diseases. In the imaging system, fluorescence imaging and color imaging technology are two important imaging means. The imaging system based on fluorescence imaging technology can obtain the image of the specified region in the target object with high sensitivity and high specificity through specific fluorescent markers; the imaging system based on color imaging technology can provide rich structural information of the target object.

[0003] In the research of imaging systems, the existing imaging systems are constructed based on fluorescence imaging technology to obtain the fluorescence image corresponding to the specified region in the target object, or based on color imaging technology to obtain the color image corresponding to the target object, or based on fluorescence imaging and color imaging technology to obtain the fluorescence image and color image of the target object respectively, and then fuse the two to obtain a two-dimensional image of the target object containing the fluorescence region. However, whether the imaging system is constructed based on single modality (fluorescence imaging technology or color imaging technology) or multi-modality (fluorescence imaging technology and color imaging technology), the final provided is a two-dimensional image of the target object, which cannot provide detailed three-dimensional structural information of the target object. Therefore, when the existing imaging system is used to obtain the image of the target region of the patient, only a two-dimensional image of the target region of the patient can be obtained, which cannot provide more detailed structural information of the target region of the patient, thereby making it difficult to assist doctors to obtain more accurate disease diagnosis results. SUMMARY

[0004] The present application provides a three-dimensional model construction system and method, device and medium to solve the problem that the two-dimensional image of the target region of the patient obtained by the existing imaging system cannot provide more detailed structural information of the target region of the patient, thereby making it difficult to assist doctors to obtain more accurate disease diagnosis results.

[0005] The first aspect of the present application provides a three-dimensional model construction system, which is used to construct a three-dimensional model corresponding to a target object, and comprises:

[0006] an image acquisition module, configured to acquire color images of the target object from different angles and fluorescence images of a target region in the target object;

[0007] a control module configured to control the image acquisition module to normally complete acquisition of the color images of the target object at different viewing angles and the fluorescence images of the target region in the target object;

[0008] an image processing module configured to obtain a three-dimensional model corresponding to the target object based on the color images of the target object at different viewing angles and the fluorescence images of the target region in the target object.

[0009] In some embodiments of the present application, the image acquisition module comprises:

[0010] a fluorescence camera configured to acquire the fluorescence images of the target region in the target object;

[0011] a color camera assembly configured to acquire the color images of the target object at different viewing angles, wherein the color camera assembly comprises a plurality of binocular color cameras, each of the binocular color cameras is configured to acquire the color images of the target object at two viewing angles, and each of the binocular color cameras is composed of two color cameras arranged at a preset distance, each of the color cameras is configured to acquire the color images of the target object at one viewing angle;

[0012] a laser light source configured to perform laser irradiation on the target region in the target object to enable the fluorescence camera to acquire the fluorescence images of the target region in the target object.

[0013] In some embodiments of the present application, the image acquisition module further comprises:

[0014] a filter configured to enable the fluorescence camera to acquire the fluorescence images of the target region in the target object at a specified wave band;

[0015] an illuminating lamp configured to provide illumination to assist the color camera assembly in acquiring the color images of the target object at different viewing angles and the fluorescence camera in acquiring the fluorescence images of the target region in the target object.

[0016] In some embodiments of the present application, the image processing module is configured to obtain the three-dimensional model corresponding to the target object in the following manner:

[0017] performing denoising and / or false colorization processing on the fluorescence images of the target region in the target object to obtain first fluorescence images of the target region in the target object;

[0018] arbitrarily selecting the color images of the target object at two viewing angles acquired by one of the binocular color cameras, and respectively registering the first fluorescence images of the target region in the target object with the color images of the target object at each of the selected viewing angles to obtain first fluorescence images of the target region in the target object at each of the viewing angles;

[0019] fusing the first fluorescent image of the target region in the target object at each view and the color image of the target object at the view to obtain a color image of the target object at each view containing the fluorescent region;

[0020] calibrating a selected binocular color camera to obtain an intrinsic matrix, relative extrinsic parameters and distortion parameters of the binocular color camera;

[0021] based on the intrinsic matrix, relative extrinsic parameters and distortion parameters of the binocular color camera, performing stereo rectification on the color image of the target object at each view containing the fluorescent region, and performing stereo matching on the color images of the target object at two views containing the fluorescent region after stereo rectification to obtain a disparity map between the color images of the target object at the two views containing the fluorescent region;

[0022] based on the disparity map between the color images of the target object at the two views containing the fluorescent region and the intrinsic matrix, relative extrinsic parameters and distortion parameters of the binocular color camera, obtaining a depth map corresponding to the target object;

[0023] performing back projection on the depth map corresponding to the target object to obtain a three-dimensional point cloud corresponding to the target object, and performing three-dimensional reconstruction on the three-dimensional point cloud corresponding to the target object to obtain a three-dimensional model corresponding to the target object.

[0024] In some embodiments of the present application, the image processing module is further configured to:

[0025] arbitrarily selecting the color images of the target object at two views collected by the binocular color camera, and registering the first fluorescent image of the target region in the target object with the color image of the selected target object at each view, respectively, and performing segmentation and / or block extraction processing on the fluorescent image of the target region in the target object at each view after registration to obtain the first fluorescent image of the target region in the target object at each view.

[0026] In some embodiments of the present application, the image processing module is further configured to:

[0027] registering the first fluorescent image of the target region in the target object with the color image of the selected target object at each view using a perspective transformation method to obtain the first fluorescent image of the target region in the target object at each view;

[0028] performing stereo matching on the two-view color images of the target object containing the fluorescent region after stereo rectification by using a semi-global block matching method to obtain an initial disparity map between the two-view color images of the target object containing the fluorescent region, and performing optimization processing on the initial disparity map between the two-view color images of the target object containing the fluorescent region by using a fast marching method to obtain a disparity map between the two-view color images of the target object containing the fluorescent region.

[0029] performing three-dimensional reconstruction on the three-dimensional point cloud corresponding to the target object by using a greedy projection triangulation method to obtain a three-dimensional model corresponding to the target object.

[0030] In some embodiments of the present application, the system further comprises:

[0031] a positioning module configured to extract coordinates of four vertices of a circumscribed matrix of a fluorescent region in a color image of each view of the target object containing the fluorescent region after stereo rectification, and calculate a length and a width of the fluorescent region based on the coordinates of the four vertices.

[0032] A second aspect of the present application provides a three-dimensional model construction method, the method comprising:

[0033] obtaining two-view color images of a target object and a fluorescent image of a target region in the target object, wherein the two-view color images of the target object are obtained by photographing the target object by a binocular color camera, and the fluorescent image of the target region in the target object is obtained by photographing the target object by a fluorescent camera;

[0034] performing denoising and / or pseudo-color processing on the fluorescent image of the target region in the target object to obtain a first fluorescent image of the target region in the target object;

[0035] registering the first fluorescent image of the target region in the target object with a selected color image of each view of the target object, respectively, to obtain a first fluorescent image of each view of the target region in the target object;

[0036] performing fusion processing on the first fluorescent image of each view of the target region in the target object and the color image of the view of the target object to obtain a color image of each view of the target object containing the fluorescent region;

[0037] obtaining an intrinsic matrix, a relative extrinsic parameter, and a distortion parameter of the binocular color camera, wherein the intrinsic matrix, the relative extrinsic parameter, and the distortion parameter of the binocular color camera are obtained by calibrating the binocular color camera;

[0038] stereo correct the color image of each view angle of the target object containing the fluorescent region based on the intrinsic matrix, relative extrinsic parameters and distortion parameters of the binocular color camera, and perform stereo matching on the stereo corrected color images of two view angles of the target object containing the fluorescent region to obtain a disparity map between the color images of two view angles of the target object containing the fluorescent region;

[0039] obtain the depth map corresponding to the target object based on the disparity map between the color images of two view angles of the target object containing the fluorescent region and the intrinsic matrix, relative extrinsic parameters and distortion parameters of the binocular color camera;

[0040] perform back projection processing on the depth map corresponding to the target object to obtain a three-dimensional point cloud corresponding to the target object, and perform three-dimensional reconstruction on the three-dimensional point cloud corresponding to the target object to obtain a three-dimensional model corresponding to the target object.

[0041] The third aspect of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of any one of the second aspect in the above embodiments when executing the computer program.

[0042] The fourth aspect of the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the method of any one of the second aspect in the above embodiments.

[0043] The three-dimensional model construction scheme of the present application acquires color images of different view angles of a target object and fluorescent images of a target region in the target object, and constructs a three-dimensional model corresponding to the target object based on the color images of different view angles of the target object and the fluorescent images of the target region in the target object. Therefore, the three-dimensional model construction scheme of the present application obtains a three-dimensional model of the target object, and when this scheme is used to acquire images of a target region of a patient, a three-dimensional image of the target region of the patient can be obtained, which can provide more detailed structural information of the target region of the patient, thereby better assisting doctors to obtain more accurate disease diagnosis results. BRIEF DESCRIPTION OF DRAWINGS

[0044] The drawings incorporated into the specification and forming a part of the specification, show embodiments consistent with the present application, and together with the specification, serve to explain the technical solutions of the present application.

[0045] Figure 1 is a first embodiment framework schematic diagram of the three-dimensional model construction system provided by the present application;

[0046] Figure 2is a second embodiment framework schematic diagram of a three-dimensional model construction system provided by the present application;

[0047] Figure 3 is a third embodiment framework schematic diagram of a three-dimensional model construction system provided by the present application;

[0048] Figure 4 is a fourth embodiment framework schematic diagram of a three-dimensional model construction system provided by the present application;

[0049] Figure 5 is a fifth embodiment framework schematic diagram of a three-dimensional model construction system provided by the present application;

[0050] Figure 6 is a sixth embodiment framework schematic diagram of a three-dimensional model construction system provided by the present application;

[0051] Figure 7 is an example framework schematic diagram of a setting mode between components in an image acquisition module of a three-dimensional model construction system provided by the present application;

[0052] Figure 8 is a flow schematic diagram of an embodiment of constructing a three-dimensional model of a target object by an image processing module in a three-dimensional model construction system provided by the present application;

[0053] Figure 9 is an example schematic diagram of a comparison between before and after pseudo-colorization of an image provided by the present application;

[0054] Figure 10 is an example schematic diagram of a result of registering a fluorescent image of a target object with color images of two perspectives provided by the present application;

[0055] Figure 11 is an example schematic diagram of a comparison between before and after block extraction processing of an image provided by the present application;

[0056] Figure 12 is an example schematic diagram of a result of stereo correction of images of different perspectives provided by the present application;

[0057] Figure 13 is an example schematic diagram of generating a disparity map based on images of two perspectives provided by the present application;

[0058] Figure 14 is an example schematic diagram of a comparison between before and after optimization of a disparity map provided by the present application;

[0059] Figure 15 is an example schematic diagram of a circumscribed matrix of a fluorescent region in an image provided by the present application;

[0060] Figure 16is an experimental result schematic diagram of different perspective three-dimensional models corresponding to a target object obtained by using the three-dimensional model construction system of the present application;

[0061] Figure 17 is a flowchart of an embodiment of the three-dimensional model construction method provided by the present application;

[0062] Figure 18 is a framework schematic diagram of an embodiment of the electronic device provided by the present application;

[0063] Figure 19 is a framework schematic diagram of an embodiment of the computer readable storage medium provided by the present application. DETAILED DESCRIPTION

[0064] The scheme of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0065] In the following description, specific details are set forth in order to provide a thorough understanding of the present application, but the present application can be practiced without these particulars. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to obscure the present application.

[0066] The term "and / or" herein merely describes an associated relationship between associated objects, and means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " herein generally means that the front and rear associated objects are in an "or" relationship. In addition, "multiple" herein means two or more than two. In addition, the term "at least one" herein means any one of multiple or any combination of at least two of multiple, for example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0067] As described in the background, when the existing imaging system is used to obtain an image of a target region of a patient, only a two-dimensional image of the target region of the patient can be obtained, and the two-dimensional image cannot provide more detailed structural information of the target region of the patient, so it is difficult to assist doctors to obtain more accurate disease diagnosis results.

[0068] Therefore, the present application proposes a new scheme. In the scheme of the present application, a color image of a target object and a fluorescence image of a target region in the target object are obtained based on color imaging and fluorescence imaging technology respectively, and the target object is three-dimensionally reconstructed based on the two images to obtain a three-dimensional model corresponding to the target object. When this scheme is used to construct an image of a target region of a patient, a three-dimensional image of the target region of the patient can be obtained, and the three-dimensional image can provide more detailed structural information of the target region of the patient, thereby better assisting doctors to obtain more accurate disease diagnosis results.

[0069] The application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0070] According to one embodiment of the application, as shown in Figure 1 The application provides a three-dimensional model construction system for constructing a three-dimensional model corresponding to a target object, the system comprising: an image acquisition module for acquiring color images of the target object from different perspectives and fluorescence images of a target region in the target object; a control module for controlling the image acquisition module to normally complete acquisition of the color images of the target object from different perspectives and the fluorescence images of the target region in the target object; and an image processing module for obtaining the three-dimensional model corresponding to the target object based on the color images of the target object from different perspectives and the fluorescence images of the target region in the target object.

[0071] As can be seen from the above embodiments, when the three-dimensional model construction system of the application is used to acquire images of a target region of a patient, a three-dimensional image of the target region of the patient can be obtained, which can provide more detailed structural information of the target region of the patient, thereby better assisting doctors to obtain more accurate disease diagnosis results.

[0072] According to one embodiment of the application, as shown in Figure 2 The system further comprises a display module for displaying the three-dimensional model corresponding to the target object. In addition, according to one embodiment of the application, as shown in Figure 3 The system further comprises a positioning module for extracting coordinates of four vertices of a circumscribed matrix constituted by a fluorescence region in a color image from each perspective of the target object after stereoscopic correction, and calculating a length and a width of the fluorescence region based on the coordinates of the four vertices.

[0073] In order to more clearly understand the scheme of the application, each module in the three-dimensional model construction system of the application will be described below.

[0074] I. Image acquisition module

[0075] The image acquisition module is used to acquire color images of a target object from different perspectives and fluorescence images of a target region in the target object.

[0076] According to one embodiment of the application, as shown in Figure 4As shown, the image acquisition module includes: a fluorescence camera configured to acquire a fluorescence image of a target region in the target object; a color camera assembly configured to acquire color images of the target object from different perspectives, wherein the color camera assembly includes a plurality of binocular color cameras, each of which is configured to acquire color images of the target object from two perspectives, and each of which is formed by two color cameras arranged at a preset distance, each of which is configured to acquire a color image of the target object from one perspective; and a laser light source (e.g., a near-infrared laser light source) configured to irradiate a target region in the target object with laser light so that the fluorescence camera acquires a fluorescence image of the target region in the target object. According to an embodiment of the present application, the fluorescence camera is a near-infrared CMOS camera, and the color camera is a color CMOS camera. The near-infrared CMOS camera can efficiently capture fluorescence signals and convert them into digital image data. It should be noted that the near-infrared CMOS camera and the color CMOS camera are only an embodiment of the present application, and with the development of image acquisition devices, better performing devices can be further selected to replace them according to application requirements, characteristics of the target object, system integration requirements, and image processing requirements, etc., without being specifically limited here.

[0077] According to an embodiment of the present application, as shown in Figure 5 The image acquisition module further includes: an optical filter configured to enable the fluorescence camera to acquire a fluorescence image of a target region in the target object in a specified waveband; and an illuminating lamp (e.g., an LED lamp) configured to provide illumination to assist the color camera assembly in acquiring color images of the target object from different perspectives and the fluorescence camera in acquiring a fluorescence image of a target region in the target object. According to an embodiment of the present application, as shown in Figure 6 The image acquisition module further includes: a laser homogenizing filter configured to adjust the light beam of the laser light source so that the light beam emitted by the laser light source can uniformly irradiate the target region of the target object.

[0078] From the above embodiments, it can be seen that the filter functions to select a specific wavelength of light source or fluorescent signal to improve the signal-to-noise ratio and contrast of imaging; the near-infrared laser light source is used to excite the ICG fluorescent probe on the target region of the target object to generate a fluorescent signal; the laser homogenizing filter is used to adjust the light beam of the laser light source to ensure that the light source can uniformly irradiate the target region of the target object, while having appropriate spot size and shape; the LED lamp serves as auxiliary illumination to provide uniform brightness of the surrounding environment; the laser homogenizing filter can make the light beam emitted by the laser light source uniformly irradiate the target region of the target object. The above embodiments are beneficial to improving the imaging quality. It should be noted that the LED lamp and the near-infrared laser light source are only one embodiment of the present application, and the lighting lamp and the laser light source in the specific three-dimensional model construction system can be further replaced by devices with better performance according to application requirements, characteristics of the target object, system integration requirements, image processing requirements and other factors, which are not limited here.

[0079] In order to more intuitively understand how the components in the image acquisition module in the above embodiments work together to complete high-quality image acquisition, the setting mode of each part will be described below in combination with specific examples.

[0080] The present application designs a three-camera hardware structure of double-color cameras and a single fluorescent camera as an example to visually illustrate the setting mode of the cameras in the image acquisition module, as shown in Figure 7 In the design of the present application, the horizontal distance between the two color cameras is 70 mm, and the distance from them to the laser light source is 35 mm. The distance from the laser light source to the near-infrared camera (i.e. the fluorescent camera) is 50 mm, and the distance from the center of the near-infrared camera lens to the line connecting the centers of the two color cameras is 70 mm. Figure 7 Such size parameter structure is designed in the above embodiment, mainly to ensure that the public field of view of the three cameras near the field of view at 40 cm object distance can reach 10 cm*10 cm. It should be noted that the setting mode of each part in the above image acquisition module is only one example, and the setting mode of each part in the specific image acquisition module is determined according to the comprehensive factors such as field of view overlap, optical property matching, interference reduction, physical space limitation and application scenario requirement, which is not limited here.

[0081] The above example of setting the camera position fully considers the requirements of the system for high precision and high sensitivity, ensures that the optical module and mechanical structure of the double-color camera and the single fluorescent camera meet the requirements and specifications of the system, so that high-quality images can be obtained under the structure of image acquisition.

[0082] II. Control module

[0083] The control module is configured to control the image acquisition module to normally complete acquisition of the color images of the target object at different viewing angles and the fluorescence images of the target region in the target object. For example, the main functions of the control module include: setting and adjusting the exposure time and gain of the camera to ensure image quality and clarity; controlling the camera hardware parameters such as frame rate, resolution, etc. to meet the system requirements; monitoring the camera state in real time, including the connection state, image acquisition state, etc. The control module needs to control three cameras (one fluorescence camera and two color cameras) at the same time to complete the acquisition of fluorescence images and color images.

[0084] As can be seen from the above embodiments, by controlling the related parameters of the devices in the image acquisition module through the control module, the quality of the acquired images can be improved and the flexible operation of the system can be improved.

[0085] III. Image processing module

[0086] The image processing module is configured to obtain a three-dimensional model corresponding to the target object based on the color images of the target object at different viewing angles and the fluorescence images of the target region in the target object.

[0087] According to one embodiment of the present application, as shown in Figure 8 The image processing module is configured to perform steps S1-S7 to complete the construction of the three-dimensional model corresponding to the target object. The contents of steps S1-S7 are described in detail below.

[0088] In step S1, the fluorescence images of the target region in the target object are denoised and / or pseudo-colored to obtain the first fluorescence images of the target region in the target object.

[0089] The fluorescence images taken by the fluorescence camera are usually grayscale images. However, the resolution of the human eye for grayscale images is relatively low, and generally only 20 or more grayscale levels can be distinguished. On the contrary, the human eye is more sensitive to color changes and can distinguish thousands of different color tones, brightness and saturation of color images at the same time. Therefore, in order to better display the details and characteristics of the image, the grayscale image is usually converted into a color image, and the visual effect of the image is enhanced by mapping the grayscale difference that the human eye cannot distinguish into color difference, which is the pseudo-color processing. As shown in Figure 9 The comparison chart before and after pseudo-color processing of the fluorescence images of the target object is shown, wherein, Figure 9 (a) is a grayscale fluorescence image of the target object acquired by the fluorescence camera; Figure 9 (b) is a fluorescence image of the target object after pseudo-color processing of the fluorescence image of the target object. It can be seen that the visual effect of the fluorescence image after pseudo-color processing is better and more in line with the observation of the human eye.

[0090] In the step S2, one of the color images of the target object collected by the binocular color camera is selected, and the first fluorescent image of the target region in the target object is registered with the color image of each view of the selected target object, respectively, to obtain the first fluorescent image of each view of the target region in the target object.

[0091] Since the shooting angle and resolution of the color camera and the fluorescent camera of the system are different, and the fluorescent region obtained by the fluorescent camera has no feature point that can be extracted, the images obtained by the two cameras will have differences in view angle and size, and cannot be registered by the traditional feature extraction method. In order to make the two images with different view angles and sizes basically aligned, according to an embodiment of the present application, the first fluorescent image of the target region in the target object is registered with the color image of each view of the selected target object by using the perspective transformation method, to obtain the first fluorescent image of each view of the target region in the target object. It should be noted that the perspective transformation method for image registration is only one embodiment of the present application, and the specific registration method needs to be determined according to various factors such as task requirements, image type and characteristics, computing resources, etc., which are not limited here.

[0092] For example, as shown in FIG. 2, it shows the registration results of the first fluorescent image of the target region in the target object with the color images of two views of the selected target object, wherein, Figure 10 (a) is the registration result of the first fluorescent image of the target region in the target object with the color image of one view of the target object, Figure 10 (b) is the registration result of the first fluorescent image of the target region in the target object with the color image of another view of the target object. Figure 10 In the above embodiment, the first fluorescent image of the target region in the target object is registered with the color image of each view of the selected target object, that is, the position of the first fluorescent image of the target region in the target object is aligned with the position of the target region in the color images of two views of the target object, respectively, so that when the fluorescent image and the color image are fused subsequently, the position of the fluorescent region of the target object obtained is accurate.

[0093]

[0094] ​According to one embodiment of the present application, the image processing module is further configured to: randomly select one color image of the target object from the two views collected by the binocular color camera, and register the first fluorescent image of the target region in the target object with the color image of each view of the target object selected, and perform segmentation and / or block extraction processing on the registered fluorescent image of each view of the target region in the target object, to obtain the first fluorescent image of each view of the target region in the target object. According to one embodiment of the present application, the image processing module is further configured to: perform segmentation processing on the registered fluorescent image of each view of the target region in the target object by using the OTSU method. It should be noted that the OTSU method is only one embodiment of the image segmentation processing of the present application, and the specific segmentation method needs to be determined according to task requirements, image types and characteristics, computing resources and other factors, which are not limited here.

[0095] In order to more intuitively understand the concept of block extraction processing in the above embodiment, the following will be described in combination with Figure 11 The block extraction is described. As shown in Figure 11 If an image contains different objects, each object in the image can be extracted by block extraction processing, that is, in the image, a pentagon, an ellipse and a polygon are connected to form an image, and different objects in the image can be extracted separately by block extraction processing.

[0096] The above embodiment improves the accuracy of three-dimensional reconstruction of the fluorescent region in the target object by performing segmentation and / or block extraction processing on the fluorescent image of each view of the target region in the target object to obtain a registered image containing only the fluorescent region. The accuracy of the three-dimensional reconstruction of the fluorescent region in the target object is improved. This scheme for three-dimensional model reconstruction of the fluorescent region in the target object can be applied to the medical field, and the accuracy of obtaining the three-dimensional image of the target region of the patient can be improved. Doctors can more intuitively and carefully observe and diagnose the condition of the target region of the patient.

[0097] In the step S3, the first fluorescent image of each view of the target region in the target object and the color image of the target object of the view are fused to obtain a color image of each view of the target object containing the fluorescent region.

[0098] In the step S4, one binocular color camera is selected for calibration to obtain the intrinsic matrix, relative extrinsic parameters and distortion parameters of the binocular color camera.

[0099] In step S5, based on the intrinsic parameter matrix, relative extrinsic parameter, and distortion parameter of the binocular color camera, stereo correction is performed on the color images of each viewpoint containing the fluorescent region of the target object, and stereo matching is performed on the stereo-corrected color images of the target object containing the fluorescent region from two viewpoints to obtain a disparity map between the color images of the target object containing the fluorescent region from two viewpoints.

[0100] According to one embodiment of this application, the image processing module is further configured to: perform stereo matching on two color images of the target object containing a fluorescent region from two different perspectives using a semi-global block matching method to obtain an initial disparity map between the two color images of the target object containing the fluorescent region; and optimize the initial disparity map between the two color images of the target object containing the fluorescent region using a fast traversal method to obtain a disparity map between the two color images of the target object containing the fluorescent region. It should be noted that the semi-global block matching method for stereo matching and the fast traversal method for optimizing the disparity map are both embodiments of this application. Specific matching and optimization methods need to be determined based on various factors such as task requirements, image type and characteristics, and computing resources, and are not specifically limited here.

[0101] For example, such as Figure 12 As shown, it demonstrates the... Figure 10 The image shows the result of stereo correction of the two images in the image. Figure 12 (a) is for Figure 10 (a) The effect after stereoscopic correction. Figure 12 (b) is for Figure 10 (b) The result after stereo correction. It is clear that corresponding points in the two images after stereo correction lie on the same horizontal line. For example... Figure 13 As shown, it illustrates the result of stereo matching between two color images of the target object containing fluorescent regions from two different viewpoints, resulting in a disparity map between the two images. Figure 13 (a) is a color image of the target object containing fluorescent regions from one viewpoint. Figure 13 (b) is a color image of the target object containing fluorescent regions from another perspective. Figure 13 (c) is a disparity map between two color images of the target object containing a fluorescent region from two different viewpoints. For example... Figure 14 As shown, it displays a comparison image before and after optimizing the disparity map, in which... Figure 14 (a) is a disparity map between two color images of the target object containing fluorescent regions from two different viewpoints before optimization. Figure 14(b) is a disparity map between color images of two views of the target object containing the fluorescent region after optimization.

[0102] As can be seen from the above embodiments, the stereo correction can eliminate image distortion caused by different camera installation angles, so that corresponding points in the two images are located on the same horizontal line, thereby simplifying and improving the accuracy and efficiency of subsequent stereo matching; the optimization processing of the disparity map can improve the smoothness and continuity of the disparity map, improve the accuracy of the disparity map, improve the quality of the depth map, enhance the robustness, and improve the calculation efficiency. These optimization measures help to generate more accurate three-dimensional models and improve the application effect of medical imaging.

[0103] In the step S6, based on the disparity map between color images of two views of the target object containing the fluorescent region and the intrinsic matrix, relative extrinsic parameters and distortion parameters of the binocular color camera, a depth map corresponding to the target object is obtained.

[0104] In the step S7, the depth map corresponding to the target object is subjected to back projection processing to obtain a three-dimensional point cloud corresponding to the target object, and the three-dimensional point cloud corresponding to the target object is subjected to three-dimensional reconstruction to obtain a three-dimensional model corresponding to the target object.

[0105] According to an embodiment of the present application, the image acquisition module is further configured to use a greedy projection triangulation method to perform three-dimensional reconstruction on the three-dimensional point cloud corresponding to the target object to obtain the three-dimensional model corresponding to the target object. It should be noted that the use of the greedy projection triangulation method to perform three-dimensional reconstruction on the target object is only one embodiment of the present application, and the specific reconstruction method needs to be determined according to task requirements, image type and characteristics, computing resources and other factors, which are not limited here.

[0106] IV. Display module

[0107] The display module is used to display the three-dimensional model corresponding to the target object.

[0108] As can be seen from the above embodiments, when the three-dimensional model construction system of the present application is used to construct a three-dimensional image of a target region of a patient, the constructed three-dimensional image of the patient is finally displayed through the display module, and the doctor can more intuitively observe the structural features of the target region of the patient, thereby making a more accurate diagnosis result.

[0109] V. Positioning module

[0110] The positioning module is used to extract the coordinates of four vertices of a circumscribed matrix formed by the fluorescent region in the color image of each view of the target object containing the fluorescent region after stereo correction, and calculate the length and width of the fluorescent region based on the coordinates of the four vertices.

[0111] In one embodiment of this application, the coordinates of the four vertices of the circumscribed matrix of the fluorescent regions in each viewpoint of the stereo-corrected color image of the target object containing the fluorescent regions are extracted using the connected component labeling method. The length and width of the fluorescent regions are then calculated based on these coordinates. It should be noted that extracting the vertex coordinates of the circumscribed matrix of the fluorescent regions in the image using the connected component labeling method is only one embodiment of this application. Specific extraction methods need to be determined based on various factors such as task requirements, image type and characteristics, and computing resources, and are not specifically limited here.

[0112] For example, such as Figure 15 As shown, it displays the circumscribed matrix ABCD formed by the fluorescent regions in the target object image. By extracting the coordinates of A, B, C, and D in three-dimensional space and calculating the Euclidean distances of AB and AC respectively, and averaging the AB and AC values ​​after multiple calculations, the length and width of the fluorescent region are measured.

[0113] As can be seen from the above embodiments, the three-dimensional model construction system of this application, when applied to the medical field, extracts the coordinates of the circumscribed rectangle vertices of the fluorescent region through the fluorescent region localization module and calculates its length and width. This allows for more precise determination of the patient's target region location, improving localization accuracy, quantitative analysis capabilities, and the accuracy of assisted diagnosis and treatment planning, while also increasing work efficiency and standardization. These benefits are of great significance for improving the effectiveness of medical diagnosis and treatment.

[0114] To verify the effectiveness of the 3D model construction system of the above embodiments of this application, the inventors conducted a series of experiments, the specific experimental results of which are as follows:

[0115] like Figure 16 As shown, it illustrates 3D models of a target object obtained from different perspectives using the 3D model construction system of this application. Among them, Figure 16 (a) is the main view of the 3D model corresponding to the target object; Figure 16 (b) is the left side view of the 3D model corresponding to the target object; Figure 16 (c) is the right-side view of the 3D model corresponding to the target object; Figure 16 (d) is an attached view of the 3D model corresponding to the target object; Figure 16 (e) is a bottom view of the 3D model corresponding to the target object. Experimental results show that the 3D model construction system of this application can obtain 3D models of the target object from different perspectives. The 3D models from different perspectives can provide patients with more comprehensive and detailed structural information of the target area, thereby better assisting doctors in obtaining more accurate disease diagnosis results.

[0116] The following section details the three-dimensional model construction scheme of this application from the perspective of methodology.

[0117] According to one embodiment of the present application, as shown in Figure 17 According to one embodiment of the present application, as shown in

[0118] According to one embodiment of the present application, the step T3 comprises: registering the first fluorescent image of the target region in the target object with the color image of each selected view of the target object, and performing segmentation and / or block extraction on the registered fluorescent image of each view of the target region in the target object to obtain the first fluorescent image of each view of the target region in the target object.

[0119] In summary, compared with the existing imaging system applied to the medical field, only two-dimensional images of the target region of the patient can be obtained, the three-dimensional model construction system of the present application can obtain three-dimensional images of the target region of the patient when applied to the medical field to obtain images of the target region of the patient, and the three-dimensional images can provide more detailed structural information of the target region of the patient, thereby better assisting doctors to obtain more accurate disease diagnosis and treatment results.

[0120] In addition, according to one embodiment of the present application, the present application provides a three-dimensional model construction method, which comprises: three-dimensional reconstruction of a target object by using the three-dimensional model construction system or method of any of the above embodiments.

[0121] As can be seen from the above embodiments, the three-dimensional model construction system of the present application is used to obtain three-dimensional images of the target region of the patient, the three-dimensional images can provide more detailed structural information of the target region of the patient, thereby better assisting doctors to obtain more accurate disease diagnosis results.

[0122] Based on the inventive concept of the above embodiments, the present application further provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the method described in the above embodiments when executing the computer program. The following will be described in detail in combination with Figure 18 .

[0123] As Figure 19 shown, the electronic device of the present application is shown, which can specifically comprise a processor 110 and a memory 120. The memory 120 is coupled to the processor 110.

[0124] The processor 110 is used to control the operation of the electronic device, and the processor 110 can also be called a CPU (Central Processing Unit). The processor 110 can be an integrated circuit chip with signal processing capability. The processor 110 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor 110 can also be any conventional processor.

[0125] The memory 120 is configured to store a computer program, and can be a RAM, a ROM, or other types of storage terminals. Specifically, the memory 120 can include one or more computer-readable storage media, which can be non-transitory or transitory. The memory 120 can further include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage terminals, flash storage terminals. In some embodiments, the non-transitory computer-readable storage medium in the memory 120 is configured to store at least one program code.

[0126] The processor 110 is configured to execute the computer program stored in the memory 120 to implement the methods described in the method embodiments of the present application.

[0127] In some embodiments, the electronic device can further include a peripheral terminal interface 130 and at least one peripheral terminal. The processor 110, the memory 120, and the peripheral terminal interface 130 can be connected through a bus or a signal line. Each peripheral terminal can be connected to the peripheral terminal interface 130 through a bus, a signal line, or a circuit board. Specifically, the peripheral terminal includes at least one of a radio frequency circuit 140, a display screen 150, an audio circuit 160, and a power supply 170.

[0128] The peripheral terminal interface 130 can be configured to connect at least one peripheral terminal related to I / O (Input / output) to the processor 110 and the memory 120. In some embodiments, the processor 110, the memory 120, and the peripheral terminal interface 130 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 110, the memory 120, and the peripheral terminal interface 130 can be implemented on a separate chip or circuit board, and the present embodiment is not limited in this regard.

[0129] The radio frequency circuit 140 is configured to receive and send RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 140 communicates with communication networks and other wireless devices through electromagnetic signals. The radio frequency circuit 140 is the communication circuit of the electronic device. The radio frequency circuit 140 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 140 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, an operating personnel identity module, and the like. The radio frequency circuit 140 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G, and 5G), a wireless local area network, and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 140 can also include NFC (Near Field Communication) related circuitry, which is not limited in the present application.

[0130] The display screen 150 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 150 is a touch display screen, the display screen 150 also has the ability to collect touch signals on or above the surface of the display screen 150. The touch signals can be input as control signals to the processor 110 for processing. At this time, the display screen 150 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 150 can be one, arranged on the front panel of the electronic device; in other embodiments, the display screen 150 can be at least two, arranged on different surfaces of the electronic device or in a folding design; in other embodiments, the display screen 150 can be a flexible display screen, arranged on a curved surface or a folding surface of the electronic device. Even, the display screen 150 can also be arranged in an irregular shape, that is, a special-shaped screen. The display screen 150 can be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), and the like.

[0131] The audio circuit 160 can include a microphone and a speaker. The microphone is used to collect sound waves of the operator and the environment, and convert the sound waves into an electrical signal input to the processor 110 for processing, or input to the radio frequency circuit 140 to realize voice communication. For the purpose of stereo sound collection or noise reduction, the microphone can be multiple, respectively arranged at different parts of the electronic device. The microphone can also be an array microphone or an omnidirectional collection type microphone. The speaker is used to convert the electrical signal from the processor 110 or the radio frequency circuit 140 into sound waves. The speaker can be a traditional diaphragm speaker, or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only can the electrical signal be converted into a sound wave that humans can hear, but also can be converted into a sound wave that humans cannot hear for ranging purposes. In some embodiments, the audio circuit 160 can also include a headphone jack.

[0132] The power supply 170 is used to supply power to each component in the electronic device. The power supply 170 can be alternating current, direct current, disposable battery or rechargeable battery. When the power supply 170 includes a rechargeable battery, the rechargeable battery can be a wired charging battery or a wireless charging battery. The wired charging battery is a battery charged through a wired line, and the wireless charging battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0133] For the detailed description of the functions and execution processes of each functional module or component in the electronic device embodiments of the present application, reference can be made to the description of each method embodiment of the present application. Here, it will not be repeated.

[0134] In several embodiments provided in the present application, it should be understood that the disclosed electronic device and method can be implemented in other ways. For example, the above-described electronic device embodiments are only schematic, for example, the division of modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0135] The units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment scheme.

[0136] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0137] Based on the inventive concept of the above embodiments, the present application further provides a computer-readable storage medium storing a computer program, which can be executed by a processor to implement the above projection area calculation method. The execution process of the above embodiments in the computer-readable storage medium is described below. Figure 19 The execution process of the above embodiments in the computer-readable storage medium is described below.

[0138] As shown in ​ The computer-readable storage medium of the present application is shown, and the integrated unit, if realized in the form of a software function unit and sold or used as an independent product, can be stored in the computer-readable storage medium 200. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions / computer programs for causing an Internet of Things device (which can be a personal computer, a server, or a network terminal, etc.) or a processor to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media having the above storage medium, and a computer, a mobile phone, a notebook computer, a tablet computer, a camera, and other electronic terminals.

[0139] The execution process of the program data in the computer-readable storage medium is described with reference to the above-described method embodiments of the present application, and will not be described here.

[0140] The above is only an embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the contents of the specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

[0141] Those skilled in the art can understand that in the above method of the specific implementation, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

Claims

1. A three-dimensional model construction system, the system being used to construct a three-dimensional model corresponding to a target object, characterized in that, The system includes: The image acquisition module is used to acquire color images of the target object from different perspectives and fluorescence images of the target region within the target object; The control module is used to control the image acquisition module so that the image acquisition module can normally complete the acquisition of color images of the target object from different perspectives and fluorescence images of the target area in the target object; An image processing module is used to obtain a three-dimensional model of the target object based on color images of the target object from different viewpoints and fluorescence images of the target region within the target object; wherein, the image processing module is configured to obtain the three-dimensional model of the target object in the following manner: The fluorescence image of the target region in the target object is denoised and / or pseudo-colorized to obtain a first fluorescence image of the target region in the target object; Arbitrarily select two color images of the target object from two perspectives captured by a binocular color camera, and register the first fluorescence image of the target region in the target object with the selected color image of the target object from each perspective to obtain the first fluorescence image of the target region in the target object from each perspective. The first fluorescence image of the target region in the target object from each viewpoint and the color image of the target object from that viewpoint are fused together to obtain a color image of the target object containing the fluorescence region from each viewpoint. A selected binocular color camera is calibrated to obtain the intrinsic parameter matrix, relative extrinsic parameters, and distortion parameters of the binocular color camera. Based on the intrinsic parameter matrix, relative extrinsic parameter, and distortion parameter of the binocular color camera, stereo correction is performed on the color image of each view containing the fluorescent region of the target object, and stereo matching is performed on the color images of the two view containing the fluorescent region of the target object after stereo correction, so as to obtain the disparity map between the color images of the two view containing the fluorescent region of the target object. Based on the disparity map between two color images of the fluorescent region in the target object from two different perspectives, and the intrinsic parameter matrix, relative extrinsic parameter, and distortion parameter of the binocular color camera, a depth map corresponding to the target object is obtained. The depth map corresponding to the target object is back-projected to obtain the three-dimensional point cloud corresponding to the target object, and the three-dimensional point cloud corresponding to the target object is reconstructed to obtain the three-dimensional model corresponding to the target object.

2. The system according to claim 1, characterized in that, The image acquisition module includes: A fluorescence camera is used to acquire fluorescence images of the target region in the target object; A color camera assembly is used to acquire color images of the target object from different perspectives. The color camera assembly includes multiple binocular color cameras. Each binocular color camera is used to acquire color images of the target object from two perspectives. Each binocular color camera consists of two color cameras placed at a preset distance. Each color camera is used to acquire color images of the target object from one perspective. A laser light source is used to irradiate a target area in the target object with a laser, so that the fluorescence camera can acquire a fluorescence image of the target area in the target object.

3. The system according to claim 2, characterized in that, The image acquisition module also includes: A filter is used to enable the fluorescence camera to acquire fluorescence images of the target region in the target object within a specified wavelength band; An illumination lamp is provided to assist the color camera assembly in acquiring color images of the target object from different perspectives and the fluorescence camera in acquiring fluorescence images of the target region within the target object.

4. The system according to claim 3, characterized in that, The image processing module is also configured to: Arbitrarily select two color images of the target object captured by the binocular color camera from two different perspectives, and register the first fluorescence image of the target region in the target object with the selected color image of the target object from each perspective. Then, perform segmentation and / or block extraction processing on the registered fluorescence image of the target region in each perspective of the target object to obtain the first fluorescence image of the target region in each perspective of the target object.

5. The system according to claim 3, characterized in that, The image processing module is also configured to: The first fluorescence image of the target region in the target object is registered with the color image of the target object from each selected viewpoint using the perspective transformation method, so as to obtain the first fluorescence image of the target region in the target object from each viewpoint. A semi-global block matching method is used to perform stereo matching on two color images of the target object containing fluorescent regions after stereo correction, so as to obtain an initial disparity map between the two color images of the target object containing fluorescent regions. A fast traversal method is used to optimize the initial disparity map between the two color images of the target object containing fluorescent regions, so as to obtain a disparity map between the two color images of the target object containing fluorescent regions. The greedy projection triangulation method is used to reconstruct the three-dimensional point cloud corresponding to the target object in three dimensions to obtain the three-dimensional model of the target object.

6. The system according to claim 5, characterized in that, The system also includes: The positioning module is used to extract the coordinates of the four vertices of the circumscribed matrix of the fluorescent regions in the color image of each viewpoint containing the fluorescent regions of the target object after stereo correction, and to calculate the length and width of the fluorescent regions based on the coordinates of the four vertices.

7. A method for constructing a three-dimensional model based on the system described in any one of claims 1-6, characterized in that, The method includes: The method acquires color images of a target object from two perspectives and a fluorescence image of a target region within the target object. The color images of the target object from the two perspectives are obtained by capturing images of the target object using a binocular color camera, and the fluorescence image of the target region within the target object is obtained by capturing images of the target object using a fluorescence camera. The fluorescence image of the target region in the target object is denoised and / or pseudo-colorized to obtain a first fluorescence image of the target region in the target object; The first fluorescence image of the target region in the target object is registered with the color image of the target object from each selected viewpoint to obtain the first fluorescence image of the target region from each viewpoint in the target object; The first fluorescence image of the target region in the target object from each viewpoint and the color image of the target object from that viewpoint are fused together to obtain a color image of the target object containing the fluorescence region from each viewpoint. The intrinsic parameter matrix, relative extrinsic parameters, and distortion parameters of the binocular color camera are obtained, wherein the intrinsic parameter matrix, relative extrinsic parameters, and distortion parameters of the binocular color camera are obtained by calibrating the binocular color camera. Based on the intrinsic parameter matrix, relative extrinsic parameter, and distortion parameter of the binocular color camera, stereo correction is performed on the color image of each view containing the fluorescent region of the target object, and stereo matching is performed on the color images of the two view containing the fluorescent region of the target object after stereo correction, so as to obtain the disparity map between the color images of the two view containing the fluorescent region of the target object. Based on the disparity map between two color images of the fluorescent region in the target object from two different perspectives, and the intrinsic parameter matrix, relative extrinsic parameter, and distortion parameter of the binocular color camera, a depth map corresponding to the target object is obtained. The depth map corresponding to the target object is back-projected to obtain the three-dimensional point cloud corresponding to the target object, and the three-dimensional point cloud corresponding to the target object is reconstructed to obtain the three-dimensional model corresponding to the target object.

8. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the method as claimed in claim 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 7.

Images