Three-dimensional reconstruction method, device and electronic equipment

By segmenting and registering ultrasonic image images, combining feature matching and spatial transformation, the problem of low accuracy of three-dimensional reconstruction is solved, and a three-dimensional model generation with higher accuracy and stability is achieved.

CN118135111BActive Publication Date: 2024-11-22脉得智能科技(无锡)有限公司
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202410340589.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-25
Publication Date
2024-11-22
Estimated Expiration
2044-03-25

AI Technical Summary

Technical Problem

In the prior art, the accuracy of the three-dimensional reconstruction of the target organ is low, mainly due to the tremor caused by the patient's breathing, heartbeat physiological movement and probe movement, resulting in data loss and discontinuity.

Method used

By obtaining the video sequence of the target organ, image segmentation and registration of two-dimensional ultrasound image images are performed, image registration is performed using feature matching and spatial transformation estimation to reduce the impact of tremor on three-dimensional reconstruction, and fusion through three-dimensional local reconstruction and registration Eliminate image distortion and discontinuity to generate more accurate three-dimensional models.

Benefits of technology

Improve the accuracy and visualization of the three-dimensional model, reduce the impact of tremor and image distortion, and provide more stable and continuous ultrasonic image processing results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118135111B_ABST
    Figure CN118135111B_ABST
Patent Text Reader

Abstract

The present application provides a three-dimensional reconstruction method, device and electronic device, the method comprising: acquiring a video sequence of a target organ, the video sequence comprising a plurality of two-dimensional ultrasound images of the target organ; performing image segmentation on the plurality of two-dimensional ultrasound images to obtain a plurality of regional images of the target organ; performing two-dimensional image registration on the plurality of regional images to obtain a plurality of registered regional images; performing three-dimensional reconstruction on the registered plurality of regional images to obtain a three-dimensional model of the target organ. By performing image segmentation on the plurality of two-dimensional ultrasound images in the video sequence, and performing two-dimensional image registration and three-dimensional reconstruction on the obtained plurality of regional images of the target organ, the calculation process of the two-dimensional image registration plays a role in coordinate positioning, effectively reducing the influence of the patient's breathing, heartbeat physiological movement, and trembling of the probe movement on the three-dimensional reconstruction, thereby improving the accuracy of the obtained three-dimensional model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of artificial intelligence and ultrasonic image analysis, and in particular to a three-dimensional reconstruction method, device and electronic equipment. Background Art

[0002] At present, the main method of 3D reconstruction of target organs is to collect ultrasound images and coordinates measured by probes equipped with positioning equipment through special hardware facilities, and perform 3D reconstruction based on the coordinates and ultrasound images to obtain a 3D model. However, in actual practice, it is found that the positioning device and the probe are affected by the patient's breathing, heartbeat physiological movement, and trembling of the probe movement, resulting in missing data at certain angles or areas, resulting in low accuracy of 3D reconstruction to obtain a 3D model. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a three-dimensional reconstruction method, device and electronic device for improving the problem of low accuracy in obtaining a three-dimensional model.

[0004] The embodiment of the present application provides a three-dimensional reconstruction method, including: acquiring a video sequence of a target organ, the video sequence including multiple two-dimensional ultrasound images of the target organ; performing image segmentation on the multiple two-dimensional ultrasound images to obtain multiple regional images of the target organ; performing two-dimensional image registration on the multiple regional images to obtain multiple regional images after registration; performing three-dimensional reconstruction on the multiple regional images after registration to obtain a three-dimensional model of the target organ. In the implementation process of the above scheme, by performing image segmentation on the multiple two-dimensional ultrasound images in the video sequence, and performing two-dimensional image registration on the obtained multiple regional images of the target organ, and then performing three-dimensional reconstruction on the multiple regional images after registration, the calculation process of the two-dimensional image registration plays a role in coordinate positioning, effectively reducing the influence of the patient's breathing, heartbeat physiological movement, and trembling of the probe movement on the three-dimensional reconstruction, thereby improving the accuracy of the three-dimensional model.

[0005] Optionally, in an embodiment of the present application, two-dimensional image registration is performed on multiple regional images, including: extracting features from each of the multiple two-dimensional ultrasonic image images to obtain multiple feature points; performing feature matching on the multiple two-dimensional ultrasonic image images based on the multiple feature points to obtain feature matching results; performing spatial transformation estimation on the multiple two-dimensional ultrasonic image images based on the feature matching results to obtain spatial transformation parameters; performing image registration on the multiple regional images based on the spatial transformation parameters to obtain multiple registered regional images. In the implementation of the above scheme, the multiple regional images are registered using spatial transformation parameters estimated by spatial transformation of the multiple two-dimensional ultrasonic image images based on the feature point matching results, so that the registered images are consistent in spatial coordinates, so that ultrasonic images acquired at different time points, different viewing angles or different imaging conditions can correspond to the same anatomical position, thereby effectively improving the image consistency of the multiple two-dimensional ultrasonic image images.

[0006] Optionally, in an embodiment of the present application, the three-dimensional reconstruction of the multiple registered regional images includes: performing three-dimensional local reconstruction on the multiple registered regional images to obtain multiple local models of the target organ; performing three-dimensional registration and fusion on the multiple local models of the target organ to obtain the three-dimensional model of the target organ. In the implementation of the above scheme, the three-dimensional local reconstruction of the multiple registered regional images and the three-dimensional registration and fusion of the multiple local models of the target organ help eliminate image distortion and discontinuity caused by factors such as probe movement and patient breathing, thereby improving the accuracy of the three-dimensional model.

[0007] Optionally, in an embodiment of the present application, a three-dimensional local reconstruction is performed on the multiple registered regional images, including: stacking the multiple registered regional images in the vertical direction to obtain a stacked image of the target organ; converting the stacked image of the target organ from a time series into a three-dimensional space to obtain multiple local models of the target organ. In the implementation process of the above scheme, by stacking the multiple registered regional images in the vertical direction and converting the stacked image of the target organ from a time series into a three-dimensional space, the anatomical information originally scattered on different two-dimensional slices can be integrated together to generate a stereoscopic three-dimensional model of the target organ, and the three-dimensional reconstruction provides a more accurate quantitative evaluation method for the size, volume, shape and internal structure of the target organ, thereby improving the visualization effect of the three-dimensional model.

[0008] Optionally, in an embodiment of the present application, the target organ is the thyroid gland, and the multiple local models of the thyroid gland include: a left lobe model, a right lobe model, and a isthmus model; the multiple local models of the target organ are fused in three dimensions, including: aligning the left lobe model with the isthmus model in three dimensions to obtain a first alignment result; aligning the right lobe model with the isthmus model in three dimensions to obtain a second alignment result; fusing the left lobe model, the right lobe model, and the isthmus model according to the first alignment result and the second alignment result to obtain a three-dimensional model of the target organ. In the implementation process of the above scheme, the left lobe model, the right lobe model, and the isthmus model are fused through the three-dimensional alignment result, so that the originally independent left lobe, right lobe, and isthmus models can be placed in the same coordinate system, thereby accurately reconstructing the complete organ structure, and the registration process will minimize the spatial misalignment between the models, so that the geometric features, boundaries, and internal structures of the local models of each part can be accurately connected, thereby improving the reconstruction accuracy of the three-dimensional model.

[0009] Optionally, in an embodiment of the present application, the left lobe model, the right lobe model and the isthmus model are fused according to the first registration and alignment result and according to the second registration and alignment result, including: fusing the left lobe model with the isthmus model according to the first registration and alignment result to obtain a first fused model; fusing the right lobe model with the isthmus model according to the second registration and alignment result to obtain a second fused model; fusing the first fused model with the second fused model to obtain a three-dimensional model of the target organ. In the implementation of the above scheme, by fusing the first fused model fused with the left lobe model and the isthmus model and the second fused model fused with the right lobe model and the isthmus model, the two registration and fusion processes can gradually reduce the potential errors between the local models, and effectively improve the spatial accuracy of the overall structure of the target organ.

[0010] An embodiment of the present application also provides a three-dimensional reconstruction device, including: a video sequence acquisition module, used to acquire a video sequence of a target organ, the video sequence including multiple two-dimensional ultrasonic images of the target organ; an ultrasonic image segmentation module, used to perform image segmentation on the multiple two-dimensional ultrasonic images to obtain multiple regional images of the target organ; a regional image registration module, used to perform two-dimensional image registration on the multiple regional images to obtain multiple registered regional images; and an organ three-dimensional reconstruction module, used to perform three-dimensional reconstruction on the multiple registered regional images to obtain a three-dimensional model of the target organ.

[0011] Optionally, in an embodiment of the present application, the regional image registration module includes: an image feature extraction submodule, which is used to extract features from each of a plurality of two-dimensional ultrasonic images to obtain a plurality of feature points; a matching result acquisition submodule, which is used to perform feature matching on a plurality of two-dimensional ultrasonic images according to a plurality of feature points to obtain feature matching results; a spatial transformation estimation submodule, which is used to perform spatial transformation estimation on a plurality of two-dimensional ultrasonic images according to the feature matching results to obtain spatial transformation parameters; and an image region registration submodule, which is used to perform image registration on a plurality of region images according to the spatial transformation parameters to obtain a plurality of registered region images.

[0012] Optionally, in an embodiment of the present application, the organ three-dimensional reconstruction module includes: a local model acquisition sub-module, used to perform three-dimensional local reconstruction of the multiple regional images after registration to obtain multiple local models of the target organ; a three-dimensional registration fusion sub-module, used to perform three-dimensional registration fusion of multiple local models of the target organ to obtain a three-dimensional model of the target organ.

[0013] Optionally, in an embodiment of the present application, a local model acquisition submodule includes: a regional image stacking unit, used to stack a plurality of registered regional images in a vertical direction to obtain a stacked image of the target organ; and an image sequence conversion unit, used to convert the stacked image of the target organ from a time series into a three-dimensional space to obtain a plurality of local models of the target organ.

[0014] Optionally, in an embodiment of the present application, the target organ is the thyroid gland, and the multiple local models of the thyroid gland include: a left lobe model, a right lobe model and a isthmus model; a three-dimensional registration and fusion submodule includes: a first registration and alignment unit, used to perform three-dimensional registration and alignment on the left lobe model and the isthmus model to obtain a first registration and alignment result; a second registration and alignment unit, used to perform three-dimensional registration and alignment on the right lobe model and the isthmus model to obtain a second registration and alignment result; a local model fusion unit, used to fuse the left lobe model, the right lobe model and the isthmus model according to the first registration and alignment result and according to the second registration and alignment result to obtain a three-dimensional model of the target organ.

[0015] Optionally, in an embodiment of the present application, the local model fusion unit includes: a first model fusion subunit, used to fuse the left lobe model and the isthmus model according to the first registration and alignment result to obtain a first fusion model; a second model fusion subunit, used to fuse the right lobe model and the isthmus model according to the second registration and alignment result to obtain a second fusion model; and a three-dimensional model acquisition subunit, used to fuse the first fusion model and the second fusion model to obtain a three-dimensional model of the target organ.

[0016] An embodiment of the present application further provides an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions execute the method described above when executed by the processor.

[0017] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is executed.

[0018] The embodiment of the present application also provides a computer program product, including: a computer program or a computer instruction, and the computer program or the computer instruction executes the method described above when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A schematic diagram of a process flow of a three-dimensional reconstruction method provided in an embodiment of the present application is shown;

[0021] Figure 2 A schematic diagram of the process flow of a three-dimensional reconstruction device provided in an embodiment of the present application is shown;

[0022] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. It should be understood that the drawings in the embodiment of the present application only serve the purpose of explanation and description, and are not used to limit the protection scope of the embodiment of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in the embodiment of the present application shows the operation implemented according to some embodiments of the embodiment of the present application. It should be understood that the operation of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the embodiment of the present application, or remove one or more operations from the flowchart.

[0024] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments. The components of the embodiments of the present application described and shown in the drawings generally here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the embodiments of the present application claimed, but merely represents the selected embodiments of the embodiments of the present application.

[0025] It is understandable that the "first" and "second" in the embodiments of the present application are used to distinguish similar objects. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit the difference. In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that there may be three relationships, such as A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship. The term "multiple" refers to more than two (including two), and similarly, "multiple groups" refers to more than two groups (including two groups).

[0026] It should be noted that the 3D reconstruction method provided in the embodiment of the present application can be executed by an electronic device, where the electronic device refers to a device terminal or server with the function of executing a computer program, such as a smart phone, a personal computer, a tablet computer, a personal digital assistant or a mobile Internet device. A server refers to a device that provides computing services through a network, such as an x86 server and a non-x86 server, and a non-x86 server includes a mainframe, a minicomputer and a UNIX server.

[0027] The following is an example of an application scenario in which the three-dimensional reconstruction method is applicable: Since the three-dimensional reconstruction method allows the calculation process of two-dimensional image registration to play the role of coordinate positioning, it effectively reduces the impact of the patient's breathing, heartbeat physiological movement, probe movement tremor, and the positioning device's susceptibility to electromagnetic interference on the three-dimensional reconstruction. The three-dimensional reconstruction method can improve the defect of unstable continuity of the image during processing (such as spatial transformation estimation) due to inaccurate positioning, and improves the stability and continuity of the ultrasound image during processing. Therefore, the three-dimensional reconstruction method can be used to improve or enhance the function of ultrasound machines with three-dimensional reconstruction functions, or use the three-dimensional reconstruction method to improve the reconstruction accuracy of three-dimensional models. These ultrasound machines with three-dimensional reconstruction functions are ultrasound devices capable of real-time or static three-dimensional imaging, volume imaging, and three-dimensional rendering.

[0028] See also Figure 1The flowchart of the three-dimensional reconstruction method provided by the embodiment of the present application is shown; the embodiment of the present application provides a three-dimensional reconstruction method, including:

[0029] Step S110: Acquire a video sequence of the target organ, where the video sequence includes a plurality of two-dimensional ultrasound images of the target organ.

[0030] The target organ refers to the organ that needs to be reconstructed in three dimensions. The organ can be an organ that can be reconstructed in three dimensions with a single continuous or spiral scan, such as the brain, lungs, and pancreas. Of course, some organs are located deeper or larger in size. In order to obtain high-quality ultrasound images, they can be scanned in multiple segments and spliced ​​together to achieve three-dimensional reconstruction. These organs include the thyroid gland, liver, gallbladder, pancreas, or spleen. Taking the thyroid gland as an example, since the ultrasound image area usually cannot fully display the thyroid gland, the scanning process of the thyroid gland can be divided into three scans, scanning the left lobe, right lobe, and isthmus (middle) part of the thyroid gland respectively, to obtain three video sequences of the thyroid gland.

[0031] Step S120: performing image segmentation on the multiple two-dimensional ultrasound images to obtain multiple regional images of the target organ.

[0032] It can be understood that in the process of image segmentation of multiple two-dimensional ultrasonic images, the contour information of the target organ in the two-dimensional ultrasonic image can be analyzed and detected, and then, multiple regional images of the target organ can be segmented from the two-dimensional ultrasonic image along the contour information of the target organ.

[0033] Step S130: performing two-dimensional image registration on the multiple regional images to obtain multiple registered regional images.

[0034] Since the surface of the human body is a soft body, it is unstable during the scanning process. Therefore, the scanned target organs are offset and rotated at different times. Therefore, image registration technology can be used to perform two-dimensional image registration on multiple regional images. The two-dimensional image registration here is to register the two-dimensional images collected in different time series, so that the registered multiple regional images can be more accurate and complete.

[0035] Step S140: Perform three-dimensional reconstruction on the registered multiple regional images to obtain a three-dimensional model of the target organ.

[0036] It can be understood that after the two-dimensional images are registered, the registered multiple regional images can be reconstructed in three dimensions. The three-dimensional reconstruction process is essentially to convert multiple regional images from a time series into a three-dimensional space, thereby completing the three-dimensional reconstruction process of the target organ and obtaining a three-dimensional model of the target organ.

[0037] In the implementation process of the above scheme, image segmentation is performed on multiple two-dimensional ultrasound images in the video sequence, and two-dimensional image registration is performed on multiple regional images of the target organ obtained. Then, three-dimensional reconstruction is performed on the registered multiple regional images, so that the calculation process of the two-dimensional image registration plays a role in coordinate positioning, which effectively reduces the influence of the patient's breathing, heartbeat physiological movement, and probe movement tremor on the three-dimensional reconstruction, thereby improving the accuracy of the obtained three-dimensional model.

[0038] As an optional implementation of the above step S110, the above-mentioned video sequence of the target organ can be obtained by scanning the target organ of the human body in real time through an ultrasonic probe without a positioning device, so as to obtain multiple two-dimensional ultrasonic images of the target organ. The above-mentioned positioning device is a positioning sensor, such as an electromagnetic tracking device or an optical tracking device. In the specific practice, the target organ of the human body can also be scanned in real time through an ultrasonic probe with a positioning device, so as to obtain multiple two-dimensional ultrasonic images and positioning data (such as the position information and posture information of the ultrasonic probe, etc.) of the target organ. These positioning data can be used in the process of image registration (two-dimensional image registration or three-dimensional image registration) to increase the accuracy of image registration. Then, the multiple two-dimensional ultrasonic images of the target organ are stored as a video sequence on the storage medium of the electronic device. Finally, the electronic device can directly read the video sequence of the target organ from the storage medium device.

[0039] Since the target organ can be an organ that can be reconstructed in three dimensions with a single continuous or spiral scan, or it can be an organ that can be reconstructed in three dimensions by scanning multiple segments and stitching them together, for ease of understanding and explanation, only organs that are scanned in multiple segments are used as examples. Assuming that the target organ is the thyroid gland, the person being scanned can lie flat on the bed in a supine position, and the scanner uses a cross-sectional scan, that is, the probe and the trachea of ​​the person being scanned are roughly vertical. Then, the scanner uses an ultrasound probe to scan from top to bottom. It can be seen from multiple two-dimensional ultrasound images that the thyroid gland in the two-dimensional ultrasound image will gradually appear, grow larger, then shrink, and finally disappear. Therefore, these multiple two-dimensional ultrasound images fully record the entire scanning process of the thyroid gland.

[0040] As an optional implementation of the above step S120, a deep neural network model (such as a U-Net network model) is used to detect the contour information of the target organ in the two-dimensional ultrasound image, wherein the U-Net network model includes an encoder and a decoder, the encoder is used to downsample the two-dimensional ultrasound image, so as to capture the image features in the ultrasound image, and the decoder is used to upsample the image features in the ultrasound image, so as to restore the contour information of the target organ. Finally, multiple regional images of the target organ are segmented from the two-dimensional ultrasound image along the contour information of the target organ.

[0041] As an optional implementation of the above step S130, the above implementation of performing two-dimensional image registration on multiple regional images may include:

[0042] Step S131: extracting features from each of the multiple two-dimensional ultrasonic image images to obtain multiple feature points.

[0043] The implementation method of the above step S131 is, for example: using a feature matching algorithm such as a Scale-Invariant Feature Transform (SIFT) algorithm, a SURF algorithm or an ORB algorithm to extract features from each of the multiple two-dimensional ultrasound images to obtain multiple feature points and feature descriptors corresponding to each feature point. Since the SIFT algorithm has scale invariance, the use of the SIFT algorithm can robustly capture key information, so that the corresponding feature points can be effectively found under different scales and rotation conditions.

[0044] Step S132: performing feature matching on the multiple two-dimensional ultrasonic images according to the multiple feature points to obtain feature matching results.

[0045] An implementation example of the above-mentioned step S132 is: feature matching is performed on multiple two-dimensional ultrasonic images based on multiple feature points, that is, multiple feature points in different two-dimensional ultrasonic images among the multiple two-dimensional ultrasonic images are compared. In this process, a matching algorithm can be used to find the correspondence between multiple feature points in different two-dimensional ultrasonic images as the feature matching result.

[0046] Step S133: performing spatial transformation estimation on the multiple two-dimensional ultrasonic images according to the feature matching results to obtain spatial transformation parameters.

[0047] Step S134: performing image registration on the multiple region images according to the spatial transformation parameters to obtain multiple registered region images.

[0048] The implementation method of the above steps S133 to S134 is, for example: using the Random sample consensus (RANSAC) algorithm to estimate the spatial transformation of multiple two-dimensional ultrasound images based on the feature matching results of the correspondence between multiple feature points in different two-dimensional ultrasound images, and obtain spatial transformation parameters that can reflect the spatial transformation relationship between the multiple two-dimensional ultrasound images, so that the multiple two-dimensional ultrasound images are aligned in the same coordinate system. Using the spatial transformation parameters obtained by the spatial transformation estimation, multiple regional images at different times are image registered to obtain multiple registered regional images, where the positions of the target organs in the multiple registered regional images are consistent.

[0049] Optionally, in the spatial transformation estimation process of the above step S133, if an ultrasonic probe with a positioning device is used to scan the target organ of the human body in real time, multiple two-dimensional ultrasonic images and positioning data of the target organ can be obtained. Assuming that the positioning data here is the position information and posture information of the ultrasonic probe, the position information and posture information here can be used as additional constraints, and the position information and posture information can be integrated with the image feature matching results to jointly optimize the spatial transformation parameters between the two-dimensional images, thereby improving the registration accuracy. Specifically, an optimization objective function can be set, which includes both a cost function based on image feature matching (such as normalized cross-correlation, Mahalanobis distance, etc.) and a constraint item generated by the positioning data. Then, the positioning data can be added to the objective function as a hard constraint or a soft constraint. For example, if the positioning data gives an accurate rotation angle and translation vector, it is added to the optimization problem as a hard constraint; if the positioning data has a certain error range, it can be included in the objective function as a regularization item to balance the weight between image feature matching and positioning data. Finally, a nonlinear optimization algorithm (such as Levenberg-Marquardt, Gauss-Newton, etc.) is used to solve the objective function containing image feature matching and positioning data constraints to obtain more accurate spatial transformation parameters.

[0050] As an optional implementation of the above step S140, the implementation of performing three-dimensional reconstruction on the registered multiple regional images may include:

[0051] Step S141: Perform three-dimensional local reconstruction on the registered multiple regional images to obtain multiple local models of the target organ.

[0052] Step S142: Perform three-dimensional registration and fusion on multiple local models of the target organ to obtain a three-dimensional model of the target organ.

[0053] It can be understood that after obtaining multiple local models of the target organ, three-dimensional fusion technology (also known as 3D fusion technology) can be used to fuse information from different sections by setting transparency, color, etc., thereby achieving three-dimensional alignment and fusion of multiple local models of the target organ to obtain a three-dimensional model of the target organ.

[0054] As an optional implementation of the above step S141, the implementation of performing three-dimensional local reconstruction on the registered multiple regional images may include:

[0055] Step S141a: stacking the multiple registered regional images in the vertical direction to obtain a stacked image of the target organ.

[0056] Step S141b: converting the stacked images of the target organ from a time series into a three-dimensional space to obtain multiple local models of the target organ.

[0057] For example, the implementation method of the above steps S141a to S141b is as follows: assuming that the target organ is the thyroid gland, the registered multiple regional images can be stacked in the vertical direction of the thyroid gland to obtain three stacked images of the thyroid gland, where the three stacked images include: a left lobe stacked image, a right lobe stacked image, and a isthmus stacked image. Then, the left lobe stacked image, the right lobe stacked image, and the isthmus stacked image are respectively converted from the time series into three-dimensional space for reconstruction to obtain the left lobe model, the right lobe model, and the isthmus model of the thyroid gland, where the left lobe model, the right lobe model, and the isthmus model of the thyroid gland can be understood as multiple local models of the above target organ, and these local models can be specifically saved in the NIFTI (Neuroimaging Informatics Technology Initiative) format.

[0058] As an optional implementation of the above step S142, the above target organ may be a thyroid gland, and the multiple local models of the thyroid gland may include: a left lobe model, a right lobe model, and an isthmus model; the above implementation of three-dimensional registration and fusion of the multiple local models of the target organ may include:

[0059] Step S142a: Perform three-dimensional registration and alignment on the left lobe model and the isthmus model to obtain a first registration and alignment result.

[0060] The registration and alignment result refers to the result of the three-dimensional registration and alignment, which may specifically be the calculated offsets and rotation angles of the three dimensions in the three-dimensional image.

[0061] The implementation method of the above step S142a is, for example: using the Iterative Closest Point (ICP) algorithm to perform three-dimensional registration and alignment of the left lobe model and the isthmus model. The ICP algorithm optimizes the spatial transformation between the left lobe model and the isthmus model, so that the left lobe model and the isthmus model are aligned in the sense of minimum mean square error, so that the local models acquired under different scanning orientations are consistent in a common reference coordinate system. The above three-dimensional registration and alignment is also called three-dimensional image registration and alignment, which can register the local three-dimensional images reconstructed from multiple scans, and can obtain a more accurate and complete first registration and alignment result of the target organ (such as the offset and rotation angle of three dimensions).

[0062] Step S142b: perform three-dimensional registration and alignment on the right lobe model and the isthmus model to obtain a second registration and alignment result.

[0063] The implementation method of the above step S142a is, for example: using the Iterative Closest Point (ICP) algorithm to perform three-dimensional registration and alignment of the right lobe model and the isthmus model. The ICP algorithm optimizes the spatial transformation between the right lobe model and the isthmus model so that the right lobe model and the isthmus model are aligned in the sense of minimum mean square error, so that the local models acquired under different scanning orientations are consistent in a common reference coordinate system. The above three-dimensional registration and alignment is also called three-dimensional image registration and alignment, which can register the local three-dimensional images reconstructed from multiple scans, and can obtain a more accurate and complete second registration and alignment result of the target organ (such as the offset and rotation angle of three dimensions).

[0064] Step S142c: The left lobe model, the right lobe model and the isthmus model are fused according to the first registration and alignment result and the second registration and alignment result to obtain a three-dimensional model of the target organ.

[0065] It can be understood that since the left lobe model, the right lobe model and the isthmus model are not complete three-dimensional models of the thyroid gland, but three local models, the left lobe model and the right lobe model have overlapping parts with the isthmus model respectively, that is, the left lobe model partially overlaps with the isthmus model, and the right lobe model partially overlaps with the isthmus model. Therefore, in order to obtain a complete three-dimensional model of the thyroid gland, the left lobe model, the right lobe model and the isthmus model can be fused according to the first registration and alignment result and the second registration and alignment result.

[0066] As an optional implementation of the above step S142c, the implementation of fusing the left lobe model, the right lobe model and the isthmus model according to the first registration alignment result and the second registration alignment result may include: fusing the left lobe model with the isthmus model according to the first registration alignment result to obtain a first fusion model. Fusing the right lobe model with the isthmus model according to the second registration alignment result to obtain a second fusion model. Fusing the first fusion model and the second fusion model to obtain a three-dimensional model of the target organ. The above fusion process integrates the registered local models (i.e., the left lobe model, the right lobe model and the isthmus model) to form a complete and seamlessly connected three-dimensional thyroid model. The final three-dimensional thyroid model not only retains the feature information of each local part of the isthmus model, the left lobe model and the right lobe model, but also presents a more consistent and coherent structure as a whole, thereby improving the reconstruction accuracy of the three-dimensional model.

[0067] See also Figure 2 The flowchart of the three-dimensional reconstruction device provided by the embodiment of the present application is shown; the embodiment of the present application provides a three-dimensional reconstruction device 200, including:

[0068] The video sequence acquisition module 210 is used to acquire a video sequence of the target organ, where the video sequence includes a plurality of two-dimensional ultrasonic images of the target organ.

[0069] The ultrasound image segmentation module 220 is used to perform image segmentation on a plurality of two-dimensional ultrasound images to obtain a plurality of regional images of the target organ.

[0070] The regional image registration module 230 is used to perform two-dimensional image registration on multiple regional images to obtain multiple registered regional images.

[0071] The organ three-dimensional reconstruction module 240 is used to perform three-dimensional reconstruction on the registered multiple regional images to obtain a three-dimensional model of the target organ.

[0072] As an optional implementation of the above device, the regional image registration module includes:

[0073] The image feature extraction submodule is used to extract features from each of the multiple two-dimensional ultrasonic image images to obtain multiple feature points.

[0074] The matching result obtaining submodule is used to perform feature matching on multiple two-dimensional ultrasonic images according to multiple feature points to obtain feature matching results.

[0075] The spatial transformation estimation submodule is used to perform spatial transformation estimation on multiple two-dimensional ultrasonic images according to the feature matching results to obtain spatial transformation parameters.

[0076] The image region registration submodule is used to perform image registration on multiple region images according to the spatial transformation parameters to obtain multiple registered region images.

[0077] As an optional implementation of the above device, the organ three-dimensional reconstruction module includes:

[0078] The local model acquisition submodule is used to perform three-dimensional local reconstruction on the multiple registered regional images to obtain multiple local models of the target organ.

[0079] The three-dimensional registration and fusion submodule is used to perform three-dimensional registration and fusion on multiple local models of the target organ to obtain a three-dimensional model of the target organ.

[0080] As an optional implementation of the above device, the local model acquisition submodule includes:

[0081] The regional image stacking unit is used to stack the multiple registered regional images in the vertical direction to obtain a stacked image of the target organ.

[0082] The image sequence conversion unit is used to convert the stacked images of the target organ from a time series into a three-dimensional space to obtain multiple local models of the target organ.

[0083] As an optional embodiment of the above device, the target organ is the thyroid gland, and the multiple local models of the thyroid gland include: a left lobe model, a right lobe model, and an isthmus model; the three-dimensional registration fusion submodule includes:

[0084] The first registration and alignment unit is used to perform three-dimensional registration and alignment on the left lobe model and the isthmus model to obtain a first registration and alignment result.

[0085] The second registration and alignment unit is used to perform three-dimensional registration and alignment on the right lobe model and the isthmus model to obtain a second registration and alignment result.

[0086] The local model fusion unit is used to fuse the left lobe model, the right lobe model and the isthmus model according to the first registration and alignment result and the second registration and alignment result to obtain a three-dimensional model of the target organ.

[0087] As an optional implementation of the above device, the local model fusion unit includes:

[0088] The first model fusion subunit is used to fuse the left lobe model and the isthmus model according to the first registration and alignment result to obtain a first fusion model.

[0089] The second model fusion subunit is used to fuse the right lobe model with the isthmus model according to the second registration and alignment result to obtain a second fusion model.

[0090] The three-dimensional model acquisition subunit is used to fuse the first fusion model and the second fusion model to obtain a three-dimensional model of the target organ.

[0091] It should be understood that the device corresponds to the above-mentioned three-dimensional reconstruction method embodiment and can perform the various steps involved in the above-mentioned method embodiment. The specific functions of the device can be referred to in the above description, and the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system (OS) of the device.

[0092] See also Figure 3 The electronic device 300 provided in the embodiment of the present application includes: a processor 310 and a memory 320, wherein the memory 320 stores machine-readable instructions executable by the processor 310, and when the machine-readable instructions are executed by the processor 310, the above method is executed.

[0093] The embodiment of the present application also provides a computer-readable storage medium 330, on which a computer program is stored, and the computer program is executed by the processor 310 to execute the above method. The computer-readable storage medium 330 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk.

[0094] The embodiment of the present application also provides a computer program product, including: a computer program or a computer instruction, and the computer program or the computer instruction executes the method described above when executed by a processor.

[0095] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0096] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed device and method can also be implemented in other ways. The device embodiments described above are only schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module, a program segment or a code, and a part of a module, a program segment or a code includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also be different from the order of occurrence marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which is mainly determined by the functions involved.

[0097] In addition, each functional module of each embodiment in the embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. In addition, in the description of this specification, the description of reference terms "one embodiment", "some embodiments", "example", "specific example", "some examples", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present application. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples without contradiction.

[0098] The above description is only an optional implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that: include: Acquiring a video sequence of a target organ, wherein the video sequence includes a plurality of two-dimensional ultrasonic images of the target organ; Performing image segmentation on the multiple two-dimensional ultrasonic images to obtain multiple regional images of the target organ; Performing two-dimensional image registration on the multiple regional images to obtain multiple registered regional images; Performing three-dimensional reconstruction on the registered multiple regional images to obtain a three-dimensional model of the target organ; The three-dimensional reconstruction of the registered multiple regional images includes: stacking the registered multiple regional images in a vertical direction to obtain a stacked image of the target organ; converting the stacked image of the target organ from a time series into a three-dimensional space to obtain multiple local models of the target organ; performing three-dimensional registration and fusion on the multiple local models of the target organ to obtain a three-dimensional model of the target organ; The target organ is a thyroid gland, and the multiple local models of the thyroid gland include: a left lobe model, a right lobe model, and an isthmus model; the three-dimensional registration and fusion of the multiple local models of the target organ includes: using an iterative closest point algorithm to perform three-dimensional registration and alignment on the left lobe model and the isthmus model to obtain a first registration and alignment result; using an iterative closest point algorithm to perform three-dimensional registration and alignment on the right lobe model and the isthmus model to obtain a second registration and alignment result; according to the first registration and alignment result and the second registration and alignment result, the left lobe model, the right lobe model, and the isthmus model are fused to obtain the three-dimensional model of the target organ; The fusing of the left lobe model, the right lobe model and the isthmus model according to the first registration alignment result and according to the second registration alignment result includes: fusing the left lobe model with the isthmus model according to the first registration alignment result to obtain a first fused model; fusing the right lobe model with the isthmus model according to the second registration alignment result to obtain a second fused model; and fusing the first fused model with the second fused model to obtain a three-dimensional model of the target organ.

2. The method according to claim 1, characterized in that The performing two-dimensional image registration on the multiple regional images comprises: Extracting features from each of the plurality of two-dimensional ultrasonic images to obtain a plurality of feature points; Performing feature matching on the multiple two-dimensional ultrasonic image images according to the multiple feature points to obtain feature matching results between the multiple feature points in different two-dimensional ultrasonic image images; Using a random sampling consensus algorithm to perform spatial transformation estimation on the multiple two-dimensional ultrasonic image images according to the feature matching result, to obtain spatial transformation parameters reflecting the spatial transformation relationship between the multiple two-dimensional ultrasonic image images; The multiple region images are registered according to the spatial transformation parameters to obtain multiple registered region images.

3. A three-dimensional reconstruction device, characterized in that: include: A video sequence acquisition module, used to acquire a video sequence of a target organ, wherein the video sequence includes a plurality of two-dimensional ultrasonic images of the target organ; An ultrasonic image segmentation module, used to perform image segmentation on the multiple two-dimensional ultrasonic images to obtain multiple regional images of the target organ; A regional image registration module, used for performing two-dimensional image registration on the multiple regional images to obtain multiple registered regional images; An organ three-dimensional reconstruction module, used for performing three-dimensional reconstruction on the registered multiple regional images to obtain a three-dimensional model of the target organ; The three-dimensional reconstruction of the registered multiple regional images includes: stacking the registered multiple regional images in a vertical direction to obtain a stacked image of the target organ; converting the stacked image of the target organ from a time series into a three-dimensional space to obtain multiple local models of the target organ; performing three-dimensional registration and fusion on the multiple local models of the target organ to obtain a three-dimensional model of the target organ; The target organ is a thyroid gland, and the multiple local models of the thyroid gland include: a left lobe model, a right lobe model, and an isthmus model; the three-dimensional registration and fusion of the multiple local models of the target organ includes: using an iterative closest point algorithm to perform three-dimensional registration and alignment on the left lobe model and the isthmus model to obtain a first registration and alignment result; using an iterative closest point algorithm to perform three-dimensional registration and alignment on the right lobe model and the isthmus model to obtain a second registration and alignment result; according to the first registration and alignment result and the second registration and alignment result, the left lobe model, the right lobe model, and the isthmus model are fused to obtain the three-dimensional model of the target organ; The fusing of the left lobe model, the right lobe model and the isthmus model according to the first registration alignment result and according to the second registration alignment result includes: fusing the left lobe model with the isthmus model according to the first registration alignment result to obtain a first fused model; fusing the right lobe model with the isthmus model according to the second registration alignment result to obtain a second fused model; and fusing the first fused model with the second fused model to obtain a three-dimensional model of the target organ.

4. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions are executed by the processor to perform the method according to any one of claims 1 to 2.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 2 is executed.

6. A computer program product, characterized in that include: A computer program or a computer instruction, wherein when the computer program or the computer instruction is executed by a processor, the method according to any one of claims 1 to 2 is executed.

Citation Information

Patent Citations

  • Method for automatically generating ultrasonic report by voice input thyroid ultrasonic anomaly description

    CN110765274A

  • Incomplete detection method and device

    CN110910379A

  • Point cloud registration method, system and device based on multi-source fusion and storage medium

    CN116152310A

  • Three-dimensional ultrasonic imaging processing method and system, electronic equipment and readable storage medium

    CN117197346A

  • ROI selection in image registration

    US20060002631A1