Visual three-dimensional reconstruction method and system for ultrasonic medical image

Abstract

The invention discloses a visual three-dimensional reconstruction method and system for an ultrasonic medical image, and belongs to the technical field of ultrasonic medicine. The method comprises the steps that 1, a two-dimensional ultrasonic image is collected; 2, preprocessing the two-dimensional ultrasonic image; step 3, searching feature points; step 4, motion frame supplementation; 5, reconstructing a three-dimensional ultrasonic model; and 6, post-processing the three-dimensional ultrasonic model. The method is novel and effective, two-dimensional ultrasonic image feature points are extracted by using the convolutional neural network and are matched, and inter-frames are obtained through optical flow field correction for frame compensation, so that the precision of subsequent three-dimensional model reconstruction is improved, and the probability of errors or blurring of model reconstruction is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of ultrasonic medicine, and in particular is a method and system for visualizing three-dimensional reconstruction of ultrasonic medical images. Background Art

[0002] Ultrasound imaging is a technology that uses ultrasound to image the internal tissues of the human body. It transmits ultrasound into the human body, then receives the reflected ultrasound signal, and processes it according to the time, intensity and other information of the signal to obtain the image of the human body. This imaging method has the advantages of repeatability, convenience, non-radiation and low price, so it has been widely used in clinical diagnosis. Traditional two-dimensional ultrasound imaging can only collect information of a very small range of sections of the examination site each time. Doctors need to infer the three-dimensional overall information of tissues, organs or lesions based on these two-dimensional local information, and give disease assessment results and treatment plans accordingly. However, due to the complex structure, diverse morphology and individual differences of human tissues and organs, even experienced doctors may find it difficult to make a comprehensive and accurate analysis based on two-dimensional ultrasound images alone.

[0003] In order to solve the limitations of two-dimensional ultrasound imaging, three-dimensional ultrasound imaging technology came into being. Three-dimensional ultrasound imaging technology can intuitively display the three-dimensional morphology and structural details of human organs, which helps to observe the details of lesions more comprehensively. This technology not only improves the accuracy and efficiency of diagnosis, but also reduces the mental burden of doctors. Based on three-dimensional ultrasound imaging technology, a visual three-dimensional reconstruction method was proposed. This method obtains two-dimensional ultrasound images of the scanning part at different acquisition angles, and then uses advanced image processing algorithms and computer graphics technology to fuse these two-dimensional images into a three-dimensional model. This three-dimensional model can intuitively display the three-dimensional structure and lesion details of human organs, providing doctors with more accurate and comprehensive diagnostic information.

[0004] Although the three-dimensional ultrasound model reconstruction method has made great progress, there are still some defects. Since the frame rate of two-dimensional ultrasound images is low and it is difficult to extract the feature points of the cross-section of the human body structure, it is easy to cause model blur when using the pixel method to reconstruct the three-dimensional model. Therefore, it is necessary to propose a method and system for visual three-dimensional reconstruction of ultrasound medical images that can improve the frame rate of two-dimensional ultrasound images by calculation and use artificial intelligence to extract the feature points of two-dimensional ultrasound images for matching. Summary of the invention

[0005] In order to solve the above problems, the purpose of the present invention is to provide a method and system for visual three-dimensional reconstruction of ultrasonic medical images, which uses a convolutional neural network to extract and match feature points of two-dimensional ultrasonic images, and uses optical flow field correction to obtain interframes for frame interpolation, thereby improving the accuracy of subsequent three-dimensional model reconstruction and reducing the probability of errors or blurs in model reconstruction.

[0006] In order to achieve the above object, the technical solution of the present invention is as follows: a method for visualizing three-dimensional reconstruction of ultrasonic medical images, comprising the following steps:

[0007] Step 1: two-dimensional ultrasound image acquisition, using a handheld ultrasound image probe equipped with a tilt sensor and an acceleration sensor to acquire a set of two-dimensional ultrasound images with motion labels of the lesion area in a manner of 180° tilt scanning, 360° rotation scanning, and linear translation scanning of the lesion part;

[0008] Step 2: preprocessing the two-dimensional ultrasound image to remove noise and enhance the contrast of the two-dimensional ultrasound image;

[0009] Step 3: Feature point search: Use the convolutional neural network trained with two-dimensional ultrasound image data to extract and match the feature points in the two-dimensional ultrasound image set to obtain the feature point pairs between the image sets. Use the random sampling consistency strategy to screen the feature point pairs according to their robustness to obtain the feature point essential matrix, thereby eliminating the mismatched feature point pairs.

[0010] Step 4: motion frame interpolation: input a set of two-dimensional ultrasound images with motion labels and an optical flow field between ultrasound image frames, repair the optical flow field between frames according to the motion sensor data to obtain several two-dimensional ultrasound image inter-frames and add them into the set of two-dimensional ultrasound images;

[0011] Step 5: Reconstructing the three-dimensional ultrasound model, using the function method to incrementally reconstruct the three-dimensional ultrasound model, starting with the essential matrix of the feature points extracted in step 3 and the two two-dimensional ultrasound images of the initial x-axis and y-axis scans, and continuously adding other angles of the two-dimensional ultrasound image set after the frame supplementation to finely reconstruct the three-dimensional ultrasound model;

[0012] Step six, post-processing of the three-dimensional ultrasound model, image segmentation of the reconstructed three-dimensional ultrasound model is performed by extracting the feature point matrix to separate and mark the lesions, and the ray casting algorithm is used to superimpose transparency and color on the model to achieve realistic visualization of the model.

[0013] The principle of the basic scheme is: by configuring a handheld ultrasound image probe with a tilt sensor and an acceleration sensor, the motion information (such as tilt angle, rotation angle, and translation distance) during image acquisition can be accurately recorded, which provides key motion labels for subsequent image matching and three-dimensional reconstruction. Using a convolutional neural network trained with two-dimensional ultrasound image data, representative feature points can be efficiently extracted from the image, and the corresponding relationship between images can be found through a matching algorithm. The random sampling consistency strategy is used to eliminate mismatched feature point pairs to ensure the accuracy of the feature point essential matrix. Based on the restoration of motion sensor data and inter-frame optical flow fields, the time or space gaps in the image sequence can be filled, and more intermediate frames can be generated, thereby increasing the data density of three-dimensional reconstruction and improving the accuracy of the model. Using the function method for incremental reconstruction, starting from the initial two two-dimensional images, images from other angles are gradually added, and the feature point essential matrix is ​​used for precise alignment, and finally a complete and detailed three-dimensional ultrasound model is formed.

[0014] The beneficial effects of the basic scheme are: 1. By using convolutional neural networks (CNN) for feature point extraction and matching, it can more accurately identify and match key feature points in two-dimensional ultrasound images compared to traditional methods. This high-precision feature point matching provides a solid foundation for the accurate reconstruction of three-dimensional models and effectively reduces the three-dimensional model errors caused by feature point mismatching. The introduction of optical flow field correction technology further improves the coherence and consistency of images between frames. By repairing the optical flow field between frames and supplementing the missing inter-frames, the data in the three-dimensional reconstruction process is more complete, thereby significantly improving the accuracy and reliability of the reconstructed model.

[0015] 2. The application of ray casting algorithm makes the 3D model more realistic and delicate in visual effect. Doctors can intuitively observe the shape, size, position and spatial relationship of the lesion with other tissue structures, which provides strong support for accurate diagnosis. The lesion area is separated and marked from the 3D model through image segmentation technology, which not only helps doctors to quickly locate the lesion, but also provides important reference information in the subsequent treatment plan formulation, further improving the accuracy and practicality of diagnosis.

[0016] 3. The visualization 3D reconstruction method and system of the present invention realizes an automated and intelligent processing flow, greatly reducing the need for manual intervention, thereby improving the efficiency and speed of diagnosis. Doctors can focus more on the analysis and diagnosis of the 3D model, reducing the tedious image processing and model reconstruction work. In addition, by providing an intuitive and easy-to-understand 3D visualization model, doctors can make diagnostic decisions more quickly, reducing the risk of misdiagnosis and missed diagnosis, and also reducing the workload of doctors.

[0017] Furthermore, the motion tag in step 1 is the tilt and acceleration state data of the ultrasound imaging probe when capturing the current two-dimensional ultrasound image.

[0018] The benefits of the basic solution are: 1. The motion tag contains the precise motion information of the probe when acquiring images, such as tilt angle, rotation speed, and acceleration changes. This information is crucial for subsequent alignment between images and reconstruction of 3D models. By using this motion data, the relative position relationship between images can be determined more accurately, thereby reducing 3D model errors caused by inaccurate image alignment.

[0019] 2. In the process of feature point search and matching, motion tags can be used as additional constraints to help the algorithm more accurately match feature points between images. Especially when there are large differences or noise interference between images, motion tags can provide additional information to support the correct matching of feature points, thereby improving the robustness and accuracy of matching.

[0020] 3. Motion tags can also be used to optimize the optical flow correction and interpolation process. By comparing the difference between the actual motion data and the optical flow estimation results, the inter-frame optical flow field can be repaired more accurately and more accurate inter-frames can be generated. This helps reduce interpolation errors caused by inaccurate optical flow estimation, thereby improving the continuity and integrity of the 3D model.

[0021] Furthermore, the image denoising method in step 2 includes spatial domain filtering and frequency domain filtering, and the contrast enhancement method includes grayscale transformation, histogram correction and image sharpening processing.

[0022] The beneficial effects of the basic scheme are: 1. By directly analyzing the relationship between image pixels and their neighboring pixels, filters or convolution kernels are applied to smooth the image, effectively removing random noise in the image, such as salt and pepper noise, Gaussian noise, etc. Some spatial domain filtering methods, such as median filtering and edge-preserving filtering, can retain image details and edge information as much as possible while removing noise, avoiding image blur.

[0023] 2. Convert the image from the spatial domain to the frequency domain, where noise and image signals are considered as different frequency components. By suppressing high-frequency noise components, low-frequency image signals can be retained to achieve the purpose of denoising. The frequency domain filtering method can flexibly select the filter type and parameters according to the noise type and image characteristics to achieve the best denoising effect.

[0024] 3. By adjusting the grayscale range of the image, the contrast of the image is expanded, the image is clearer, and the features are more obvious. Grayscale transformation can highlight the details in the image, allowing doctors to observe the morphology and structure of the lesion area more clearly.

[0025] 4. By redistributing the pixel values ​​of the image, the histogram of the image is made more uniform, thereby improving the contrast and clarity of the image. Histogram correction can make the visual effect of the image more vivid and bright, and improve the image's viewing and readability.

[0026] 5. By enhancing the edge and detail information of the image, the image contour is clearer and the lesion area is more prominent. Sharpening can improve the resolution of the image, allowing doctors to more accurately judge the size, location and shape of the lesion.

[0027] Furthermore, the two-dimensional ultrasound image data used to train the convolutional neural network in step three includes typical two-dimensional ultrasound image data of different anatomical structures and different disease types.

[0028] The benefits of the basic scheme are: 1. By including image data of different anatomical structures and disease types, the network can learn more extensive and diverse features. This enables the network to more accurately extract and identify key information when faced with new and unknown ultrasound images.

[0029] 2. Diverse training data helps the network adapt to various image changes and noise interference, thereby improving its robustness. In practical applications, ultrasound images may be affected by factors such as equipment, operators, and patients, and fully trained CNNs can better cope with these challenges.

[0030] A visualized three-dimensional reconstruction system for ultrasonic medical images includes an image acquisition module, an image preprocessing module, a feature point search module, an image frame supplementation module, a model reconstruction module and a model post-processing module.

[0031] An image acquisition module, used to acquire two-dimensional ultrasonic images at different angles and corresponding tilt and acceleration data;

[0032] An image preprocessing module, used for performing denoising and contrast enhancement processing on the acquired two-dimensional ultrasound image;

[0033] A feature point search module is used to extract and match feature point pairs in a two-dimensional ultrasound image using a convolutional neural network, and to remove mismatched feature points to obtain a feature point essential matrix;

[0034] An image frame interpolation module is used to input tilt and acceleration data and an optical flow field between two-dimensional ultrasound image frames, repair the frames between two-dimensional ultrasound images, and insert them into a two-dimensional ultrasound image set in time sequence;

[0035] A model reconstruction module, used for incrementally reconstructing a three-dimensional ultrasound model using a function method;

[0036] The model post-processing module is used to perform image segmentation and labeling on the reconstructed three-dimensional ultrasound model and to superimpose transparency and color on the model.

[0037] The benefits of the basic solution are: 1. By acquiring two-dimensional ultrasound images at different angles and the corresponding tilt and acceleration data, it provides a rich information basis for subsequent three-dimensional reconstruction. This multi-angle acquisition method helps capture more anatomical structures and lesion details, thereby improving the accuracy of diagnosis.

[0038] 2. De-noising and contrast enhancement of the acquired two-dimensional ultrasound images can significantly improve the visual quality of the images. De-noising can reduce noise interference in the image and make the image clearer; while contrast enhancement can highlight the key features in the image, making it easier for doctors to identify the lesion area.

[0039] 3. Use convolutional neural networks to extract and match feature point pairs in two-dimensional ultrasound images, remove mismatched feature points, and obtain the feature point essential matrix. This process helps to establish the correspondence between images and provide accurate reference for subsequent three-dimensional reconstruction.

[0040] 4. By inputting the tilt and acceleration data and the optical flow field between the 2D ultrasound image frames, the frames between the 2D ultrasound images are repaired and inserted into the 2D ultrasound image set in chronological order. This step can fill the missing frames in the image sequence, making the image sequence more complete and continuous, thereby improving the accuracy and smoothness of 3D reconstruction.

[0041] 5. Use the function method to incrementally reconstruct the three-dimensional ultrasound model. This method can gradually build a three-dimensional model based on the two-dimensional ultrasound image sequence, with high reconstruction efficiency and accuracy. At the same time, the incremental reconstruction method can also flexibly adjust the reconstruction accuracy and range according to actual needs.

[0042] 6. Perform image segmentation and labeling on the reconstructed 3D ultrasound model, and add transparency and color to the model. This step can make the 3D model more intuitive and easy to understand, helping doctors to better observe and analyze the structure and morphology of the lesion area. At the same time, by adjusting parameters such as transparency and color, doctors can also highlight specific anatomical structures or lesion areas as needed.

[0043] Furthermore, the image acquisition module includes a handheld ultrasonic image probe, a tilt sensor and an acceleration sensor.

[0044] A handheld ultrasound imaging probe used to acquire two-dimensional ultrasound images of lesions in the patient's body;

[0045] A tilt sensor, used to collect tilt state data of a handheld ultrasonic imaging probe;

[0046] The acceleration sensor is used to collect acceleration state data of the handheld ultrasonic imaging probe.

[0047] The beneficial effects of the basic scheme are: 1. By directly contacting the lesions in the patient's body, high-quality two-dimensional ultrasound images can be acquired in real time and accurately. This direct acquisition method reduces distortion and errors in the image transmission and conversion process, and improves the accuracy and reliability of the image. The tilt sensor can accurately record the tilt angle and tilt direction of the probe when acquiring the image. This data is crucial for subsequent three-dimensional reconstruction because it provides information on the spatial relationship between images, which helps to ensure that the reconstructed three-dimensional model is consistent with the actual situation. The acceleration sensor can monitor the acceleration changes of the probe in real time, including the magnitude and direction of the speed and acceleration. This data helps to identify the motion trajectory and speed changes of the probe during the acquisition process, and provides important reference information for subsequent image alignment and three-dimensional reconstruction.

[0048] 2. By combining tilt and acceleration data with 2D ultrasound images, the relative position relationship between images can be determined more accurately, thereby reducing 3D model errors caused by inaccurate image alignment. This helps to improve the accuracy and reliability of 3D reconstruction. Tilt and acceleration data provide additional constraints for image alignment, which helps to speed up image alignment and improve the efficiency of 3D reconstruction. This is especially important for real-time or quasi-real-time 3D reconstruction applications.

[0049] 3. The handheld ultrasound image probe enables doctors to collect images more conveniently without being restricted by fixed equipment. This design improves the applicability and flexibility of the system, allowing doctors to perform ultrasound examinations in different clinical environments. The tilt and acceleration sensors can collect data in real time and record it synchronously with the ultrasound image. This allows doctors to instantly obtain the motion information of the probe, which helps to better control the image acquisition process and improve the user experience.

[0050] Furthermore, the image preprocessing module includes a denoising unit and an image enhancement unit.

[0051] A denoising unit, used for filtering and denoising the two-dimensional ultrasonic image;

[0052] The image enhancement unit is used to enhance the contrast of the two-dimensional ultrasound image.

[0053] The beneficial effects of the basic scheme are: 1. By filtering and denoising the two-dimensional ultrasound image, the noise components in the image can be significantly reduced. These noises may come from the equipment itself, environmental interference, or irregular movements in the patient's body. Denoising can make the image clearer, reduce interference information, and help doctors observe and analyze the lesions more accurately. The image enhancement unit enhances the contrast of the two-dimensional ultrasound image, which helps to highlight the key information in the image. In ultrasound images, the grayscale values ​​of different tissues and lesion areas may be similar, resulting in blurred image details. By enhancing the contrast, the grayscale differences between these areas can be made more obvious, thereby improving the recognition and readability of the image.

[0054] 2. In the subsequent steps of feature point search, image interpolation and model reconstruction, accurate image information is needed. Denoising and enhancement processing can improve image quality, thereby improving the accuracy and reliability of these steps. High-quality images can reduce the computational complexity in subsequent processing. For example, in the process of feature point search, clear images can make it easier to find accurate feature points, thereby reducing search time and computing resources.

[0055] Furthermore, the feature point search module includes a convolutional neural network and a feature point matching unit.

[0056] Convolutional neural network, used to extract and pair feature points in two-dimensional ultrasound images;

[0057] The feature point matching unit is used to screen and obtain the feature point essential matrix and eliminate the wrongly matched feature point pairs.

[0058] The beneficial effects of the basic scheme are: 1. Using the powerful ability of deep learning, the convolutional neural network can automatically learn and extract feature points in two-dimensional ultrasound images. These feature points are usually the identifiers of key structures or lesion areas in the image, which are crucial for subsequent three-dimensional reconstruction. Through the precise extraction of the convolutional neural network, the accuracy and representativeness of the feature points can be ensured. After the feature points are extracted, the feature point matching unit is responsible for matching these feature points between different images. By calculating the similarity or distance between the feature points, the correct matching pairs can be screened out and the mismatched feature point pairs can be eliminated. This process helps to ensure the accuracy and stability of subsequent three-dimensional reconstruction.

[0059] 2. Accurate feature point extraction and matching are the basis of 3D reconstruction. The feature point search module ensures that the feature points used in the 3D reconstruction process are accurate and reliable. This helps improve the precision and accuracy of the 3D model, allowing doctors to more accurately observe and analyze lesions in patients. During the feature point search process, the convolutional neural network and feature point matching unit can efficiently process large amounts of image data. By optimizing algorithms and accelerating calculations, the computational complexity and time cost of the 3D reconstruction process can be significantly reduced.

[0060] 3. Convolutional neural networks have strong adaptive capabilities and can process two-dimensional ultrasound images of different qualities, different angles, and different lighting conditions. This enables the feature point search module to operate stably in various clinical environments and provide reliable diagnostic support for doctors. During the feature point extraction and matching process, the feature point search module can automatically remove mismatched feature point pairs. This helps reduce mismatches caused by factors such as image noise, interference, or irregular movement in the patient's body, thereby improving the system's anti-interference ability and stability.

[0061] Furthermore, the image frame interpolation module includes an inter-frame correction unit and a sequential frame interpolation unit.

[0062] An inter-frame correction unit, used for correcting the inter-frame according to the optical flow field between the two-dimensional ultrasound image frames and the related motion sensor data;

[0063] The sequential frame insertion unit is used to insert the corrected inter-frames into the two-dimensional ultrasound image set in time sequence.

[0064] The beneficial effects of the basic scheme are: 1. The inter-frame correction unit captures dynamic changes in the image sequence by analyzing the optical flow field between two-dimensional ultrasound image frames, that is, the trend and speed of pixel movement in the image. At the same time, combined with motion sensor data (such as data provided by accelerometers and tilt sensors), the inter-frame correction unit can more accurately predict and correct the transition state between image frames. This correction helps to generate more natural and coherent inter-frames, thereby reducing visual jumps and freezes caused by insufficient frame rate.

[0065] 2. The sequential interpolation unit inserts the corrected interframes into the original 2D ultrasound image set in chronological order, thereby improving the frame rate. The inserted interframes can smoothly connect adjacent original frames, making the entire image sequence look more fluent and coherent.

[0066] Furthermore, the model post-processing module includes an image segmentation unit and a volume rendering unit.

[0067] An image segmentation unit, used for image segmentation and marking of lesion areas and feature areas in a three-dimensional ultrasound reconstruction model;

[0068] The volume rendering unit is used to superimpose transparency and color on the three-dimensional ultrasound reconstruction model.

[0069] The beneficial effects of the basic solution are: 1. The image segmentation unit can automatically identify and separate the lesion area and feature area in the 3D ultrasound reconstruction model. This precise separation helps doctors to more clearly identify and understand the morphology, location and size of the lesion, thereby improving the accuracy of diagnosis. By marking the separated areas, the image segmentation unit can provide doctors with intuitive visual guidance to help them quickly locate and analyze key information.

[0070] 2. The volume rendering unit makes the 3D ultrasound reconstruction model present a more three-dimensional and realistic visual effect by superimposing transparency and color. This effect helps doctors better perceive and understand the structure and details in the model. Volume rendering technology allows doctors to observe the 3D model from multiple angles and perspectives, so as to have a more comprehensive understanding of the morphology and location of the lesion. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 Schematic diagram of a method for visualizing three-dimensional reconstruction of ultrasonic medical images in an embodiment of the present invention.

[0072] Figure 2 Schematic diagram of a visualized three-dimensional reconstruction system for ultrasonic medical images in an embodiment of the present invention.

[0073] Figure 3 Schematic diagram of an image acquisition module of a visualized three-dimensional reconstruction system for ultrasonic medical images in an embodiment of the present invention.

[0074] Figure 4 Schematic diagram of an image preprocessing module of a visualized three-dimensional reconstruction system for ultrasonic medical images in an embodiment of the present invention.

[0075] Figure 5 Schematic diagram of a feature point search module of a visualized three-dimensional reconstruction system for ultrasonic medical images in an embodiment of the present invention.

[0076] Figure 6 Schematic diagram of an image frame interpolation module of a visualized three-dimensional reconstruction system for ultrasonic medical images in an embodiment of the present invention.

[0077] Figure 7 Schematic diagram of a post-processing module of a system model for visualizing three-dimensional reconstruction of ultrasonic medical images in an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The following is further described in detail through specific implementation methods:

[0079] Example 1

[0080] Basically as attached Figure 1 As shown: A method for visual three-dimensional reconstruction of ultrasonic medical images, comprising the following steps:

[0081] Step 1: two-dimensional ultrasound image acquisition, using a handheld ultrasound image probe equipped with a tilt sensor and an acceleration sensor, to acquire a set of two-dimensional ultrasound images with motion tags of the lesion area in a manner of 180° tilt scanning, 360° rotation scanning, and linear translation scanning of the lesion part, where the motion tag is the tilt and acceleration state data of the ultrasound image probe when taking the current two-dimensional ultrasound image;

[0082] Step 2: preprocessing the two-dimensional ultrasound image to denoise the two-dimensional ultrasound image and enhance the contrast. The image denoising method includes spatial domain filtering and frequency domain filtering. The contrast enhancement method includes grayscale transformation, histogram correction and image sharpening processing.

[0083] Step 3: feature point search: use the convolutional neural network trained with two-dimensional ultrasound image data to extract and match the feature points in the two-dimensional ultrasound image set to obtain the feature point pairs between the image sets, and use the random sampling consistency strategy to obtain the feature point essential matrix according to the robustness of the feature point pairs, so as to eliminate the mismatched feature point pairs. The two-dimensional ultrasound image data used to train the convolutional neural network include typical two-dimensional ultrasound image data of different anatomical structures and different disease types;

[0084] Step 4: motion frame interpolation: input a set of two-dimensional ultrasound images with motion labels and an optical flow field between ultrasound image frames, repair the optical flow field between frames according to the motion sensor data to obtain several two-dimensional ultrasound image inter-frames and add them into the set of two-dimensional ultrasound images;

[0085] Step 5: Reconstructing the three-dimensional ultrasound model, using the function method to incrementally reconstruct the three-dimensional ultrasound model, starting with the essential matrix of the feature points extracted in step 3 and the two two-dimensional ultrasound images of the initial x-axis and y-axis scans, and continuously adding other angles of the two-dimensional ultrasound image set after the frame supplementation to finely reconstruct the three-dimensional ultrasound model;

[0086] Step six, post-processing of the three-dimensional ultrasound model, image segmentation of the reconstructed three-dimensional ultrasound model is performed by extracting the feature point matrix to separate and mark the lesions, and the ray casting algorithm is used to superimpose transparency and color on the model to achieve realistic visualization of the model.

[0087] Example 2

[0088] The difference from the above embodiment is that, as shown in the attached Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 and Figure 7Shown: A visualized three-dimensional reconstruction system for ultrasonic medical images, including an image acquisition module, an image preprocessing module, a feature point search module, an image frame supplementation module, a model reconstruction module and a model post-processing module.

[0089] The image acquisition module is used to acquire two-dimensional ultrasonic images at different angles and the corresponding tilt and acceleration data. The image acquisition module includes a handheld ultrasonic image probe, a tilt sensor and an acceleration sensor. The handheld ultrasonic image probe is used to acquire two-dimensional ultrasonic images of lesions in the patient's body; the tilt sensor is used to acquire the tilt state data of the handheld ultrasonic image probe; and the acceleration sensor is used to acquire the acceleration state data of the handheld ultrasonic image probe.

[0090] The image preprocessing module is used to perform denoising and contrast enhancement on the collected two-dimensional ultrasonic image. The image preprocessing module includes a denoising unit and an image enhancement unit. The denoising unit is used to filter and denoise the two-dimensional ultrasonic image; the image enhancement unit is used to enhance the contrast of the two-dimensional ultrasonic image.

[0091] The feature point search module is used to use a convolutional neural network to extract and match feature point pairs in a two-dimensional ultrasonic image, and eliminate mismatched feature points to obtain a feature point essential matrix. The feature point search module includes a convolutional neural network and a feature point matching unit. The convolutional neural network is used to extract feature points in a two-dimensional ultrasonic image and pair them; the feature point matching unit is used to screen and obtain a feature point essential matrix and eliminate mismatched feature point pairs.

[0092] The image frame interpolation module is used to input tilt and acceleration data and the optical flow field between two-dimensional ultrasonic image frames, repair the two-dimensional ultrasonic image interframes and insert them into the two-dimensional ultrasonic image set in chronological order. The image frame interpolation module includes an interframe correction unit and a sequential frame insertion unit. The interframe correction unit is used to correct the interframes according to the optical flow field between two-dimensional ultrasonic image frames and related motion sensor data; the sequential frame insertion unit is used to insert the corrected interframes into the two-dimensional ultrasonic image set in chronological order.

[0093] A model reconstruction module, used for incrementally reconstructing a three-dimensional ultrasound model using a function method;

[0094] The model post-processing module is used to perform image segmentation and labeling on the reconstructed three-dimensional ultrasound model and to superimpose transparency and color on the model. The model post-processing module includes an image segmentation unit and a volume rendering unit. The image segmentation unit is used to perform image segmentation and labeling on the lesion area and feature area in the three-dimensional ultrasound reconstruction model; the volume rendering unit is used to superimpose transparency and color on the three-dimensional ultrasound reconstruction model.

[0095] The specific implementation process is as follows: a handheld ultrasound image probe is used to collect a two-dimensional ultrasound image of the lesion in the patient's body. At the same time, the tilt sensor and the acceleration sensor collect the tilt and acceleration state data of the probe. For example, the above three methods are used to detect the possible lesion site of the patient, and the motion sensor will record the motion state of the detection probe. Then the denoising unit performs filtering and denoising on the two-dimensional ultrasound image to reduce noise interference, and the image enhancement unit enhances the contrast of the image and improves the clarity of the image. The trained convolutional neural network extracts the feature points in the two-dimensional ultrasound image and pairs them. The shape and position of the lesions caused by different diseases are different. According to the anatomical model comparison, the detection position of the current two-dimensional ultrasound image can be determined by extracting the image feature points. The combination with the lesion feature points can be preliminarily compared with the disease type. The feature point matching unit screens the feature point essential matrix and eliminates the mismatched feature point pairs to improve the accuracy of subsequent three-dimensional reconstruction. The inter-frame correction unit corrects the inter-frame according to the optical flow field between the two-dimensional ultrasound image frames and the related motion sensor data, and the sequential interpolation unit inserts the corrected inter-frame into the two-dimensional ultrasound image set in chronological order to improve the smoothness and coherence of the image. The three-dimensional ultrasound model is incrementally reconstructed using the function method to obtain the three-dimensional structure of the lesion and surrounding tissue. The image segmentation unit performs image segmentation and marking on the lesion area and feature area in the three-dimensional ultrasound reconstruction model, which facilitates the doctor's judgment and identification of the lesion condition during subsequent diagnosis. The volume rendering unit superimposes transparency and color on the model to generate realistic three-dimensional visualization effects. For example, the color and transparency of the rendered lesion are used to show the real lesion morphology. Doctors can judge the severity of the patient's condition by the size of the rendered color lesion, thereby improving diagnostic efficiency and model expressiveness.

[0096] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0097] The above is only an embodiment of the present invention. The common sense such as the known specific structure and characteristics in the scheme is not described in detail here. The ordinary technicians in the relevant field know all the common technical knowledge in the technical field of the invention before the application date or priority date, can obtain all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the protection scope of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent. The protection scope required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A method for visualizing three-dimensional reconstruction of ultrasonic medical images, characterized in that: The following steps are included: Step 1: two-dimensional ultrasound image acquisition, using a handheld ultrasound image probe equipped with a tilt sensor and an acceleration sensor to acquire a set of two-dimensional ultrasound images with motion labels of the lesion area in a manner of 180° tilt scanning, 360° rotation scanning, and linear translation scanning of the lesion part; Step 2: preprocessing the two-dimensional ultrasound image to remove noise and enhance the contrast of the two-dimensional ultrasound image; Step 3: Feature point search: Use the convolutional neural network trained with two-dimensional ultrasound image data to extract and match the feature points in the two-dimensional ultrasound image set to obtain the feature point pairs between the image sets. Use the random sampling consistency strategy to screen the feature point pairs according to their robustness to obtain the feature point essential matrix, thereby eliminating the mismatched feature point pairs. Step 4: motion frame interpolation: input a set of two-dimensional ultrasound images with motion labels and an optical flow field between ultrasound image frames, repair the optical flow field between frames according to the motion sensor data to obtain several two-dimensional ultrasound image inter-frames and add them into the set of two-dimensional ultrasound images; Step 5: Reconstructing the three-dimensional ultrasound model, using the function method to incrementally reconstruct the three-dimensional ultrasound model, starting with the essential matrix of the feature points extracted in step 3 and the two two-dimensional ultrasound images of the initial x-axis and y-axis scans, and continuously adding other angles of the two-dimensional ultrasound image set after the frame supplementation to finely reconstruct the three-dimensional ultrasound model; Step six, post-processing of the three-dimensional ultrasound model, image segmentation of the reconstructed three-dimensional ultrasound model is performed by extracting the feature point matrix to separate and mark the lesions, and the ray casting algorithm is used to superimpose transparency and color on the model to achieve realistic visualization of the model.

2. The method for visualizing three-dimensional reconstruction of ultrasonic medical images according to claim 1, characterized in that: The motion tag in step 1 is the tilt and acceleration state data of the ultrasound imaging probe when capturing the current two-dimensional ultrasound image.

3. The method for visualizing three-dimensional reconstruction of ultrasonic medical images according to claim 1, characterized in that: The image denoising method in step 2 includes spatial domain filtering and frequency domain filtering, and the contrast enhancement method includes grayscale transformation, histogram correction and image sharpening processing.

4. The method for visualizing three-dimensional reconstruction of ultrasonic medical images according to claim 1, characterized in that: The two-dimensional ultrasound image data used to train the convolutional neural network in step three includes typical two-dimensional ultrasound image data of different anatomical structures and different disease types.

5. A visualization 3D reconstruction system for ultrasonic medical images, characterized in that: It includes image acquisition module, image preprocessing module, feature point search module, image frame filling module, model reconstruction module and model post-processing module. An image acquisition module, used to acquire two-dimensional ultrasonic images at different angles and corresponding tilt and acceleration data; An image preprocessing module, used for performing denoising and contrast enhancement processing on the acquired two-dimensional ultrasound image; A feature point search module is used to extract and match feature point pairs in a two-dimensional ultrasound image using a convolutional neural network, and to remove mismatched feature points to obtain a feature point essential matrix; An image frame interpolation module is used to input tilt and acceleration data and an optical flow field between two-dimensional ultrasound image frames, repair the frames between two-dimensional ultrasound images, and insert them into a two-dimensional ultrasound image set in time sequence; A model reconstruction module, used for incrementally reconstructing a three-dimensional ultrasound model using a function method; The model post-processing module is used to perform image segmentation and labeling on the reconstructed three-dimensional ultrasound model and to superimpose transparency and color on the model.

6. The visualization 3D reconstruction system of ultrasonic medical images according to claim 5, characterized in that: The image acquisition module includes a handheld ultrasonic image probe, a tilt sensor and an acceleration sensor. A handheld ultrasound imaging probe used to acquire two-dimensional ultrasound images of lesions in the patient's body; A tilt sensor, used to collect tilt state data of a handheld ultrasonic imaging probe; The acceleration sensor is used to collect acceleration state data of the handheld ultrasonic imaging probe.

7. The visualization 3D reconstruction system of ultrasonic medical images according to claim 5, characterized in that: The image preprocessing module includes a denoising unit and an image enhancement unit. A denoising unit, used for filtering and denoising the two-dimensional ultrasonic image; The image enhancement unit is used to enhance the contrast of the two-dimensional ultrasound image.

8. The visualization 3D reconstruction system of ultrasonic medical images according to claim 5, characterized in that: The feature point search module includes a convolutional neural network and a feature point matching unit. Convolutional neural network, used to extract and pair feature points in two-dimensional ultrasound images; The feature point matching unit is used to screen and obtain the feature point essential matrix and eliminate the wrongly matched feature point pairs.

9. The visualization 3D reconstruction system of ultrasonic medical images according to claim 5, characterized in that: The image interpolation module includes an inter-frame correction unit and a sequential interpolation unit. An inter-frame correction unit, used for correcting the inter-frame according to the optical flow field between the two-dimensional ultrasound image frames and the related motion sensor data; The sequential frame insertion unit is used to insert the corrected inter-frames into the two-dimensional ultrasound image set in time sequence.

10. The visualization 3D reconstruction system of ultrasonic medical images according to claim 5, characterized in that: The model post-processing module includes an image segmentation unit and a volume rendering unit. An image segmentation unit, used for image segmentation and marking of lesion areas and feature areas in a three-dimensional ultrasound reconstruction model; The volume rendering unit is used to superimpose transparency and color on the three-dimensional ultrasound reconstruction model.

Images