Three-dimensional reconstruction method and system based on ultrasonic image

Abstract

The invention provides a three-dimensional reconstruction method and system based on an ultrasonic image. The method comprises the following steps: acquiring an ultrasonic two-dimensional image generated in an ultrasonic scanning process; obtaining pose information corresponding to the ultrasonic two-dimensional image; determining a nearest neighbor sampled voxel grid corresponding to a target voxel grid from the plurality of sampled voxel grids, and assigning the target voxel grid according to a voxel value of the nearest neighbor sampled voxel grid to obtain an assigned target voxel grid, the sampled voxel grid is a voxel grid which is endowed with the voxel value, the target voxel grid is a voxel grid which is not endowed with the voxel value, and the voxel value is obtained according to a pixel value in the ultrasonic two-dimensional image and the pose information; and generating a corresponding three-dimensional ultrasonic volume image according to the sampled voxel grid and the assigned target voxel grid. According to the invention, the high-quality three-dimensional reconstruction image can be generated in real time in the scanning process.

Description

Technical Field

[0001] The present invention relates to the field of ultrasonic imaging technology, and in particular, to a three-dimensional reconstruction method and system based on ultrasonic images. Background Art

[0002] Ultrasonic imaging technology in the field of three-dimensional imaging is a method that has attracted much attention and is developing rapidly. This technology can provide a detailed visualization and depiction of the human body interior in the form of a three-dimensional volume.

[0003] In existing free-arm three-dimensional ultrasonic imaging systems, three-dimensional reconstruction is a key step. Two-dimensional ultrasonic image data is obtained through a scanning device, and then algorithms such as interpolation, segmentation, and registration are used to process this data in two-dimensional or three-dimensional space to generate a three-dimensional surface or three-dimensional rendered volume of the target object and the region of interest (ROI).

[0004] However, existing free-arm three-dimensional ultrasonic imaging technology separates the three steps of scanning, three-dimensional reconstruction, and visualization, resulting in the inability to provide necessary interaction information in actual operation. For example, some systems may perform reconstruction and display only after all scanning data has been collected; while other systems may perform acquisition and reconstruction during scanning, but do not perform display until scanning is completed. This non-real-time processing method causes the inability to grasp imaging information in real time during scanning, and thus it is impossible to interactively adjust scanning parameters in real time according to actual needs to obtain a complete three-dimensional image. Therefore, there is an urgent need for a three-dimensional reconstruction method and system based on ultrasonic images to solve the above problems. Summary of the Invention

[0005] In view of the problems existing in the prior art, the present invention provides a three-dimensional reconstruction method and system based on ultrasonic images.

[0006] The present invention provides a three-dimensional reconstruction method based on ultrasonic images, comprising: Obtaining ultrasonic two-dimensional images generated during an ultrasonic scanning process; Obtaining pose information corresponding to the ultrasonic two-dimensional images; Determining a nearest-neighbor sampled voxel grid corresponding to a target voxel grid from a plurality of sampled voxel grids, and assigning a value to the target voxel grid according to the voxel value of the nearest-neighbor sampled voxel grid, so as to obtain an assigned target voxel grid, wherein the sampled voxel grid is a voxel grid to which the voxel value has been assigned, the target voxel grid is a voxel grid to which the voxel value has not been assigned, and the voxel value is obtained according to the pixel value in the ultrasonic two-dimensional image and the pose information; Generating a corresponding three-dimensional ultrasonic volume image according to the sampled voxel grid and the assigned target voxel grid.

[0007] A three-dimensional reconstruction method based on ultrasonic images provided by the present invention, obtaining the pose information corresponding to the ultrasonic two-dimensional image includes: During the ultrasonic scanning process, based on a binocular camera, obtaining six-degree-of-freedom pose information corresponding to each frame of the ultrasonic two-dimensional image; The method further includes: Based on the six-degree-of-freedom pose information, mapping the pixel values in the ultrasonic two-dimensional image as voxel values into the corresponding voxel grid in three-dimensional space to obtain the sampled voxel grid.

[0008] A three-dimensional reconstruction method based on ultrasonic images provided by the present invention, determining the nearest-neighbor sampled voxel grid corresponding to the target voxel grid from multiple sampled voxel grids includes: Based on the nearest-neighbor voxel search method, obtaining the Euclidean distance between each of the sampled voxel grids and the target voxel grid, and taking the sampled voxel grid with the smallest Euclidean distance to the target voxel grid as the nearest-neighbor sampled voxel grid.

[0009] A three-dimensional reconstruction method based on ultrasonic images provided by the present invention, generating a corresponding three-dimensional ultrasonic volume image according to the sampled voxel grid and the assigned target voxel grid includes: Based on the sampled voxel grids and the assigned target voxel grids corresponding to the ultrasonic two-dimensional images of the currently scanned area, generating real-time three-dimensional volume data corresponding to the currently scanned area; After determining that the ultrasonic scanning process is completed, generating the three-dimensional ultrasonic volume image corresponding to all scanned areas according to the sampled voxel grids and the assigned target voxel grids corresponding to all the ultrasonic two-dimensional images.

[0010] A three-dimensional reconstruction method based on ultrasonic images provided by the present invention, the method further includes: Receiving a first input, wherein the first input includes an instruction for adjusting the display orientation of the three-dimensional reconstruction image of the real-time three-dimensional volume data or the three-dimensional ultrasonic volume image; In response to the first input, adjusting the current display orientation of the real-time three-dimensional volume data or the three-dimensional ultrasonic volume image to a target display orientation, wherein the target display orientation includes the coronal plane, the sagittal plane, and the axial plane.

[0011] A three-dimensional reconstruction method based on ultrasonic images provided by the present invention, after obtaining the ultrasonic two-dimensional images generated during the ultrasonic scanning process, the method further includes: Perform image preprocessing on the two-dimensional ultrasound image to obtain the preprocessed two-dimensional ultrasound image, where the image preprocessing at least includes denoising, smoothing, image enhancement, and image correction.

[0012] The present invention also provides a three-dimensional reconstruction system based on ultrasound images, including: An ultrasound scanning module for acquiring two-dimensional ultrasound images generated during ultrasound scanning; An optical positioning module for acquiring the pose information corresponding to the two-dimensional ultrasound image; A three-dimensional reconstruction module for determining the nearest neighbor sampled voxel grid corresponding to the target voxel grid from multiple sampled voxel grids, and assigning a value to the target voxel grid according to the voxel value of the nearest neighbor sampled voxel grid to obtain the assigned target voxel grid, where the sampled voxel grid is a voxel grid to which the voxel value has been assigned, the target voxel grid is a voxel grid to which the voxel value has not been assigned, and the voxel value is obtained according to the pixel value in the two-dimensional ultrasound image and the pose information; A three-dimensional reconstruction display module for generating a corresponding three-dimensional ultrasound volume image according to the sampled voxel grid and the assigned target voxel grid.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the three-dimensional reconstruction method based on ultrasound images as described in any one of the above.

[0014] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the three-dimensional reconstruction method based on ultrasound images as described in any one of the above.

[0015] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the three-dimensional reconstruction method based on ultrasound images as described in any one of the above.

[0016] The three-dimensional reconstruction method and system based on ultrasound images provided by the present invention obtain the two-dimensional image of ultrasound scanning and its corresponding pose information in real time, assign corresponding voxel values to the voxel grids in the three-dimensional space, and use the voxel values of the nearest neighbor sampled voxel grids to assign values to the unassigned target voxel grids in the three-dimensional space, thereby dynamically constructing a three-dimensional ultrasound volume image and realizing the real-time generation of high-quality three-dimensional reconstruction images during the scanning process. Description of the Drawings

[0017] To more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0018] Figure 1 It is a schematic flowchart of the 3D reconstruction method based on ultrasonic images provided by the present invention; Figure 2 It is a schematic structural diagram of the 3D reconstruction system based on ultrasonic images provided by the present invention; Figure 3 It is a schematic structural diagram of the electronic device provided by the present invention. Detailed embodiments

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention in conjunction with the drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0020] The 3D imaging methods used in existing 3D ultrasonic imaging systems mainly include direct 3D ultrasonic imaging methods (using a 3D volume array probe for one-time imaging) and reconstructed 3D ultrasonic imaging methods. Among them, the reconstructed 3D ultrasonic imaging methods specifically include robotic arm 3D ultrasonic imaging, handheld mechanical scanning 3D ultrasonic imaging, and free-arm 3D ultrasonic imaging with a positioning device.

[0021] During ultrasonic scanning, the 3D anatomical structure visualization technology can be presented on the user interface, without relying on the operator to mentally convert the 2D B-mode scan into a 3D image, thereby reducing the dependence on operation interpretation during data acquisition and simultaneously lowering the technical threshold. Compared with the use of 2D ultrasonic probes, which often require the operator to have profound knowledge of human anatomy, existing 3D ultrasonic scanning systems simplify the acquisition process and reduce the requirement for user professionalism. For the monitoring of regions of interest, in 2D ultrasonic imaging, due to registration errors and difficult probe positioning, it is quite challenging to achieve this goal. However, the application of 3D ultrasonic imaging technology significantly improves the feasibility of such evaluations and the accuracy of the results. Moreover, the multimodal registration of the 3D module of ultrasonic data can be combined with the output results of other imaging modalities (such as MRI), making it more efficient and valuable than image fusion in the 2D field.

[0022] 3D ultrasound scanning technology is not limited by viewing angles, allowing you to freely view planes in any direction in 3D space, including views that are difficult to capture with traditional 2D ultrasound technology. This feature allows professionals to discuss, review and describe conditions from different angles even when they leave the inspection site, and share data with other experts to mine potential information and optimize the quality of the assessment.

[0023] In the existing 3D ultrasound program system, the hardware price of the direct 3D ultrasound imaging system is relatively expensive and is not suitable for 3D ultrasound imaging of large areas. The free-arm 3D ultrasound imaging system has lower cost, higher volume quality and wider field of view. During the ultrasound scanning process, users need to obtain as detailed information as possible to obtain more accurate test results. Therefore, users hope to know which parts of the test area have been scanned and which parts need to be scanned while scanning, so that they can interactively scan based on this information to obtain a more complete 3D image of the test area.

[0024] However, the existing 3D ultrasound imaging system separates scanning, 3D reconstruction and visualization, or reconstructs and displays after all scanning data are acquired, or acquires and reconstructs at the same time as scanning, and displays after scanning is completed, resulting in the inability to provide necessary interactive information. Users can only see the results of 3D scanning imaging after 3D reconstruction is completed and displayed, but cannot grasp the scanning imaging information in real time during the 3D scanning process, and thus cannot obtain complete 3D images in real time and interactively according to actual needs. In addition, it is difficult and sometimes even cumbersome to obtain 3D ultrasound volumes with existing free-arm 3D ultrasound imaging technology, which is caused by errors in positioning sensors, low calibration accuracy, and many delays in each step of the reconstruction algorithm.

[0025] In view of the problems existing in existing three-dimensional ultrasound imaging, the present invention provides a free-arm three-dimensional ultrasound imaging method, which can further reduce costs, make operation easier, and improve imaging accuracy. At the same time, the present invention can use more common two-dimensional B-ultrasound probes (such as palm ultrasound devices and cloud ultrasound devices) to perform three-dimensional reconstruction through algorithms, which can greatly reduce the cost of ultrasound systems.

[0026] Figure 1 A schematic diagram of the process flow of the three-dimensional reconstruction method based on ultrasound images provided by the present invention is shown in FIG. Figure 1 As shown, the present invention provides a three-dimensional reconstruction method based on ultrasound images, comprising: Step 101: Acquire an ultrasonic two-dimensional image generated during an ultrasonic scanning process.

[0027] In the present invention, an ultrasonic device is used to scan the area to be detected, thereby obtaining a two-dimensional ultrasonic image of the area to be detected. Among them, the ultrasonic device can be a palm ultrasonic device or a cloud ultrasonic device. As a portable ultrasonic instrument, the palm ultrasonic device has the characteristics of small size and easy to carry, and is suitable for on-site detection in various environments. It integrates an ultrasonic emission and reception system and image processing technology, and can perform instant two-dimensional ultrasonic examinations in the hands of the operator. The cloud ultrasonic device integrates cloud computing and Internet of Things technologies on the basis of traditional ultrasonic devices, and can realize remote transmission, storage and analysis of ultrasonic data. It can not only provide high-quality ultrasonic examinations, but also support cloud management of data.

[0028] In the present invention, short-pulse ultrasonic waves are emitted towards the area to be detected through a two-dimensional B-ultrasound probe on a palm ultrasonic device or a cloud ultrasonic device. These ultrasonic beams penetrate the area to be detected (such as the skin and tissues), and when they encounter interfaces with different densities or structural changes (such as organ boundaries, blood flow, etc.), they will be reflected. The reflected ultrasonic waves (i.e., echoes) are received by the probe and converted into electrical signals, which contain details about the tissue structure and dynamic information. The received electrical signals go through high-precision signal processing algorithms inside the ultrasonic device, including steps such as amplification, filtering, digitization, and image reconstruction, and finally are converted into a visualized two-dimensional ultrasonic image.

[0029] Step 102, obtain the pose information corresponding to the two-dimensional ultrasonic image.

[0030] In the present invention, the existing ultrasonic device is improved. By equipping the ultrasonic device with an optical positioning device, the pose information of the two-dimensional ultrasonic image obtained by the current scan can be obtained while the ultrasonic device is scanning.

[0031] Optionally, in the present invention, the optical positioning device includes one or several groups of optical markers (such as light-emitting diodes LED or reflective markers), one or more cameras (such as binocular cameras), and corresponding image processing and positioning algorithms. Among them, the optical markers are fixed at specific positions of the ultrasonic probe. The light emitted or reflected by these optical markers within the field of view of the camera can be accurately captured. For example, the optical positioning device outputs the pose information (such as six-degree-of-freedom pose information) of the two-dimensional ultrasonic image by using a binocular camera to track the reflective markers on the ultrasonic device. In one embodiment, a two-dimensional code can also be marked on the ultrasonic device, and then the position can be obtained by using the camera on a mobile terminal (such as a mobile phone, a tablet computer, etc.) to track the two-dimensional code on the ultrasonic device.

[0032] Specifically, before starting the ultrasound scan, it is necessary to initialize the optical positioning device, including calibrating the camera, determining the reference coordinate system, and ensuring that all optical markers are clearly visible. During the ultrasound scan, the camera on the optical positioning device continuously captures images of the optical markers. Since the ultrasound probe moves during the scan, the position of the markers in the camera's field of view also changes accordingly.

[0033] Furthermore, the optical positioning device analyzes the captured images of the optical markers, identifies the positions of the optical markers, and calculates the three-dimensional pose of the ultrasound probe through an algorithm. Then, the calculated pose information is associated with the simultaneously acquired two-dimensional ultrasound image, so that each frame of the two-dimensional ultrasound image is marked with its corresponding spatial position and orientation information.

[0034] Step 103: Determine the nearest-neighbor sampled voxel grid corresponding to the target voxel grid from multiple sampled voxel grids, and assign a value to the target voxel grid according to the voxel value of the nearest-neighbor sampled voxel grid, so as to obtain the target voxel grid after assignment. Herein, the sampled voxel grid is a voxel grid to which the voxel value has been assigned, the target voxel grid is a voxel grid to which the voxel value has not been assigned, and the voxel value is obtained according to the pixel value in the two-dimensional ultrasound image and the pose information.

[0035] A voxel grid is a three-dimensional space divided into regular cubic units (voxels), and each voxel stores a value (such as a grayscale value, a density value). In the present invention, the voxels filled with values in the three-dimensional space through the two-dimensional ultrasound image and its pose information (position and orientation) represent the scanned or processed areas. For the areas that have not been directly scanned or assigned values, their values need to be inferred through an algorithm.

[0036] Each two-dimensional ultrasound image contains pixel values, which need to be mapped to the three-dimensional space through the pose information (sensor position, angle). In the present invention, first, according to the pose information (such as a translation matrix and a rotation matrix), the pixel points in the two-dimensional ultrasound image are converted into three-dimensional space coordinates. Then, the pixel values are directly assigned to the corresponding three-dimensional coordinate voxels, or the voxel values are generated by interpolating / fusing the data of multiple images. In the present invention, it is necessary to find the nearest sampled voxel in the space for the unassigned voxels (target voxels). The Euclidean distance from the target voxel to all sampled voxels can be calculated, and the nearest sampled voxel is selected. Then, the value of the nearest neighbor voxel is directly assigned to the target voxel.

[0037] Step 104: Generate a corresponding three-dimensional ultrasound volume image according to the sampled voxel grid and the target voxel grid after assignment.

[0038] In the present invention, a three-dimensional ultrasound volume image is a three-dimensional data set containing a large amount of voxel data, and these data sets can reflect the three-dimensional structure and acoustic characteristics inside the region to be detected. During the generation process of the three-dimensional ultrasound volume image, the pixel values of the ultrasound two-dimensional image are mapped to the voxel grid in the three-dimensional space, and after using the Voxel Nearest Neighbor (VNN) method to assign values to the voxel grid that has not been directly assigned voxel values (i.e., the target voxel grid), the pixel values of the ultrasound two-dimensional image are mapped to the three-dimensional coordinate system in real time, and then visualization techniques (such as volume rendering, surface rendering, etc.) are used to convert these data into a visualized three-dimensional image, obtaining the three-dimensional ultrasound volume image corresponding to the currently scanned ultrasound two-dimensional image.

[0039] The three-dimensional reconstruction method based on ultrasound images provided by the present invention dynamically constructs a three-dimensional ultrasound volume image by obtaining the two-dimensional image of the ultrasound scan and its corresponding pose information in real time, assigning corresponding voxel values to the voxel grid in the three-dimensional space, and using the voxel values of the sampled voxel grids closest to the target voxel grid without assigned values in the three-dimensional space to assign values, thereby realizing the real-time generation of high-quality three-dimensional reconstruction images during the scanning process.

[0040] Based on the above embodiments, the obtaining of the pose information corresponding to the ultrasound two-dimensional image includes: During the ultrasound scanning process, based on a binocular camera, obtaining the six-degree-of-freedom pose information corresponding to each frame of the ultrasound two-dimensional image; The method further includes: Based on the six-degree-of-freedom pose information, mapping the pixel values in the ultrasound two-dimensional image as the voxel values to the corresponding voxel grid in the three-dimensional space to obtain the sampled voxel grid.

[0041] In the present invention, a binocular camera captures two images of the same scene simultaneously, thereby calculating the depth information of the scene. The six-degree-of-freedom pose information describes the position and orientation of an object (i.e., each frame of the ultrasound two-dimensional image) in the three-dimensional space. Among them, the position includes the translation amounts of the ultrasound two-dimensional image in the X-axis, Y-axis, and Z-axis directions, and the orientation includes the rotation amounts (pitch, yaw, and roll) around these three axes. During the ultrasound scanning process, the six-degree-of-freedom pose information corresponding to each frame of the ultrasound two-dimensional image reflects the position and orientation of the probe relative to a certain reference coordinate system.

[0042] During the ultrasonic scanning process, a binocular camera is used to capture the position and attitude changes of the ultrasonic probe. By processing the image pair captured by the binocular camera, the six-degree-of-freedom pose information of the probe at the time of each frame of ultrasonic two-dimensional image shooting can be calculated, so as to accurately calculate the position and attitude of the probe at the time of each frame of image shooting. Further, the pixel values in the ultrasonic two-dimensional image are mapped onto a voxel grid in three-dimensional space. The mapping process involves converting each pixel in the ultrasonic two-dimensional image to a corresponding voxel grid in three-dimensional space according to its position and depth information in the image (provided by the binocular camera). After the mapping is completed, each pixel value in the ultrasonic two-dimensional image is used as the voxel value of the corresponding voxel in the voxel grid, and these voxel values reflect the acoustic characteristics (such as echo intensity) in the ultrasonic image. As the scanning process progresses, each frame of ultrasonic two-dimensional image is converted and mapped onto the voxel grid in three-dimensional space, and these voxel grids gradually accumulate to form a complete three-dimensional data set, namely the sampled voxel grid. The sampled voxel grid contains detailed three-dimensional information of the organizational structure within the scanning area and can be used for further three-dimensional visualization, analysis, and processing.

[0043] Based on the above embodiments, determining the nearest neighbor sampled voxel grid corresponding to the target voxel grid from multiple sampled voxel grids includes: Based on the nearest neighbor voxel search method, obtain the Euclidean distance between each of the sampled voxel grids and the target voxel grid, and use the sampled voxel grid with the smallest Euclidean distance to the target voxel grid as the nearest neighbor sampled voxel grid.

[0044] In the present invention, the obtained ultrasonic two-dimensional slice image is associated with corresponding spatial position information, that is, including the coordinates and directions of each frame. When performing three-dimensional reconstruction, first establish a regular voxel grid in three-dimensional space, and set the initial value of each voxel grid to "undefined" or a default value.

[0045] Then, according to the pose information of the ultrasonic two-dimensional slice image, map the two-dimensional pixel values into the corresponding voxel grids, and directly assign the pixel values on the ultrasonic two-dimensional slice image to the sampled voxel grids. Further, traverse all undefined voxel grids (i.e., target voxel grids), and assign a reasonable value to each target voxel grid. The present invention adopts the nearest neighbor voxel search method. For each target voxel grid, search for the voxel grid with the smallest Euclidean distance to the target voxel grid in the sampled voxel set, and assign the value of this sampled voxel grid to the target voxel grid.

[0046] Specifically, in the present invention, let v target be the target voxel grid, v iFor any voxel grid in the set of sampled voxels S The Euclidean distance is a measure of the straight-line distance between two points (here, the voxel grids) in three-dimensional space. For each target voxel grid v target , traverse all the voxel grids S in the set of sampled voxels v i , and calculate the Euclidean distance between the target voxel grid and the sampled voxel grids. At the same time, record the minimum value among all the calculated Euclidean distances, and use the voxel grid v source corresponding to the minimum distance as the nearest-neighbor sampled voxel grid of the target voxel grid v target .

[0047] Furthermore, assign the voxel value of the nearest-neighbor sampled voxel v source to the target voxel grid v target , and the formula is: ; wherein, represents the Euclidean distance between the target voxel grid v target and the voxel grid v source .

[0048] Repeat the above search and assignment process, traverse all the undefined voxel grids until all the voxels with undefined values are assigned a corresponding voxel value. Finally, when the values of all the voxel grids are determined, a complete three-dimensional volume data is obtained, which can be used for further three-dimensional visualization, analysis, or processing.

[0049] The present invention uses two-dimensional ultrasound images for three-dimensional reconstruction. Compared with the existing three-dimensional volume probes, it can reduce the cost of ultrasound equipment and replace the existing high-cost positioning system with a low-cost visual positioning system; at the same time, compared with other deep learning algorithms, the VNN algorithm does not require training, has a fast algorithm speed, and has low hardware requirements.

[0050] On the basis of the above embodiments, generating a corresponding three-dimensional ultrasound volume image according to the sampled voxel grid and the assigned target voxel grid includes: Generating real-time three-dimensional volume data corresponding to the currently scanned area based on the sampled voxel grid and the assigned target voxel grid corresponding to each of the two-dimensional ultrasound images of the currently scanned area; After determining that the ultrasound scanning process is completed, a three-dimensional ultrasound volume image corresponding to all the scanned areas is generated according to all the two-dimensional ultrasound images corresponding to the sampled voxel grid and the assigned target voxel grid.

[0051] In the present invention, an ultrasound probe emits ultrasonic waves and receives the reflected signals, which are converted into a series of two-dimensional ultrasound images. During the ultrasound scanning process, as the probe moves, a real-time three-dimensional volume data can be generated based on the sampled voxel grid and the assigned target voxel grid corresponding to each two-dimensional ultrasound image of the currently scanned area. This three-dimensional volume data contains the positions and attributes of all voxels within the currently scanned area, that is, before the ultrasound scanning is completed, a real-time three-dimensional volume data can be constructed according to the two-dimensional ultrasound image obtained from the current frame and the two-dimensional ultrasound images scanned from the previous frames.

[0052] When the ultrasound probe finishes scanning the entire scanning area, all the two-dimensional ultrasound images of the area to be detected are collected. Then, a complete three-dimensional ultrasound volume image can be generated according to the sampled voxel grid and the assigned target voxel grid corresponding to all the two-dimensional ultrasound images.

[0053] Based on the above embodiments, the method further includes: Receiving a first input, where the first input includes an instruction for adjusting the display orientation of the three-dimensional reconstruction image of the real-time three-dimensional volume data or the three-dimensional ultrasound volume image; In response to the first input, adjusting the current display orientation of the real-time three-dimensional volume data or the three-dimensional ultrasound volume image to a target display orientation, where the target display orientation includes the coronal plane, the sagittal plane, and the axial plane.

[0054] In the present invention, the information or instruction provided by the user operation interface (such as a touch screen, a keyboard, a mouse, or other input devices) of the ultrasound device, that is, the first input, is used to adjust the display orientation of the three-dimensional reconstruction image. The first input contains clear instructions on how to adjust the current display orientation of the three-dimensional image. For example, direct orientation names (such as the coronal plane, the sagittal plane, the axial plane), or instructions to reach the target orientation through a series of operations (such as rotation, translation, scaling, etc.).

[0055] Furthermore, according to the above instruction, the current display orientation of the real-time three-dimensional volume data or the three-dimensional ultrasound volume image is adjusted, and the adjustment process includes operations such as rotation, translation, or scaling of the image, so that the image can be displayed according to the target display orientation, where the display orientation includes the coronal plane, the sagittal plane, and the axial plane.

[0056] Based on the above embodiments, after obtaining the two-dimensional ultrasound image generated during the ultrasound scanning process, the method further includes: Performing image preprocessing on the two-dimensional ultrasound image to obtain a preprocessed two-dimensional ultrasound image, where the image preprocessing at least includes denoising processing, smoothing processing, image enhancement processing, and image correction processing.

[0057] In the present invention, the two-dimensional ultrasound image is preprocessed to improve the accuracy and efficiency of the three-dimensional reconstruction result. Specifically, when performing ultrasound image scanning, it often contains noise, which may come from the device itself, environmental factors, or internal human body movements. These noises are removed through denoising processing to improve the clarity and readability of the image. At the same time, through smoothing processing, small particles and irregular edges in the two-dimensional ultrasound image are removed to make the image clearer.

[0058] In the present invention, in order to improve attributes such as the contrast, brightness, and color saturation of the image, image enhancement processing is performed on the two-dimensional ultrasound image, which can help users more easily identify the lesion sites and abnormal structures in the area to be detected. And image correction processing for geometric distortion and color distortion in the two-dimensional ultrasound image can ensure the accuracy and consistency of the image.

[0059] Next, the three-dimensional reconstruction system based on ultrasound images provided by the present invention will be described. The three-dimensional reconstruction system based on ultrasound images described below can be mutually referred to corresponding to the three-dimensional reconstruction method based on ultrasound images described above.

[0060] Figure 2 is a schematic structural diagram of the three-dimensional reconstruction system based on ultrasound images provided by the present invention, as Figure 2 shown, the present invention provides a three-dimensional reconstruction system based on ultrasound images, including an ultrasound scanning module 201, an optical positioning module 202, a three-dimensional reconstruction module 203, and a three-dimensional reconstruction display module 204. Among them, the ultrasound scanning module 201 is used to obtain the two-dimensional ultrasound image generated during the ultrasound scanning process; the optical positioning module 202 is used to obtain the pose information corresponding to the two-dimensional ultrasound image; the three-dimensional reconstruction module 203 is used to determine the nearest neighbor sampled voxel grid corresponding to the target voxel grid from multiple sampled voxel grids, and assign a value to the target voxel grid according to the voxel value of the nearest neighbor sampled voxel grid to obtain an assigned target voxel grid, where the sampled voxel grid is a voxel grid that has been assigned the voxel value, the target voxel grid is a voxel grid that has not been assigned the voxel value, and the voxel value is obtained according to the pixel value in the two-dimensional ultrasound image and the pose information; the three-dimensional reconstruction display module 204 is used to generate a corresponding three-dimensional ultrasound volume image according to the sampled voxel grid and the assigned target voxel grid.

[0061] The 3D reconstruction system based on ultrasonic images provided by the present invention dynamically constructs a 3D ultrasonic volume image by acquiring in real time the two-dimensional images obtained from ultrasonic scans and their corresponding pose information, assigning corresponding voxel values to the voxel grids in the three-dimensional space, and using the voxel values of the sampled voxel grids in the nearest neighborhood to assign values to the target voxel grids in the three-dimensional space that have not been assigned values, thereby realizing the real-time generation of high-quality 3D reconstruction images during the scanning process.

[0062] The system provided by the embodiments of the present invention is used to execute the above-mentioned method embodiments. For the specific processes and detailed contents, please refer to the above embodiments and will not be elaborated here.

[0063] Figure 3 The following is a schematic structural diagram of the electronic device provided by the present invention. As Figure 3 shown, the electronic device may include: a processor (Processor) 301, a communication interface (Communications Interface) 302, a memory (Memory) 303, and a communication bus 304. Among them, the processor 301, the communication interface 302, and the memory 303 complete the communication with each other through the communication bus 304. The processor 301 may call the logic instructions in the memory 303 to execute the 3D reconstruction method based on ultrasonic images, and the method includes: acquiring the ultrasonic two-dimensional images generated during the ultrasonic scanning process; acquiring the pose information corresponding to the ultrasonic two-dimensional images; determining the nearest-neighbor sampled voxel grid corresponding to the target voxel grid from a plurality of sampled voxel grids, and assigning a value to the target voxel grid according to the voxel value of the nearest-neighbor sampled voxel grid to obtain the target voxel grid after value assignment, where the sampled voxel grid is the voxel grid to which the voxel value has been assigned, the target voxel grid is the voxel grid to which the voxel value has not been assigned, and the voxel value is obtained according to the pixel value in the ultrasonic two-dimensional image and the pose information; generating a corresponding 3D ultrasonic volume image according to the sampled voxel grid and the target voxel grid after value assignment.

[0064] In addition, when the logical instructions in the above-mentioned memory 303 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0065] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the three-dimensional reconstruction method based on ultrasonic images provided by the above-mentioned various methods. The method includes: obtaining ultrasonic two-dimensional images generated during an ultrasonic scanning process; obtaining pose information corresponding to the ultrasonic two-dimensional images; determining a nearest-neighbor sampled voxel grid corresponding to a target voxel grid from a plurality of sampled voxel grids, and assigning a value to the target voxel grid according to the voxel value of the nearest-neighbor sampled voxel grid to obtain an assigned target voxel grid, where the sampled voxel grid is a voxel grid to which the voxel value has been assigned, the target voxel grid is a voxel grid to which the voxel value has not been assigned, and the voxel value is obtained according to the pixel value in the ultrasonic two-dimensional image and the pose information; generating a corresponding three-dimensional ultrasonic volume image according to the sampled voxel grid and the assigned target voxel grid.

[0066] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is configured to execute the three-dimensional reconstruction method based on ultrasonic images provided in the above various embodiments. The method includes: acquiring ultrasonic two-dimensional images generated during an ultrasonic scanning process; acquiring pose information corresponding to the ultrasonic two-dimensional images; determining a nearest-neighbor sampled voxel grid corresponding to a target voxel grid from a plurality of sampled voxel grids, and assigning a value to the target voxel grid according to the voxel value of the nearest-neighbor sampled voxel grid, so as to obtain an assigned target voxel grid, where the sampled voxel grid is a voxel grid to which the voxel value has been assigned, the target voxel grid is a voxel grid to which the voxel value has not been assigned, and the voxel value is obtained according to the pixel value in the ultrasonic two-dimensional image and the pose information; generating a corresponding three-dimensional ultrasonic volume image according to the sampled voxel grid and the assigned target voxel grid.

[0067] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative efforts.

[0068] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A three-dimensional reconstruction method based on ultrasonic images, characterized in that, including: obtaining two-dimensional ultrasonic images generated during an ultrasonic scanning process; obtaining pose information corresponding to the two-dimensional ultrasonic images; determining a nearest-neighbor sampled voxel grid corresponding to a target voxel grid from a plurality of sampled voxel grids, and assigning a value to the target voxel grid according to a voxel value of the nearest-neighbor sampled voxel grid, to obtain an assigned target voxel grid, where the sampled voxel grid is a voxel grid to which the voxel value has been assigned, the target voxel grid is a voxel grid to which the voxel value has not been assigned, and the voxel value is obtained according to a pixel value in the two-dimensional ultrasonic image and the pose information; generating a corresponding three-dimensional ultrasonic volume image according to the sampled voxel grid and the assigned target voxel grid; 2. The three-dimensional reconstruction method based on ultrasonic images according to claim 1, wherein The obtaining pose information corresponding to the two-dimensional ultrasonic images includes: during the ultrasonic scanning process, obtaining six-degree-of-freedom pose information corresponding to each frame of the two-dimensional ultrasonic images based on a binocular camera; The method further includes: mapping, based on the six-degree-of-freedom pose information, a pixel value in the two-dimensional ultrasonic image as the voxel value into a corresponding voxel grid in three-dimensional space, to obtain the sampled voxel grid; 3. The three-dimensional reconstruction method based on ultrasonic images according to claim 1 or 2, characterized in that, The determining a nearest-neighbor sampled voxel grid corresponding to a target voxel grid from a plurality of sampled voxel grids includes: obtaining an Euclidean distance between each of the sampled voxel grids and the target voxel grid based on a nearest-neighbor voxel search method, and using the sampled voxel grid with the minimum Euclidean distance from the target voxel grid as the nearest-neighbor sampled voxel grid; 4. The three-dimensional reconstruction method based on ultrasonic images according to claim 1, characterized in that The generating a corresponding three-dimensional ultrasonic volume image according to the sampled voxel grid and the assigned target voxel grid includes: generating real-time three-dimensional volume data corresponding to a currently scanned area based on the sampled voxel grids and the assigned target voxel grids corresponding to the two-dimensional ultrasonic images of the currently scanned area; after determining that the ultrasonic scanning process is completed, generating the three-dimensional ultrasonic volume image corresponding to all scanned areas according to the sampled voxel grids and the assigned target voxel grids corresponding to all the two-dimensional ultrasonic images; 5. The three-dimensional reconstruction method based on ultrasonic images according to claim 4, wherein, The method further includes: receiving a first input, where the first input includes an instruction for adjusting a three-dimensional reconstruction image display orientation of the real-time three-dimensional volume data or the three-dimensional ultrasonic volume image; in response to the first input, adjusting a current display orientation of the real-time three-dimensional volume data or the three-dimensional ultrasonic volume image to a target display orientation, where the target display orientation includes a coronal plane, a sagittal plane, and an axial plane; 6. The three-dimensional reconstruction method based on ultrasonic images according to claim 1, wherein after the obtaining two-dimensional ultrasonic images generated during an ultrasonic scanning process, the method further includes: performing image preprocessing on the two-dimensional ultrasonic images to obtain preprocessed two-dimensional ultrasonic images, where the image preprocessing at least includes denoising processing, smoothing processing, image enhancement processing, and image correction processing; 7. A three-dimensional reconstruction system based on ultrasonic images, characterized in that, including: an ultrasonic scanning module, configured to obtain two-dimensional ultrasonic images generated during an ultrasonic scanning process; an optical positioning module, configured to obtain pose information corresponding to the two-dimensional ultrasonic images; A three-dimensional reconstruction module, configured to determine a nearest-neighbor sampled voxel grid corresponding to a target voxel grid from multiple sampled voxel grids, and assign a value to the target voxel grid according to the voxel value of the nearest-neighbor sampled voxel grid, so as to obtain an assigned target voxel grid, wherein the sampled voxel grid is a voxel grid to which the voxel value has been assigned, the target voxel grid is a voxel grid to which the voxel value has not been assigned, and the voxel value is obtained according to the pixel value in the two-dimensional ultrasound image and the pose information; A three-dimensional reconstruction display module, configured to generate a corresponding three-dimensional ultrasound volume image according to the sampled voxel grid and the assigned target voxel grid.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the three-dimensional reconstruction method based on ultrasound images according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional reconstruction method based on ultrasound images according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the three-dimensional reconstruction method based on ultrasound images according to any one of claims 1 to 6.

Images