Vascular ultrasound image three-dimensional reconstruction method based on depth camera and IMU

Abstract

The invention relates to a vascular ultrasound image three-dimensional reconstruction method based on a depth camera and an IMU (Inertial Measurement Unit). High-precision three-dimensional reconstruction of a vascular region is realized by fusing data of a multi-source sensor. In the scanning process, a medical worker holds an ultrasonic probe by hand to obtain a two-dimensional ultrasonic image, meanwhile, a plurality of depth cameras distributed around the probe synchronously collect a depth image of a marker, and an IMU sensor synchronously obtains acceleration and angular velocity data of the probe. First preliminary three-dimensional pose information is extracted through depth image registration, a second preliminary pose is estimated in combination with an IMU pre-integration model, the two kinds of pose information are further fused through nonlinear optimization, and the precise three-dimensional pose of the ultrasonic probe at each moment is obtained. And for abnormal image frames appearing in scanning, image restoration is carried out based on pose trajectory interpolation and spatial interpolation reconstruction. And finally, the repaired two-dimensional image is registered with the corresponding pose, so that high-quality three-dimensional reconstruction of the blood vessel region is realized, and the image integrity and the reconstruction precision are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of three-dimensional ultrasonic imaging, and particularly relates to a blood vessel ultrasonic image three-dimensional reconstruction method based on a depth camera and an IMU. BACKGROUND

[0002] Vascular diseases have high morbidity and great harm in clinical practice, and early diagnosis and evaluation are of great significance for the development of treatment plans and patient prognosis. Ultrasonic imaging, as a non-invasive, real-time and low-cost medical imaging technology, is widely used in the examination of vascular structure and blood flow state. In actual operation, two-dimensional ultrasonic images are still the main form of clinical application, and the image acquisition method usually relies on the operator holding the probe to manually scan the region of interest.

[0003] However, the traditional two-dimensional ultrasonic image has certain limitations due to the lack of spatial information, which is not conducive to the comprehensive judgment of the overall structure, spatial shape and three-dimensional lesions of the blood vessels. Therefore, how to recover three-dimensional structural information from two-dimensional ultrasonic image sequences has become a hot research topic in recent years.

[0004] Some existing researches attempt to estimate the attitude change of ultrasonic images in space through image registration, image tracking or external positioning devices (such as optical tracking systems, electromagnetic tracking systems, etc.), so as to realize three-dimensional reconstruction. For example, in the prior art, Chinese patent CN111487320A discloses a three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor, which comprises: an ultrasonic probe for ultrasonic scanning of a target region of interest; a two-dimensional ultrasonic imaging device for generating two-dimensional ultrasonic images of the target region of interest based on the ultrasonic scanning; a three-dimensional optical imaging sensor for acquiring distance information between at least one marker within the visual range of the three-dimensional optical imaging sensor and the camera of the three-dimensional optical imaging sensor and image information of the marker; a spatial information processing module for acquiring three-dimensional spatial information of the ultrasonic probe based on the distance information and the image information; and a three-dimensional reconstruction module for reconstructing three-dimensional ultrasonic images based on the three-dimensional spatial information and the two-dimensional ultrasonic images.

[0005] However, the method has the following limitations: first, the two-dimensional pattern recognition based on the ordinary camera is extremely sensitive to environmental lighting, the marker is easy to be blocked, blurred or recognized due to the change of the visual angle, resulting in low accuracy and poor stability of the pose estimation, which cannot meet the real-time and continuity requirements of the clinical dynamic scanning; second, the frame rate of the traditional visual method is limited, which is difficult to capture the rapid or large amplitude motion of the handheld probe, and is easy to cause image registration error, drift or even reconstruction failure; third, some methods rely on external positioning equipment, which increases the system complexity and cost, and limits the clinical popularization; fourth, abnormal frames (such as structure fracture, ghosting) often occur in two-dimensional ultrasound image sequences due to hand jitter, environmental interference or pose error, and the existing technology lacks effective means for automatic identification and repair of abnormal frames, which affects the continuity and accuracy of three-dimensional reconstruction.

[0006] Therefore, the existing two-dimensional ultrasound three-dimensional reconstruction technology still has obvious technical bottlenecks in the aspects of spatial information loss, insufficient accuracy and stability of pose estimation, and limited processing ability of abnormal images, and needs to be improved to meet the clinical demand for high-quality three-dimensional vascular imaging. SUMMARY

[0007] The purpose of the present application is to overcome the defects of the prior art and provide a vascular ultrasound image three-dimensional reconstruction method based on a depth camera and an IMU.

[0008] The purpose of the present application can be realized by the following technical solutions:

[0009] The present application provides a vascular ultrasound image three-dimensional reconstruction method based on a depth camera and an IMU, comprising the following steps:

[0010] The medical staff holds the ultrasound probe to scan the target vascular region and obtains a plurality of two-dimensional ultrasound images of the target vascular region;

[0011] At the same time of the ultrasound scanning, the depth image data of the marker arranged on the ultrasound probe is synchronously obtained through a plurality of depth cameras arranged around the ultrasound probe, and the acceleration and angular velocity data of the ultrasound probe in the scanning process is synchronously obtained through the IMU sensor arranged on the ultrasound probe;

[0012] The depth image data obtained by the plurality of depth cameras is registered and fused to extract the three-dimensional spatial position information of the marker and estimate the first preliminary three-dimensional pose information of the ultrasound probe in the scanning process;

[0013] The acceleration and angular velocity data obtained by the IMU sensor are pre-integrated to estimate the pose increment of the ultrasound probe between adjacent time points, and the second preliminary three-dimensional pose information under the optimization of the IMU is obtained;

[0014] The first preliminary three-dimensional pose information obtained by the depth camera and the second preliminary three-dimensional pose information obtained by the IMU sensor are fused and corrected through a nonlinear optimization method to obtain accurate three-dimensional pose information of the ultrasonic probe;

[0015] Based on the fused and corrected accurate three-dimensional pose information, an abnormal frame in the obtained multiple two-dimensional ultrasonic images is subjected to trajectory interpolation and image repair;

[0016] The repaired two-dimensional ultrasonic image is registered with the accurate three-dimensional pose information at the corresponding time, and based on the two-dimensional ultrasonic image and the corresponding accurate three-dimensional pose information, a target blood vessel region is subjected to three-dimensional image reconstruction to obtain a three-dimensional ultrasonic image of the target blood vessel region.

[0017] Further, the multiple depth cameras arranged around the ultrasonic probe are arranged in multiple different directions of the ultrasonic probe, and have overlapping field of view ranges between each other, forming a spatial full-view coverage of the ultrasonic probe and the marker.

[0018] The multiple depth cameras are fixed on a scanning region peripheral support or support structure, and their relative positions have been determined through calibration, and each depth image has a unified coordinate reference system.

[0019] Further, the depth image data obtained by the multiple depth cameras is registered and fused to extract three-dimensional spatial position information of the marker, specifically including:

[0020] The depth image obtained by each depth camera is projected into the respective camera coordinate system, and the three-dimensional coordinates (X ci ,Y ci ,Z ci ) of the marker in the respective camera coordinate system are calculated according to the marker image pixel coordinates (u i ,v i ) and the corresponding depth value d i in the depth image, and the calculation method is as follows:

[0021]

[0022] Z ci =d i

[0023] where (ω ci ,I ci ,I ci ) represents the three-dimensional coordinates of the marker in the camera coordinate system of the i-th depth camera, c x and c y are the pixel coordinates of the depth camera principal point, i.e., the image center coordinates of the depth image, f x and fy The focal length of the depth camera;

[0024] Through the extrinsic matrix of the depth camera The three-dimensional coordinates (X) of the marker in the camera coordinate system ci ,Y ci Z ci Transform to world coordinate system:

[0025]

[0026] Among them, (X) wi ,Y wi Z wi () represents the three-dimensional coordinates of the marker in the world coordinate system of the i-th depth camera;

[0027] Transform multiple depth cameras simultaneously to the 3D coordinates of the marker in the world coordinate system {(X wi ,Y wi Z wi Rigid body registration was performed, and the spatial attitude matrix of the markers was estimated using the least squares method.

[0028] Using a fixed spatial transformation matrix between pre-calibrated markers and ultrasonic probes Combined with the spatial attitude matrix of the markers Obtain preliminary three-dimensional pose information of the ultrasonic probe in the world coordinate system in the current frame.

[0029]

[0030] in, Let t be the time t of the ultrasonic probe in the world coordinate system. k The first preliminary three-dimensional pose information.

[0031] Furthermore, the pre-integration of the acceleration and angular velocity data acquired by the IMU sensor to estimate the pose increment of the ultrasonic probe between adjacent time points specifically includes:

[0032] Acceleration data a collected by the IMU sensor k and angular velocity data ω k After performing gravity removal and zero-bias compensation, a pre-integral model is used to calculate the time interval [t] between two frames. k-1 ,t k ] Above, calculate the pre-integral quantity:

[0033]

[0034] in, denotes the rotation increment matrix from time t k-1 to time t k ; ω j denotes the measured angular velocity at time t j ; b ω is the bias of the angular velocity; Δt is the sampling time interval of the IMU sensor; exp is the exponential map of Lie group SO(3); Ω denotes the mapping of a vector to its skew-symmetric matrix form; denotes the position increment from time t k-1 to time t k ; a j , v j are the measured acceleration and velocity at time t k ; R j is the rotation matrix at time t j , which is obtained by the accurate three-dimensional pose information T j at time t j ; b a is the bias of the acceleration;

[0035] The rotation increment and the position increment obtained based on the IMU pre-integration are used to obtain the second preliminary three-dimensional pose information of the ultrasound probe under the IMU optimization.

[0036] Further, the rotation increment and the position increment obtained based on the IMU pre-integration are used to obtain the second preliminary three-dimensional pose information of the ultrasound probe under the IMU optimization, and specifically include:

[0037] The rotation increment and the position increment obtained based on the IMU pre-integration are combined with the accurate three-dimensional pose information T k-1 of the ultrasound probe known at time t k-1 ,

[0038]

[0039] wherein T k-1 is the accurate three-dimensional pose information of the ultrasound probe at time t k-1 ; R k-1 , p k-1 are the rotation matrix and the translation vector at time t k-1 ;

[0040] The second preliminary three-dimensional pose information of the ultrasound probe under the IMU optimization at time t k is estimated as:

[0041]

[0042] wherein, is the time t k the second preliminary three-dimensional pose information of the ultrasound probe, are the rotation matrix and the translation vector under the IMU optimization, respectively.

[0043] Further, the sampling time interval of the IMU sensor is the same as the sampling time interval of the ultrasound probe, that is, at any two adjacent time points t k-1 and t k , the IMU sampling interval Δt is consistent with the time interval of the ultrasound image frame acquisition, ensuring that the estimated pose increment strictly corresponds to the two-dimensional ultrasound image data in time.

[0044] Further, the first preliminary three-dimensional pose information obtained by the depth camera and the second preliminary three-dimensional pose information obtained by the IMU sensor are fused and corrected by a nonlinear optimization method, specifically including:

[0045] based on the first preliminary three-dimensional pose information and the second preliminary three-dimensional pose information a factor graph optimization model is constructed to obtain the true accurate three-dimensional pose information T k of the ultrasound probe at each time point, T k is an optimization variable, and the accurate three-dimensional pose of the ultrasound probe is estimated by minimizing the pose error between the sensor observation value and the optimization variable, and the optimization objective function is:

[0046]

[0047] wherein, T k is the accurate three-dimensional pose information of the ultrasound probe at time t k obtained by nonlinear optimization; is the first preliminary three-dimensional pose information at time t k estimated by the depth camera and the marker, log(·) represents a logarithmic mapping function on the SE(3) Lie group, which is used to convert the pose matrix into a Lie algebra element, that is, a six-dimensional vector form representing the pose error, and ||·|| represents the Euclidean norm of the vector;

[0048] the optimization objective function is solved by Gauss-Newton algorithm to obtain the accurate three-dimensional pose information of the ultrasound probe at time t k .

[0049] Further, based on the fused and corrected accurate three-dimensional pose information, the abnormal frames in the obtained multiple two-dimensional ultrasound images are trajectory interpolated and image repaired, specifically including:

[0050] Based on image gradient change rate and edge consistency, the quality of the obtained two-dimensional ultrasound image sequence {I0, I1, …, I N} is evaluated, and abnormal frames {I j} in the image sequence are judged and located, wherein the abnormal frames exhibit strong structure fracture, ghosting or signal drift;

[0051] For each abnormal frame I j , the normal image frames I k-1 ,I j+1 before and after it are obtained, and the corresponding accurate three-dimensional pose information T j-1 =[R j-1 ,p j-1 ], T j+1 =[R j+1 ,p j+1 ] are obtained, and the interpolation pose at the intermediate time t j is reconstructed by using spherical linear interpolation and linear translation interpolation:

[0052]

[0053] wherein, is the normalized interpolation weight of the time position of the current frame I j between the front and rear frames, are the interpolation rotation matrix and the interpolation translation vector of the abnormal frame I j respectively; Slerp represents a spherical linear interpolation function; and the interpolation pose matrix T

[0054] With the interpolation pose matrix T as a projection reference frame, the pixel data of the front and rear image frames I j-1 ,I j+1 is mapped to the plane where the interpolation frame is located by spatial voxel interpolation, image reconstruction is performed, and the interpolation image frame I

[0055] The interpolation image frame I is replaced with the original abnormal frame I j , and a repaired two-dimensional ultrasound image sequence is constructed.

[0056] Further, the image reconstruction adopts a method based on spatial voxel interpolation, specifically: the pixel points (x, y) corresponding to I j-1 ,I j+1 are projected to the world coordinate system in their respective coordinate systems, are transformed to the interpolation coordinate system according to the interpolation ratio, and the intensity value of each pixel is calculated by bilinear interpolation.

[0057] Further, the repaired two-dimensional ultrasound image is registered with the accurate three-dimensional pose information at the corresponding time, and based on the two-dimensional ultrasound image and its corresponding accurate three-dimensional pose information, a three-dimensional image of the target blood vessel region is reconstructed to obtain a three-dimensional ultrasound image of the target blood vessel region, specifically comprising:

[0058] The repaired two-dimensional ultrasound image sequence is spatially registered with the accurate three-dimensional pose information of the ultrasound probe at the corresponding time, and the position of each pixel point in the two-dimensional ultrasound image in the image coordinate system is mapped to a unified world coordinate system through the accurate three-dimensional pose information of the ultrasound probe, to obtain the voxel position of the pixel point in the three-dimensional space;

[0059] The voxel points and their gray values in all registered ultrasound image frames are fused into a unified three-dimensional voxel space in time sequence to construct a voxel set containing three-dimensional structural information of the target blood vessel region;

[0060] During the fusion process, the voxel data mapped to the same spatial position in different image frames are fused in intensity value, and the pixel gray values in the overlapping regions of multiple frames are processed by weighted average or maximum value method;

[0061] After the spatial registration and fusion of all frame image data are completed, a complete three-dimensional voxel data set is obtained, and the data set is post-processed, and a three-dimensional ultrasound image with a real spatial form is generated by volume rendering or surface extraction of the target blood vessel region using a three-dimensional reconstruction algorithm.

[0062] Compared with the prior art, the present application has the following advantages:

[0063] (1) The present application proposes a blood vessel ultrasound image three-dimensional reconstruction method combining depth camera and IMU sensor, aiming to solve the technical problems of existing two-dimensional ultrasound imaging technology in the aspects of spatial information loss, inaccurate image positioning and insufficient three-dimensional reconstruction precision. Through the combination of multi-source heterogeneous sensor data fusion, spatial geometric modeling and optimization method, and the collaborative design of image repair and three-dimensional reconstruction, the accuracy and stability of ultrasound image positioning are improved, and the effect and visualization quality of three-dimensional imaging of soft tissues such as blood vessels are significantly improved, effectively filling the technical gap of two-dimensional ultrasound in structural expression ability in current clinical use, and having good engineering implementation foundation and clinical application prospect.

[0064] (2) The application adopts multiple depth cameras to obtain the depth image of the marker on the probe, directly extracts the three-dimensional spatial position information through multi-view registration and fusion, and overcomes the limitations of traditional image recognition. The depth camera has high-precision three-dimensional ranging capability and is less affected by the environment. Combined with the multi-camera system, the accuracy and robustness of the three-dimensional pose estimation of the probe can be greatly improved. The technical problem that the existing technology generally uses ordinary cameras combined with marker patterns for ultrasound probe pose estimation, relies on pattern recognition and PnP algorithm in two-dimensional images. This method is sensitive to light conditions, is easily affected by occlusion, blur, and changes in viewing angle, resulting in low pose estimation accuracy and poor stability, and is difficult to meet the needs of dynamic and high-precision ultrasound imaging in clinical practice. The application realizes accurate positioning of the probe position and attitude without relying on two-dimensional pattern recognition, provides accurate geometric support for subsequent spatial registration and high-quality three-dimensional reconstruction of ultrasound images, and significantly improves the imaging effect and clinical practicability.

[0065] (3) The existing technology generally uses a visual method based on ordinary cameras and markers to estimate the pose of the ultrasound probe. This method is highly sensitive to image quality and lighting conditions, and the image frame rate is limited, which cannot meet the requirements of continuous pose and real-time in high-speed or large-amplitude handheld scanning. When the image is blurred, occluded, or the marker recognition fails, the pose estimation is easily interrupted, resulting in registration errors, image drift, or even reconstruction failure. Therefore, the application sets an IMU sensor on the ultrasound probe, real-time acquires acceleration and angular velocity data, and introduces a pre-integration model for incremental pose estimation. After the original IMU data is compensated for gravity and zero bias and temperature drift, the model calculates the rotation increment AR and the position increment Ap between two frames to realize high-frequency fitting of the motion trajectory in a short time, thereby calculating the second preliminary pose information between adjacent time points. Compared with the image-based method, the IMU has a millisecond-level sampling capability, does not depend on environmental lighting and image features, and can effectively make up for the low frame rate, high delay, and data discontinuity of the depth camera. By fusing the incremental pose estimated by the IMU with the accurate pose of the previous frame, the spatial motion trajectory of the ultrasound probe can be continuously and stably tracked, especially when the depth image is missing or abnormal, the trajectory continuity can still be maintained. Further, by setting the IMU sampling period and the ultrasound frame acquisition time to strictly correspond, it is ensured that each frame of ultrasound image is matched with the corresponding three-dimensional pose, improving the timeliness and accuracy of image registration, and significantly enhancing the spatial consistency and imaging accuracy of three-dimensional reconstruction.

[0066] (4)The application fuses two sets of preliminary three-dimensional pose information obtained by a depth camera and an IMU sensor respectively through a nonlinear optimization method, effectively overcoming the shortcomings of single sensor pose estimation. The pose information provided by the depth camera is affected by environmental lighting, occlusion, image noise and other factors, and may have errors and discontinuities; while the IMU has high-frequency sampling and strong anti-interference ability, but long-time integration will accumulate drift errors. It is difficult to achieve high-precision and stable tracking of the ultrasonic probe by relying on any single sensor. To solve this problem, the application constructs a factor graph optimization model based on the first preliminary three-dimensional pose obtained by the depth camera and the second preliminary three-dimensional pose obtained by the IMU as the observation, sets the real accurate three-dimensional pose of the ultrasonic probe at each time as the optimization variable, and jointly corrects by minimizing the error between the observed pose and the optimization variable. The error is expressed by a logarithmic mapping function on the SE(3) Lie group, which converts the pose error into a six-dimensional Lie algebra vector, facilitating quantization and solution. The final corrected accurate three-dimensional pose of the ultrasonic probe is obtained by iterative solution of the Gauss-Newton algorithm. This method makes full use of the complementarity of the data of the two types of sensors, utilizes the spatial absolute positioning ability of the depth camera, and combines the high-frequency dynamic information of the IMU, significantly improving the accuracy, continuity and robustness of the pose estimation, and ensuring the accurate registration of the subsequent two-dimensional ultrasound images and the quality of three-dimensional reconstruction.

[0067] (5)The application proposes a trajectory interpolation and image repair method based on accurate three-dimensional pose for the problem of abnormal frames (such as structure rupture, ghosting, signal drift) in the two-dimensional ultrasound image sequence caused by acquisition environment, handheld jitter or probe pose estimation error, etc., effectively improving the continuity and quality of the image sequence. By analyzing the image gradient change rate and edge consistency, the quality of the two-dimensional ultrasound image sequence is evaluated, and abnormal frames are automatically identified, solving the problem that traditional methods cannot accurately locate image quality mutations. For each abnormal frame, the high-precision three-dimensional pose information of the normal frames before and after it is used, spherical linear interpolation (Slerp) is used for smooth interpolation of the rotation part, linear interpolation is used for interpolation of the translation part, the spatial pose of the abnormal frame is accurately restored, and the discontinuity and distortion problems of simple linear interpolation in rotation processing are effectively avoided. Based on the interpolated pose, the pixels of the adjacent two frames are mapped to the plane where the interpolated frame is located through spatial voxel interpolation, the pixel intensity is calculated by bilinear interpolation, and a high-quality interpolated image frame is reconstructed. This reconstruction process combines spatial pose information and can accurately reflect the real spatial structure, effectively repairing the image information loss and artifacts caused by abnormal frames. Replacing the original abnormal frames, the complete repair of the two-dimensional ultrasound image sequence is realized, ensuring the spatial consistency and continuity of the image data in subsequent three-dimensional reconstruction, thereby significantly improving the accuracy and reliability of three-dimensional vascular imaging. This technical scheme effectively solves the technical bottleneck of abnormal frames being difficult to handle and affecting the reconstruction quality in the prior art, and improves the image quality and reliability of clinical diagnosis. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 A flow chart of a blood vessel ultrasound image three-dimensional reconstruction method according to an embodiment of the present application;

[0069] Figure 2 A system model diagram of a blood vessel ultrasound image three-dimensional reconstruction system according to an embodiment of the present application. DETAILED DESCRIPTION

[0070] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work should fall within the protection scope of the present application.

[0071] Embodiment 1:

[0072] The present embodiment provides a blood vessel ultrasound image three-dimensional reconstruction method based on a depth camera and an IMU, as shown in FIG. 1, comprising the following steps: Figure 1

[0073] Step S1: A medical staff holds an ultrasound probe to perform ultrasound scanning on a target blood vessel region, and acquires a plurality of two-dimensional ultrasound images of the target blood vessel region.

[0074] Among them, the plurality of depth cameras arranged around the ultrasound probe are arranged in a plurality of different directions of the ultrasound probe, and the field of view ranges of the depth cameras overlap, forming a spatial full-view angle coverage of the ultrasound probe and the markers arranged thereon; the plurality of depth cameras are fixed on a scanning region peripheral support or support structure, and the relative positions thereof are determined through calibration, and the depth images have a unified coordinate reference system.

[0075] Step S2: While performing ultrasound scanning, the depth image data of the markers arranged on the ultrasound probe are synchronously acquired through the plurality of depth cameras arranged around the ultrasound probe, and the acceleration and angular velocity data of the ultrasound probe in the scanning process are synchronously acquired through the IMU sensor arranged on the ultrasound probe.

[0076] Among them, the acquired depth image data and IMU sensor data strictly correspond to the two-dimensional ultrasound image frames acquired in the previous step S1 according to time stamps, and each frame of two-dimensional ultrasound image has matching depth image and IMU data. Through a time synchronization mechanism (such as hardware triggering or unified clock calibration), the time consistency of the depth image, the IMU data and the two-dimensional ultrasound image is realized, so as to ensure the accurate pairing and spatial correlation of the data in the subsequent pose estimation and three-dimensional reconstruction process.

[0077] ​Step S3: registering and fusing the depth image data acquired by the plurality of depth cameras, extracting the three-dimensional spatial position information of the marker, and estimating the first preliminary three-dimensional pose information of the ultrasound probe in the scanning process, specifically including:

[0078] Projecting the depth images acquired by each depth camera into the respective camera coordinate system, and according to the marker image pixel coordinates (u i ,v i ) and the corresponding depth value d i in the depth image, calculating the three-dimensional coordinates (X ci ,Y ci ,Z ci ) of the marker in the respective camera coordinate system, and the calculation method is as follows:

[0079]

[0080] Z ci =d i

[0081] Where (X ci ,Y ci ,Z ci ) represents the three-dimensional coordinates of the marker in the camera coordinate system of the i-th depth camera, c x and c y are the pixel coordinates of the principal point of the depth camera, i.e., the image center coordinates of the depth image, f x and f y are the focal lengths of the depth camera.

[0082] Convert the three-dimensional coordinate point (X ci ,Y ci ,Z ci ) of the marker in the camera coordinate system to the world coordinate system through the depth camera extrinsic matrix

[0083]

[0084] Where (X wi ,Y wi ,Z wi ) represents the three-dimensional coordinates of the marker in the world coordinate system of the i-th depth camera.

[0085] Rigidly register the three-dimensional coordinates {(X wi ,Y wi ,Z wi )} of the marker converted to the world coordinate system by the plurality of depth cameras at the same time, and estimate the spatial pose matrix

[0086] ​a fixed spatial transformation relationship matrix between the marker and the ultrasound probe obtained through prior calibration a spatial pose matrix of the marker obtain preliminary three-dimensional pose information of the ultrasound probe in the world coordinate system in the current frame

[0087]

[0088] wherein, is the first preliminary three-dimensional pose information of the ultrasound probe in the world coordinate system at time t k .

[0089] Step S3 synchronously acquires depth image data of the marker on the ultrasound probe by multiple depth cameras, and realizes the first preliminary three-dimensional pose estimation of the ultrasound probe through a series of coordinate transformations and multi-view fusion processing. Traditional two-dimensional vision methods usually rely on ordinary cameras to identify the marker in two-dimensional images, and calculate the three-dimensional pose in combination with the PnP algorithm, but such methods have many technical bottlenecks. First, two-dimensional images lack depth information, which limits the spatial accuracy of pose estimation; second, ordinary cameras are sensitive to environmental lighting conditions, and the marker is easily obscured or blurred, leading to identification failure or instability; in addition, two-dimensional feature matching is difficult when the viewing angle changes greatly, affecting the continuity and accuracy of the pose; more importantly, a single camera cannot achieve multi-view information fusion, making it difficult to break through the viewing angle blind area and limiting the improvement of positioning accuracy and robustness. These problems make it difficult for existing technologies to achieve high-precision and stable estimation of the pose of the ultrasound probe during dynamic scanning, affecting the spatial registration and three-dimensional reconstruction of the ultrasound image.

[0090] To solve the above problems, this step simultaneously collects depth image data of the marker by multiple depth cameras, converts pixel coordinates and corresponding depth values into three-dimensional coordinates using camera intrinsic parameters, realizes the conversion of two-dimensional depth data to three-dimensional point cloud, and solves the problem of lack of spatial information in two-dimensional images. Subsequently, the three-dimensional point clouds in the coordinate systems of each camera are uniformly converted to the world coordinate system through the extrinsic matrix of the depth camera, providing a unified spatial reference for multi-view fusion. Then, a rigid registration algorithm (such as the least squares method) is used to register the three-dimensional point clouds of the marker obtained by different cameras at the same time, accurately estimating the spatial pose of the marker in the world coordinate system. This process fully utilizes multi-view information, reduces single-view blind areas, and improves the accuracy and stability of positioning. Finally, the rigid spatial transformation relationship between the marker and the ultrasound probe obtained through prior calibration is combined to calculate the first preliminary three-dimensional pose of the ultrasound probe in the world coordinate system, ensuring the practical usability of the pose estimation result.

[0091] The application overcomes the technical difficulties of lack of depth information and strong environmental sensitivity of traditional two-dimensional visual methods, and significantly improves the accuracy and robustness of the pose estimation of the ultrasonic probe. The system is not sensitive to light changes by using the structured light or time-of-flight ranging technology of the depth camera, avoiding the recognition failure problem caused by occlusion or blur. Multi-view synchronous acquisition and rigid body registration effectively improve the integrity and stability of spatial positioning, ensuring continuous and stable pose tracking during dynamic scanning. In addition, combined with the fixed spatial transformation between the marker and the probe, the three-dimensional pose of the probe in the world coordinate system is directly output, which is convenient for subsequent spatial registration and three-dimensional reconstruction of the ultrasonic image.

[0092] Step S4: pre-integration based on acceleration and angular velocity data obtained by the IMU sensor to estimate the pose increment of the ultrasonic probe between adjacent time points, and obtain second preliminary three-dimensional pose information under IMU optimization, specifically including:

[0093] The acceleration data a k and angular velocity data ω k collected by the IMU sensor are subjected to gravity and zero offset compensation processing, and then a pre-integration model is used to calculate the pre-integration quantity on the time period [t k-1 , t k ] between two images:

[0094]

[0095] Wherein, represents the rotation increment matrix from time t k-1 to time t k ; ω j represents the measured angular velocity at time t j , b ω is the bias of the angular velocity; Δt is the sampling time interval of the IMU sensor; exp is the exponential mapping of Lie group SO(3); Ω represents mapping a vector to its skew-symmetric matrix form; represents the position increment from time t k-1 to time t k ; a j , v j are the measured acceleration and velocity at time t j , R j is the rotation matrix at time t j , which is obtained through the accurate three-dimensional pose information T j at time t j , b a is the bias of the acceleration;

[0096] The rotation increment and the position increment Obtain the second preliminary three-dimensional pose information of the ultrasound probe under IMU optimization, specifically including:

[0097] Rotation increment obtained based on IMU pre-integration and position increment Combined at time t k-1 Known precise three-dimensional pose information T of the ultrasound probe k-1 ,

[0098]

[0099] Among them, T k-1 Let time t k-1 The precise three-dimensional pose information of the ultrasonic probe, R k-1 p k-1 They are time t respectively k-1 The rotation matrix and translation vector;

[0100] Estimate time t k Preliminary 3D pose information of the ultrasound probe under IMU optimization:

[0101]

[0102] in, Timing t optimized for IMU k Preliminary three-dimensional pose information of the ultrasonic probe. These are the rotation matrix and translation vector under IMU optimization, respectively.

[0103] Step S4 utilizes the acceleration and angular velocity data acquired by the IMU sensor to estimate the pose increment of the ultrasonic probe between adjacent time points using a pre-integration method, thereby obtaining the second preliminary three-dimensional pose information based on the IMU. Traditional ultrasonic probe pose estimation largely relies on visual information, but visual methods alone have several shortcomings, such as sensitivity to environmental factors like lighting and occlusion, and the potential for tracking failure during rapid motion or when image quality is poor. The IMU sensor can acquire motion acceleration and angular velocity at high frequency, providing continuous, real-time motion state information, compensating for the intermittency or inaccuracy caused by image factors in visual positioning.

[0104] This step first preprocesses the raw IMU data, including removing gravity effects and zero-bias compensation, to reduce measurement errors and drift. Then, an IMU pre-integration model is used to accumulate rotation and position increments over a two-frame image acquisition interval. The exponential mapping of the Lie group SO(3) ensures the mathematical properties of the rotation matrix, and the rotation and position increments are calculated. This pre-integration technique can achieve continuous pose increment estimation using high-frequency IMU data even at low visual frame rates, improving dynamic response speed and estimation accuracy.

[0105] The IMU pre-integrated pose increment is multiplied by the accurate pose information known at the last time to obtain the IMU optimized three-dimensional pose at the time. In this way, the pose estimation provided by the IMU can fill the time interval of the visual information update, improve the continuity and accuracy of the overall pose estimation, and especially show good robustness when the ultrasonic probe moves quickly or the visual information is limited.

[0106] The present application solves the problems of low frequency, discontinuity and strong dependence on environment existing in traditional visual positioning through this step. The IMU pre-integration algorithm not only improves the spatio-temporal resolution of pose estimation, but also effectively reduces the cumulative error and enhances the long-term stability through zero offset compensation. Finally, the combination of visual information and IMU data fusion effectively improves the accuracy, real-time and robustness of the ultrasonic probe pose estimation, and promotes high-quality spatial registration and three-dimensional reconstruction of ultrasonic images.

[0107] The sampling time interval of the IMU sensor is the same as the sampling time interval of the ultrasonic probe, that is, at any two adjacent times t k-1 and t k , the IMU sampling interval Δt is consistent with the time interval of the ultrasonic image frame acquisition, ensuring that the estimated pose increment is strictly corresponding to the two-dimensional ultrasonic image data in time.

[0108] Step S5: The first preliminary three-dimensional pose information obtained by the depth camera and the second preliminary three-dimensional pose information obtained by the IMU sensor are fused and corrected by a nonlinear optimization method to obtain accurate three-dimensional pose information of the ultrasonic probe, specifically including:

[0109] Based on the first preliminary three-dimensional pose information and the second preliminary three-dimensional pose information , a factor graph optimization model is constructed to obtain the true accurate three-dimensional pose information T k of the ultrasonic probe at each time, which is used as an optimization variable. By minimizing the pose error between the sensor observation value and the optimization variable, the accurate three-dimensional pose of the ultrasonic probe is estimated, and the optimization objective function is:

[0110]

[0111] Wherein, T k is the accurate three-dimensional pose information of the ultrasonic probe at time t k obtained by nonlinear optimization; is the first preliminary three-dimensional pose information at time t k estimated by the depth camera and the marker, log(·) represents the logarithmic mapping function on the SE(3) Lie group, which is used to convert the pose matrix into Lie algebra elements, i.e. six-dimensional vector form representing the pose error, and ||·|| represents the Euclidean norm of the vector.

[0112] Solve the optimization objective function by Gauss-Newton algorithm to obtain the accurate three-dimensional pose information of the ultrasound probe at time t k , which specifically includes:

[0113] First, take the preliminary estimated pose as the initial value, usually selected from the estimation from the depth camera or IMU. For the estimation of the current iteration, calculate the error vector:

[0114]

[0115] and calculate the Jacobian matrix of these errors on the Lie algebra increment . Here, the increment δξ k is used to update the pose by the exponential mapping:

[0116]

[0117] where is the skew-symmetric matrix representation of the Lie algebra increment δξ k .

[0118] Integrate all the errors and Jacobian matrices of iterations to form a linearization problem:

[0119]

[0120] where e is a large vector composed of all the error vectors of iterations, and J is the corresponding Jacobian matrix.

[0121] Solve the increment δξ by the normal equation:

[0122] J T Jδξ=-J T e

[0123] Usually, numerical methods such as Cholesky decomposition are used. Update the pose estimation at each time by the increment to complete one iteration. Repeat the iteration process until the increment norm or the target function drop meets the set convergence condition, or reaches the maximum number of iterations.

[0124] Step S6: Based on the fused and corrected accurate three-dimensional pose information, perform trajectory interpolation and image inpainting on the abnormal frames in the obtained multiple two-dimensional ultrasound images, specifically including:

[0125] Based on the image gradient change rate and edge consistency, perform quality assessment on the obtained two-dimensional ultrasound image sequence {I0, I1, …, I N}, judge and locate the abnormal frames {I jwhere the abnormal frame is represented as strong structure break, ghosting or signal drift;

[0126] For each abnormal frame I j , the normal image frames I j-1 I j+1 corresponding accurate three-dimensional pose information T j-1 = [R j-1 , p j-1 ], T j+1 = [R j+1 , p j+1 ] are obtained, and the interpolation pose at the intermediate time t j is reconstructed by using the spherical linear interpolation and linear translation interpolation:

[0127]

[0128] wherein, is the normalized interpolation weight of the time position of the current frame I j between its front and back frames, are the interpolation rotation matrix and interpolation translation vector of the abnormal frame I j respectively; Slerp represents the spherical linear interpolation function; and the interpolation pose matrix T

[0129] With the interpolation pose matrix T as the projection reference frame, the pixel data of the front and back image frames I j-1 I j+1 is mapped to the plane where the interpolation frame is located by spatial voxel interpolation, image reconstruction is performed, and the interpolation image frame I

[0130] The interpolation image frame I is replaced with the original abnormal frame I j , and a repaired two-dimensional ultrasound image sequence is constructed.

[0131] The image reconstruction adopts a method based on spatial voxel interpolation, specifically: the pixel points (x, y) corresponding to I j-1 I j+1 are projected to the world coordinate system in their respective coordinate systems, transformed to the interpolation coordinate system according to the interpolation ratio, and the intensity value of each pixel is calculated by bilinear interpolation.

[0132] In step S6, the reason for introducing the trajectory interpolation and image inpainting processing of the abnormal frame based on the fused corrected accurate three-dimensional pose information is to solve the problem of abnormal frames in ultrasonic image acquisition due to complex acquisition environment, operator hand-held probe shaking, unstable probe and human body contact, or inaccurate probe pose estimation, etc. These abnormal frames usually show image structure rupture, obvious ghosting phenomenon, signal drift and other distortion problems, which seriously affect the quality of subsequent image stitching, registration and three-dimensional reconstruction.

[0133] The prior art often lacks a compensation mechanism combining three-dimensional spatial pose information when processing abnormal frames in an ultrasonic image sequence, or only relies on image interpolation frame filling methods, which easily leads to geometric distortion, inter-frame discontinuity or structure connection errors. The present application uses corrected high-precision three-dimensional pose trajectory information from the source to establish a consistency relationship between the image and the spatial geometry, thereby performing interpolation positioning and reconstruction of abnormal frames from the motion trajectory level.

[0134] Specifically, for the detected abnormal frame, the accurate three-dimensional pose information corresponding to its adjacent normal frames is extracted, the intermediate value of the rotation attitude is calculated using spherical linear interpolation (Slerp), and the translation vector is calculated using linear interpolation, to accurately calculate the pose of the probe in space at the abnormal frame time.

[0135] Based on the interpolated pose, the present application further reconstructs the image by projecting each pixel point in the front and rear frame images from its local coordinate system to the world coordinate system and mapping it to the reference coordinate plane of the interpolated frame, and then calculating the pixel gray value by bilinear interpolation, to realize high-geometric-consistency image reconstruction. This method not only ensures the continuity of image texture and structure in space, but also avoids the misplacement and blur problem caused by ignoring the pose change in traditional image frame interpolation.

[0136] Replacing the reconstructed interpolated image frame with the original abnormal frame completes the repair of the image sequence, improves the spatiotemporal coherence and data integrity of the image sequence, and ensures the accurate registration of each image frame in space for subsequent three-dimensional reconstruction.

[0137] Step S7: registering the repaired two-dimensional ultrasonic image with the corresponding accurate three-dimensional pose information at the corresponding time, performing three-dimensional image reconstruction on the target blood vessel region based on the two-dimensional ultrasonic image and the corresponding accurate three-dimensional pose information, to obtain a three-dimensional ultrasonic image of the target blood vessel region, specifically including:

[0138] The repaired two-dimensional ultrasound image sequence is spatially registered with the accurate three-dimensional position information of the ultrasound probe at the corresponding time, the position of each pixel in the two-dimensional ultrasound image in the image coordinate system is mapped to a unified world coordinate system through the accurate three-dimensional position information of the ultrasound probe, and a voxel position of the pixel in the three-dimensional space is obtained;

[0139] The voxel points and their gray values in all registered ultrasound image frames are fused into a unified three-dimensional voxel space in chronological order, and a voxel set containing three-dimensional structural information of the target blood vessel region is constructed;

[0140] In the fusion process, the voxel data mapped to the same spatial position in different image frames is fused in intensity value, and the pixel gray values in the overlapping regions of multiple frames are processed in a weighted average or maximum value manner;

[0141] After the spatial registration and fusion of all frame image data are completed, a complete three-dimensional voxel data set is obtained, the data set is post-processed, a three-dimensional reconstruction algorithm is used to perform volume rendering or surface extraction on the target blood vessel region, and a three-dimensional ultrasound image with a real spatial form is generated.

[0142] Embodiment 2:

[0143] The embodiment also provides a blood vessel ultrasound image three-dimensional reconstruction system based on a depth camera and an IMU, as shown in Figure 2 , comprising:

[0144] An ultrasound probe, which is used to scan a target blood vessel region and acquire a plurality of two-dimensional ultrasound images;

[0145] An IMU sensor arranged on the ultrasound probe, which is used to collect acceleration and angular velocity data of the ultrasound probe in real time during scanning;

[0146] A plurality of depth cameras arranged around the ultrasound probe, which are used to synchronously collect depth image data of a marker arranged on the ultrasound probe;

[0147] A data processing module, which is used to register and fuse the depth image data acquired by the depth cameras, extract three-dimensional spatial position information of the marker, estimate first preliminary three-dimensional position information of the ultrasound probe, and pre-integrate the IMU data to calculate second preliminary three-dimensional position information, fuse the two types of position information through nonlinear optimization, and obtain accurate three-dimensional position of the ultrasound probe;

[0148] An image processing module, which is used to perform quality assessment, trajectory interpolation and image repair on abnormal frames in the ultrasound images;

[0149] A three-dimensional reconstruction module is configured to register the repaired two-dimensional ultrasound image with the corresponding accurate three-dimensional pose information, and complete three-dimensional image reconstruction of the target blood vessel region.

[0150] The system realizes high-precision positioning and fusion of ultrasound images in space, and effectively improves the accuracy and robustness of three-dimensional imaging of blood vessels.

[0151] If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the prior art that essentially contributes or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0152] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for 3D reconstruction of vascular ultrasound images based on depth camera and IMU, characterized in that, The method comprises the following steps: A medical staff holds an ultrasound probe to perform ultrasound scanning on a target blood vessel region to obtain a plurality of two-dimensional ultrasound images of the target blood vessel region; During the ultrasound scanning, a plurality of depth cameras arranged around the ultrasound probe are used to synchronously obtain depth image data of a marker arranged on the ultrasound probe, and an IMU sensor arranged on the ultrasound probe is used to synchronously obtain acceleration and angular velocity data of the ultrasound probe during the scanning process; The depth image data obtained by the plurality of depth cameras is registered and fused to extract three-dimensional spatial position information of the marker and estimate first preliminary three-dimensional pose information of the ultrasound probe during the scanning process; The acceleration and angular velocity data obtained by the IMU sensor are pre-integrated to estimate a pose increment of the ultrasound probe between adjacent time instants and obtain second preliminary three-dimensional pose information optimized by the IMU; The first preliminary three-dimensional pose information obtained by the depth cameras and the second preliminary three-dimensional pose information obtained by the IMU sensor are fused and corrected by a nonlinear optimization method to obtain accurate three-dimensional pose information of the ultrasound probe; Based on the accurate three-dimensional pose information after fusion and correction, abnormal frames in the obtained plurality of two-dimensional ultrasound images are subjected to trajectory interpolation and image inpainting; The repaired two-dimensional ultrasound images are respectively registered with the accurate three-dimensional pose information at corresponding time instants, and based on the two-dimensional ultrasound images and the accurate three-dimensional pose information corresponding thereto, three-dimensional image reconstruction is performed on the target blood vessel region to obtain a three-dimensional ultrasound image of the target blood vessel region.

2. The method of claim 1, wherein, The plurality of depth cameras arranged around the ultrasound probe are arranged in a plurality of different directions of the ultrasound probe, and the depth cameras have overlapping field of view ranges, thereby forming a spatial full-view angle coverage of the ultrasound probe and the marker thereof; The plurality of depth cameras are fixed on a scanning region peripheral support or support structure, and the relative positions of the depth cameras are determined through calibration, and the depth images have a unified coordinate reference system. 3.The blood vessel ultrasound image three-dimensional reconstruction method based on a depth camera and an IMU according to claim 1, characterized in that, The depth image data obtained by the plurality of depth cameras is registered and fused, The three-dimensional spatial position information of the marker is extracted, specifically including: Projecting the depth images acquired by each depth camera into the respective camera coordinate system, and according to the identified object image pixel coordinates (u i ,v i ) and the corresponding depth values d i in the depth image, calculating the three-dimensional coordinates (X ci ,Y ci ,Z ci ) of the identified object in the respective camera coordinate system, the calculation method being: Z ci = d i wherein (X ci ,Y ci ,Z ci ) represents the three-dimensional coordinates of the marker in the camera coordinate system of the i-th depth camera, c x , c y are the pixel coordinates of the principal point of the depth camera, i.e. the image center coordinates of the depth image, f x , f y are the focal length of the depth camera; By the depth camera extrinsic matrix The three-dimensional coordinate point (X ci ,Y ci ,Z ci ) of the marker in the camera coordinate system is converted to the world coordinate system: wherein (X wi ,Y wi ,Z wi ) represents the three-dimensional coordinates of the marker in the world coordinate system of the i-th depth camera; Converting the three-dimensional coordinates of the marker in the world coordinate system {(X wi ,Y wi ,Z wi )} of multiple depth cameras at the same time to rigid registration, using the least square method to estimate the spatial pose matrix of the marker Using the fixed spatial transformation relationship matrix between the marker and the ultrasonic probe obtained by prior calibration , combined with the spatial pose matrix of the marker Obtain the preliminary three-dimensional pose information of the ultrasonic probe in the world coordinate system of the current frame wherein, is a first preliminary three-dimensional pose information of the ultrasound probe at the time instant t in the world coordinate system. k is a first preliminary three-dimensional pose information of the ultrasound probe at the time instant t in the world coordinate system.

4. The method of claim 1, wherein, The acceleration and angular velocity data obtained by the IMU sensor are pre-integrated to estimate a pose increment of the ultrasound probe between adjacent time instants, specifically including: Acceleration data a collected by the IMU sensor k and angular velocity data ω k After performing gravity removal and zero-bias compensation, a pre-integral model is used to calculate the time interval [t] between two frames. k-1 ,t k ] Above, calculate the pre-integral quantity: wherein, represents the rotation increment matrix from time t k-1 to time t k ; ω j represents the measured angular velocity at time t j , b ω is the bias of the angular velocity; Δt is the sampling time interval of the IMU sensor; exp is the exponential mapping of Lie group SO(3); Ω represents the mapping of a vector to its skew-symmetric matrix form; represents the position increment from time t k-1 to time t k ; a j , v j are the measured acceleration and velocity at time t j , respectively; R j is the rotation matrix at time t j , which is obtained through the accurate three-dimensional pose information T j at time t j ; b a is the bias of the acceleration; Rotational increments based on IMU pre-integration and positional increments Obtain second preliminary three-dimensional pose information of the ultrasound probe under IMU optimization.

5. The method of claim 4, wherein, The rotation increment based on the IMU pre-integration and the position increment Obtaining the second preliminary three-dimensional pose information of the ultrasound probe under the IMU optimization, specifically comprising: a rotation increment based on the IMU pre-integration and a position increment combined at time t k-1 known precise three-dimensional pose information T of the ultrasound probe k-1 , wherein T k-1 is the accurate three-dimensional pose information of the ultrasound probe at time t k-1 R k-1 and p k-1 are the rotation matrix and the translation vector at time t k-1 , respectively. Estimation time t k Second preliminary three-dimensional pose information of the ultrasound probe under IMU optimization: wherein, t is the time instant under IMU optimization k ultrasound probe second preliminary three-dimensional pose information, are respectively the rotation matrix and the translation vector under IMU optimization.

6. The method of claim 4, wherein, The sampling time interval of the IMU sensor is the same as the sampling time interval of the ultrasound probe, that is, at any two adjacent time points t k-1 and t k , the IMU sampling interval Δt is consistent with the time interval of the acquisition of the ultrasound image frame, ensuring that the estimated pose increment strictly corresponds in time to the two-dimensional ultrasound image data.

7. The method of claim 1 or 4, wherein, The first preliminary three-dimensional pose information obtained by the depth cameras and the second preliminary three-dimensional pose information obtained by the IMU sensor are fused and corrected by a nonlinear optimization method, specifically including: based on the first preliminary three-dimensional pose information with the second preliminary three-dimensional pose information a factor graph optimization model is constructed to obtain the real accurate three-dimensional pose information T of the ultrasound probe at each time k For the optimization variable, the estimation of the accurate three-dimensional pose of the ultrasound probe is realized by minimizing the pose error between the sensor observation value and the optimization variable, and the optimization objective function is: where T k is the accurate 3D pose information of the ultrasound probe at time t k obtained by nonlinear optimization; is the first preliminary 3D pose information of the ultrasound probe at time t k estimated by the depth camera and the marker, log(·) represents a logarithmic mapping function on the SE(3) Lie group, which is used to convert the pose matrix into a Lie algebra element, i.e., a pose error represented in the form of a six-dimensional vector, and ||·|| represents the Euclidean norm of the vector. The optimization objective function is solved by a Gauss-Newton algorithm to obtain the accurate three-dimensional pose information of the ultrasound probe at time t k . 8.The blood vessel ultrasound image three-dimensional reconstruction method based on a depth camera and an IMU according to claim 1, characterized in that, Based on the accurate three-dimensional pose information after fusion and correction, abnormal frames in the obtained plurality of two-dimensional ultrasound images are subjected to trajectory interpolation and image inpainting, specifically including: Based on image gradient change rate and edge consistency, the quality of the acquired two-dimensional ultrasound image sequence {I0, I1, …, I N} is evaluated, and abnormal frames {I j} in the image sequence are judged and located, where the abnormal frames exhibit strong structure fracture, ghosting or signal drift; For each abnormal frame I j , the normal image frames I j-1 ,I j+1 corresponding to the time before and after the abnormal frame I j-1 are obtained j-1 , T j-1 , T j+1 = [R j+1 , p j+1 ], the interpolation pose of the intermediate time t j is reconstructed by using the spherical linear interpolation and linear translation interpolation wherein, is a normalized interpolation weight between the frames preceding and following the current frame I j is a normalized interpolation weight between the frames preceding and following the current frame I are the interpolation rotation matrix and the interpolation translation vector, respectively, of the abnormal frame I j are the interpolation rotation matrix and the interpolation translation vector, respectively, of the abnormal frame I With the interpolation pose matrix As the projection reference frame, the pixel data of the front and rear two image frames I j-1 ,I j+1 is mapped to the plane where the interpolation frame is located through spatial voxel interpolation, image reconstruction is performed, and the interpolation image frame I said interpolated image frames replacing the original abnormal frame I j , constructing a repaired two-dimensional ultrasound image sequence.

9. The method of claim 8, wherein, The image reconstruction adopts a method based on spatial voxel interpolation, specifically: I j-1 ,I j+1 The corresponding pixel points (x, y) are projected into the world coordinate system in their respective coordinate systems, transformed to the interpolation coordinate system according to the interpolation ratio, and the intensity value of each pixel is calculated through bilinear interpolation.

10. The method of claim 1, wherein, The repaired two-dimensional ultrasound images are respectively registered with the accurate three-dimensional pose information at corresponding time instants, and based on the two-dimensional ultrasound images and the accurate three-dimensional pose information corresponding thereto, three-dimensional image reconstruction is performed on the target blood vessel region to obtain a three-dimensional ultrasound image of the target blood vessel region, specifically including: The repaired two-dimensional ultrasound image sequence is spatially registered with the accurate three-dimensional pose information of the ultrasound probe at the corresponding time, and the position of each pixel in the two-dimensional ultrasound image under the image coordinate system is mapped to a unified world coordinate system through the accurate three-dimensional pose information of the ultrasound probe, so as to obtain the voxel position of the pixel in the three-dimensional space; The voxel points and their gray values in all registered ultrasound image frames are fused into a unified three-dimensional voxel space in time sequence, so as to construct a voxel set containing three-dimensional structure information of the target blood vessel region; During the fusion process, the voxel data mapped to the same spatial position in different image frames is fused in intensity value, and the pixel gray values in the overlapping regions of multiple frames are processed in a weighted average or maximum value manner; After the spatial registration and fusion of all frame image data are completed, a complete three-dimensional voxel data set is obtained, and the data set is post-processed, a three-dimensional reconstruction algorithm is adopted to perform volume rendering or surface extraction on the target blood vessel region, and a three-dimensional ultrasound image with a real spatial form is generated.

Images