A method for quickly acquiring an ultrasonic three-dimensional structure without relying on an external positioning device

Abstract

The present invention discloses a method for a rapid acquisition system of ultrasonic three-dimensional structure that does not rely on external positioning equipment. Without auxiliary positioning equipment, the existing two-dimensional ultrasonic imaging system can be used to achieve rapid acquisition and presentation of three-dimensional tissue structure, which is used for accurate lesion positioning and assists in accurate surgical navigation. The invention will be conducive to promoting the clinical use rate of ultrasonic precision surgical navigation, improving the effect and success rate of surgical treatment, and is of great significance to reducing the labor intensity of doctors and the trauma of patients.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical ultrasound positioning and reconstruction, and in particular relates to a method for a rapid ultrasonic three-dimensional structure acquisition system that does not rely on external positioning equipment. Background Art

[0002] Ultrasound imaging has become one of the main medical modes nowadays due to its advantages such as lower price cost, higher safety standards, and easier operation. It can complete the task of imaging internal tissues of the human body through its non-invasive, painless, intuitive and effective imaging method. Doctors can observe the patient's body structure and functional information in real time, and then monitor and track the location of the lesion to achieve the purpose of clinical disease diagnosis and interventional application.

[0003] However, the two-dimensional ultrasound imaging systems commonly used in hospitals are mainly used for disease diagnosis and do not play a guiding role in the treatment of patients' diseases, that is, the direct usage rate in the operating room is low. For example, in actual scenarios, doctors need to obtain detailed information such as the shape or range of the lesion structure in the patient's body, and align the real-time image data obtained by the intraoperative device scan with the three-dimensional magnetic resonance imaging data reconstructed before the operation. However, due to the differences in the size of the image and the appearance of the individual target tissue of the patient, the portable two-dimensional ultrasound system cannot accurately meet the needs of surgical navigation tasks, because it can only provide flat two-dimensional images and lacks three-dimensional information on the target tissue structure.

[0004] At present, mechanical volume ultrasound probes use the large difference in sound velocity propagation at the amniotic fluid-fetus interface to provide three-dimensional tissue imaging for prenatal screening of pregnant women. However, this solution has a small imaging range and relies on an interface with a large difference in sound velocity propagation. Therefore, this type of ultrasound probe is mainly used for three-dimensional fetal imaging diagnosis, and it is difficult to directly apply it to three-dimensional imaging of other parts of the human body. At the same time, some researchers have designed special probes with sensor elements that can collect two-dimensional arrays to replace ordinary probes, directly capturing three-dimensional data of target human tissues, but they often have problems such as high cost and complex scanning conditions, and the arrangement matrix obtained by using sensor elements will reduce image quality and affect the quality of three-dimensional reconstruction results.

[0005] In order to directly present the three-dimensional structure using the two-dimensional ultrasound imaging system, the probe posture can be collected with the help of a spatial positioner to complete the three-dimensional reconstruction of the tissue structure. For example, mechanical, optical or electromagnetic sensors are rigidly installed on the ultrasound probe to continuously measure the position and orientation data of the target tissue. This type of system can count ultra-long ultrasound scans with arbitrary trajectories, which is of great value for applications such as vascular mapping and bone reconstruction. Fusion analysis with three-dimensional images obtained before surgery can present more intuitive target tissue information to assist surgical navigation. However, the spatial positioning technology used is easily affected by the surgical environment, has limited accuracy and a smaller range of free movement of the instrument. The tracking and positioning instruments involved are also expensive and have limited observation angles. Electromagnetic and other influences will occur near the scanned body structure, so this type of system has too many constraints and restrictions and is not applicable.

[0006] Therefore, in order to realize the direct transition of ultrasound imaging systems from disease diagnosis to surgical navigation applications, it is necessary to develop ultrasound three-dimensional structure rapid acquisition technology that does not rely on external positioning equipment to accurately locate the lesion. Pure image analysis technology performed by computers can be introduced to estimate the relative position transformation between ultrasound images. Traditional exploration of the correlation between consecutive frames and the time domain changes at the pixel level restores the motion in the plane, but it is difficult to estimate the out-of-plane motion. At present, speckle decorrelation methods are often used to solve the problem of out-of-plane motion by mining the visible and special speckle noise information in ultrasound images. However, most speckle decorrelation methods based on physical and statistical properties fail on real clinical image data. It also shows that the displacement of the speckle pattern is inconsistent with the movement of the underlying patient's body tissue, resulting in errors in the patient's body data collection. The accumulated error will produce obvious drift, so the speckle decorrelation method has not been transformed into clinical trials and commercial solutions.

[0007] Based on the current situation that two-dimensional ultrasound imaging systems in hospitals are difficult to apply to surgical guidance, we can draw the following conclusions: Ultrasound imaging has developed rapidly in medical diagnostic applications, and there are many related products and equipment, but it is unable to generate the three-dimensional structural data required for clinical trials, facing a development bottleneck. Even if there are special probes such as volumetric or spatial positioning instruments to generate three-dimensional data of target tissues and organs in the human body, the shortcomings of small application range, high cost, limited accuracy, and damage to the body cannot be transformed into a solution. Therefore, there is an urgent need for a pure image analysis technology that can quickly obtain ultrasound three-dimensional structure without the help of external positioning equipment and computer assistance, which can adapt to different organs and tissues of the human body, and achieve the three-dimensional surgical navigation function using a two-dimensional ultrasound probe based on disease diagnosis, improve the effect and success rate of surgical treatment, and reduce the labor intensity of doctors and patient trauma. Summary of the invention

[0008] In view of the lack of existing applications of ultrasound imaging technology, the present invention aims to use the data collected by a two-dimensional ultrasound probe based on disease diagnosis to realize the three-dimensional auxiliary surgical navigation function and promote the clinical use rate of ultrasound precision surgical navigation. To this end, the present invention proposes a method for quickly acquiring three-dimensional ultrasound structure with the help of a graphics processor, which collects a two-dimensional ultrasound image sequence of the patient's target tissue through an ordinary two-dimensional ultrasound probe, obtains the trajectory estimation of the ultrasound probe after image preprocessing and three-dimensional reconstruction, and provides doctors with more accurate information such as the location of the patient's lesions for preoperative preparation. Without the help of external positioning equipment, it not only reduces costs, but also avoids the impact of electromagnetic and other equipment on the body. At the same time, unlike the physical and statistical methods based on the simple use of speckle decorrelation, a real-time, high-precision, and widely adaptable three-dimensional ultrasound modeling system and method is realized.

[0009] To achieve the above object, the technical solution adopted by the present invention is:

[0010] A method for a rapid ultrasonic three-dimensional structure acquisition system that does not rely on an external positioning device, using an ordinary two-dimensional ultrasonic probe to acquire image data to construct a three-dimensional model of a target organ, comprising the following steps:

[0011] S1. Use ordinary ultrasound probe to collect spatial positioning information Two-dimensional ultrasound image sequence (I n ) n , and rigidly install an inertial measurement unit (IMU) on the probe to collect auxiliary data T IMU ;

[0012] S2, the spatial positioning information obtained in step S1 Two-dimensional ultrasound image sequence (I n ) n Reconstruction into three-dimensional ultrasound structure by a method that does not rely on external magnetic positioning or optical positioning equipment;

[0013] S21, the image sequence set (I n ) n According to the two adjacent ultrasound slices (I n-1 ,I n ) is used as a unit to segment each ultrasound image sequence, then each ultrasound image sequence is divided into N segments, where: n-1 and I n They correspond to the n-1th and nth frames in a single image sequence respectively;

[0014] S22, the unit input after the division in step S21 (I n-1 ,I n ) are processed by optical flow algorithm to obtain the relative motion components d corresponding to adjacent ultrasound slices. flow ;

[0015] S23, the motion component d obtained in step S22 flow The unit input (I n-1 ,I n ) are connected and used together as the superposition input of the long short-term memory network model LSTM based on the sequence method;

[0016] S24, sending the superimposed input in step S23 into the long short-term memory network model LSTM to obtain the hidden state h corresponding to the time t t After that, the auxiliary data T collected by IMU in step S1 IMU Connect, and then output 6 value result parameters after linear transformation To represent the rigid motion between adjacent frames, where: T IMU Depend on It consists of three parameters: Respectively represent the angles of rotation of the IMU device around the x, y, and z axes at time t. The three degrees of freedom are respectively the three translation degrees of freedom along the x, y, and z coordinate axes and the three degrees of freedom of rotation around the coordinate axes; the N-1 adjacent slices remaining in step S21 are sequentially used as the second, third, ..., N-1 inputs of the long short-term memory network model LSTM, and the corresponding output parameters are obtained in turn to reconstruct the three-dimensional structure;

[0017] S25, for the adjacent frames (I) of the unit ultrasound image in step S24, n-1 ,I n ) Calculate the mean absolute parameter error, maximum center error, and length error for the nth ordered ultrasound image sequence (I n ) n The overall loss function Loss value of the entire network model is calculated; the model of the present invention is trained by the loss function and the optimization method, and the trained model is directly used to process two-dimensional ultrasonic slices, thereby being used for three-dimensional trajectory reconstruction of the ultrasonic probe.

[0018] Furthermore, in step S1, a two-dimensional ultrasound image sequence (I n ) n The method is as follows:

[0019] A Cicada ultrasound machine was used to capture 3D optically tracked ultrasound scans, using a 128-element linear probe at 5 MHz to generate images consisting of 256 scan lines, capturing ultrasound images at 35 frames per second.

[0020] In step S1, the real space position label corresponding to the ultrasound image is tracked and recorded. The method is as follows:

[0021] Using Stryker NAV3 TM The optical positioning system of the camera records the real spatial position labels of all ultrasound images after sensor calibration The absolute positioning accuracy is within 0.2mm translation range.

[0022] In step S1, the inertial measurement unit IMU integrated hardware acquires auxiliary data T IMU The method is as follows:

[0023] Inertial measurement unit (IMU) refers to a chip with a size of less than 1 cm that integrates a three-dimensional magnetometer, a rate gyroscope, and an accelerometer. It is installed in a mobile phone or other handheld device to provide the movement direction and acceleration information of the ultrasound probe, and only uses the collected direction information; a sensor with IMU is used to estimate the orientation of the ultrasound probe to obtain rotation measurement data T IMU Used as auxiliary data to reconstruct the trajectory of the ultrasound probe and obtain the ultrasound three-dimensional structure;

[0024] IMU devices directly return useful representations of Euler directions The angle data is recorded in the 3×3 rotation matrix T IMU In which: Respectively represent the rotation angle of the IMU device around the x, y, and z coordinate axes; due to the real spatial position label The camera coordinate system is established with the optical tracking scanning camera as the origin of the coordinate system, and the auxiliary data T IMU The data is collected in the world coordinate system, where the spatial tags recorded by optical tracking have been calibrated with the ultrasound image. Therefore, the auxiliary data T obtained by the IMU device is IMU The real label information collected After time and space calibration, it can be used in the method of reconstructing ultrasound three-dimensional structure; Formula (101) maximizes the use of the real spatial position label value Auxiliary data T detected by IMU IMU Time calibration is performed by correlation between:

[0025]

[0026] Where: t IMU→GT Represents the real space position label at time t With auxiliary data T IMU The correlation between them is the largest, so the best calibration effect is achieved in the time dimension; ω IMU Represents the measured angular velocity of the gyroscope sensor in the IMU, ω GT is the angular velocity of the real spatial position tag; in order to align the directions of the two tracking coordinate systems, the auxiliary data T IMU Real-space labeling with optical tracking Perform spatial calibration according to formula (102):

[0027]

[0028] Among them: For the kth interval of a certain ultrasound sequence: matrix contains the rotation part of the real space position label, and the matrix Contains the Euler direction parameters captured by the IMU; C and R represent the constant calibration matrix and registration matrix respectively: C describes the local transformation between the IMU sensor and the optical tracking target, and R describes the global transformation between the world coordinate system referenced by the IMU and the optical tracking camera; the two matrices C and R are minimized by using the MLSL algorithm T k GT The absolute Euler residual between the right side of equation (102) and the right side of equation (102) is used to perform numerical global optimization.

[0029] Furthermore, in step S21, the two-dimensional ultrasound image sequence (I n ) n , trying to reconstruct the nth frame of ultrasound image I n The corresponding time t n The trajectory of the ultrasound probe is represented by a matrix sequence (T n ) n Encoding: Encode the trajectory into a homogeneous matrix and then pass the 6-value result vector Represents the relative transformation corresponding to the trajectory, where 3 degrees of freedom represent translation along the x, y, and z axes, and 3 degrees of freedom represent rotation around the x, y, and z axes. Since there is no absolute coordinate system, only relative changes are considered: it is estimated that each frame I n Compared with the previous frame ultrasound image I n-1 Relative transformation of T n-1→n represents the n-1th transformation T of the ultrasound image sequence n-1 With the nth conversion T n A rigid relative transformation, T n-1→n With 6-value vector Parameterization, Represents the matrix T n-1 The inverse matrix of n-1 ,I n ), train a suitable long short-term memory network LSTM to act on adjacent frames, through LSTM (I n-1 ,I n ) process estimates the outcome parameters that represent relative changes:

[0030]

[0031] Design a long short-term memory network model LSTM, iterate all pairs of consecutive adjacent frame ultrasound images, and use the chain rule to estimate T using the previous relative transformation motion 2→3 ,T 3→4 ,...,T k-1→k ,....Calculate any matrix T n-1→n , where: T 1→2 Refers to the relative transformation matrix from the first frame to the second frame of the ultrasound sequence, T 2→3 ,T 3→4 ,...,T k-1→k ,....with T 1→2 The meaning is consistent; the ultrasound image sequence (I n ) n The units of two consecutive frames are divided into N segments, and the rigid motion is divided into t x ,t y ,t z The three parameters represent the translational motion and θ x ,θ y ,θ z The 3 Euler angles represent the rotation components:

[0032] Furthermore, in step S22, the continuous adjacent frames (I n-1 ,I n ); Assume that the previous frame I n-1 A pixel u=[u x ,u y ] and the next frame I n The pixel v = [u x +d x ,u y +d y ] matches, that is, the grayscale difference between pixels u and v is the smallest, and u and v correspond to each other one by one, where: [u x ,u y ] represents the x and y axis coordinates of pixel u in the two-dimensional plane of the image; d flow =[d x ,d y ] is I n-1 Pixel u moves to I n The relative motion component of pixel v, [d x ,d y ] represents the movement length of pixel u along the x and y axes of the two-dimensional plane when it moves to v; H n (u x ,u y ) means that the coordinates of the point in the nth frame are (u x ,u y ) pixel value size, Hn-1 (x,y) means the coordinates of the point in the n-1 frame are (u x +d x ,u y +d y ) pixel value, then I n-1 with I n The relationship between the pixel values ​​of the image frame after optical flow processing is as shown in formula (2):

[0033] H n (u x ,u y )=H n-1 (u x +d x ,u y +d y ) (2)

[0034] Furthermore, in step S23, each ultrasound image sequence (I n ) n Divide into N segments according to the adjacent frames and process them sequentially: divide the adjacent frames (I n-1 ,I n ) and the corresponding relative motion component d extracted in step S22 flow After fusion according to formula (3), we get the input X n-1 , directly fed into the long short-term memory network model LSTM:

[0035] X n-1 =[I n-1 ,I n ,d flow ] (3)

[0036] The LSTM network has three control gates f at time t t 、i t and t Each control gate essentially uses a nonlinear function Sigmoid, which is determined by the input x corresponding to the current time t t And the hidden state h output at time t-1 t-1 The weighted sum of is activated; where: the forget gate f t , input gate i t and output gate o t Defined as formula (4)-formula (6):

[0037] f t =Sigmoid(W f ·h t-1 +W f ·x t +b f ) (4)

[0038] i t =Sigmoid(W i ·h t-1 +W i ·x t +b i ) (5)

[0039] o t =Sigmoid(W o ·h t-1 +W o ·x t +b o ) (6)

[0040] Where: W f ,W i ,W o are the weight matrices of the forget gate, input gate, and output gate, respectively, and b f ,b i ,b o is its corresponding bias. Sigmoid represents a smooth, easy-to-derive activation function commonly used in neural networks, mapping the input of the function to the interval [0,1]. The forget gate f t Decide whether to keep the state c of the LSTM network at the previous moment t-1 , input gate i t Decide whether to use the current input to update the state of the long short-term memory network, output gate o t Decide whether to change the current state h t Passed to the next iteration; by the state c of the latest unit t The final output of the long short-term memory network is determined by the following definition:

[0041]

[0042]

[0043]

[0044] in: is the new gate input at time t, c t-1 is the gate state at the previous time t-1, W c ,b c are the state matrix and bias of the input unit, tanh is the activation function, Represents an element-wise multiplication operation.

[0045] Furthermore, in step S24, the hidden state h of the long short-term memory network LSTM obtained in step S23 ist Before performing linear transformation to obtain the final output, non-image information is incorporated into the long short-term memory network LSTM structure: the three Euler rotation angle data measured by the IMU device are included The matrix T IMU Connect to the hidden state h output by the network at time t t The relative transformation matrix T is obtained from kt =[h t ,T IMU ], and finally the LSTM network outputs 6 values ​​including 3 degrees of freedom for translation along the x, y, and z axes and 3 degrees of freedom for rotation around the x, y, and z axes It is obtained by the following formula (10):

[0046]

[0047] Among them: The hidden state h corresponding to time t contains the relative position transformation information of consecutive adjacent frames t After connecting with the additional direction information provided by IMU, a linear transformation is performed, and then the nonlinear activation function ReLU is applied. The left saturation characteristic of the ReLU function with sparse activation can solve the problem of gradient disappearance during neural network training. pt ,b pt are the weight matrix and bias parameters respectively.

[0048] Furthermore, in step S25, the relative transformation matrix T of the kth interval frame is obtained by using the ultrasound three-dimensional structure reconstruction method of the present invention in step S24. k Calculate the mean absolute parameter error, maximum center error, and length error to judge the reconstruction effect; at the same time, use the final 6-value result parameter obtained in step S24 Calculate the overall loss function value, update the model training parameters by calculating the gradient, and continuously optimize to find the best ultrasound 3D structure reconstruction effect;

[0049] The mean absolute parameter error is calculated by averaging the absolute difference between the relative transformation parameters estimated at the kth interval of adjacent frames in the scan and the true position label: Where:|||| F represents the Frobenius norm, It represents the inverse matrix of the relative transformation matrix of two adjacent frames of ultrasound images in the kth segment. represents the real position conversion label matrix corresponding to the kth interval, E is the unit matrix; the final drift of each scan is calculated as the distance between the center position of the last frame image predicted by the probe trajectory and the real label in space: in: It represents the center position coordinates of the last frame image predicted for the reconstruction of the three-dimensional structure, i.e. the Mth frame. Indicates the center position coordinates of the last frame of the real spatial position label corresponding to the data acquisition; the maximum center error is the maximum distance between the predicted center and the real center of the kth frame of each scan: in: Indicates the center position coordinates of the k-th frame image predicted during the reconstruction of the three-dimensional structure, It represents the center position coordinates of the kth frame of the corresponding real space position label in data acquisition; the length error is the difference between the distance from the head frame to the last frame in the probe trajectory estimation and the real trajectory: in: Represents the position coordinates of the center point of the first frame of ultrasound image in the prediction data of the reconstruction work, Refers to the coordinates of the center point of the head frame in the corresponding real space label;

[0050] A set of true spatial position labels for ordered ultrasound image sequences For each ultrasound image sequence, the average parameters are calculated according to formula (11):

[0051]

[0052] in: is the average value of the true relative transformation matrix label parameters of this ultrasound sequence, Represents the true relative transformation matrix label of the ultrasound image between adjacent frames in the i-th segment; To optimize the 3D reconstruction results, the overall loss function Loss of the network model is designed, which includes two loss functions, namely, mean square error MSE and case-related loss Loss_corr; the mean square error MSE loss is defined as: and The calculation process is consistent, and the processed output parameters of all adjacent frames of the ultrasound sequence are averaged to obtain The case correlation loss Loss_corr calculates the correlation coefficient between the motion estimate of each degree of freedom and the spatial true label, expressed as formula (12):

[0053]

[0054] Where: j represents a 6-value parameter or The corresponding value of the j-th degree of freedom, Cov and σ are the covariance calculation and standard deviation calculation respectively; the overall loss function of the network model for reconstructing the three-dimensional structure of ultrasound is calculated by the mean square error MSE and the individual correlation loss Loss_corr: Loss = MSE + Loss_corr.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] The present invention reconstructs a two-dimensional ultrasonic image sequence into a three-dimensional image according to the trajectory of the ultrasonic probe, replaces the traditional convolutional neural network that processes data with a long short-term memory network that is good at extracting temporal features, uses an optical flow algorithm to extract the relative motion component between the two frames of images before taking adjacent ultrasonic frames as the network input, fuses the image data and motion information to construct a four-channel network input, connects the network hidden state obtained at the corresponding moment input with the auxiliary data collected by the IMU device, and then performs a linear transformation, finally obtaining a matrix output encoding rigid motion at the corresponding moment, and at the same time, the hidden state will be transferred to the next moment, and the corresponding unit output will be obtained in turn on the network input at the next moment. Combined with the average absolute parameter error, maximum center error, length error and overall loss function of the ultrasonic image sequence between unit continuous frames, the long short-term memory network is trained using adjacent ultrasonic frame sets, optical flow field motion components and IMU auxiliary data, and a three-dimensional image reconstructed by the ultrasonic probe trajectory is generated after obtaining a series of matrix outputs. Compared with a series of reconstruction methods with the help of external positioning equipment, the present invention uses a two-dimensional ultrasonic probe for disease diagnosis to obtain a three-dimensional ultrasonic structure, assists in surgical guidance, is low-cost, easy to operate, and improves the accuracy of positioning, and is more suitable for real clinical environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flow chart for quickly acquiring ultrasonic three-dimensional structure in the present invention;

[0058] Figure 2 It is a design diagram of the image interface of the reconstruction system in the present invention;

[0059] Figure 3 It is an effect overview diagram of the present invention;

[0060] Figure 4 It is the overall model structure diagram of the reconstruction system in the present invention. DETAILED DESCRIPTION

[0061] The present invention will be further described below in conjunction with embodiments.

[0062] Example 1

[0063] like Figure 1 As shown, a method for a rapid ultrasonic three-dimensional structure acquisition system that does not rely on an external positioning device comprises the following steps:

[0064] S1. Use low-cost common ultrasound probes to collect spatial positioning information Two-dimensional ultrasound image sequence (I n ) n , and rigidly install an inertial measurement unit (IMU) on the probe to collect auxiliary data T IMU, used to train a patient to perform three-dimensional reconstruction to assist surgical guidance through computer-assisted pure image analysis technology without the aid of external positioning equipment;

[0065] As a preferred embodiment, in step S1, a two-dimensional ultrasonic image containing actual content, spatial position information, and auxiliary data collected by IMU are obtained, wherein the method for recording a two-dimensional ultrasonic image sequence is as follows:

[0066] Three-dimensional optical tracking ultrasound scans were captured using a Cicada ultrasound machine, using a linear probe with 128 elements at a frequency of 5 MHz to generate images consisting of 256 scan lines, capturing ultrasound images at a rate of approximately 35 frames per second without any filtering or scan line conversion. This scanning method is intended to balance the problem that the captured ultrasound images lose a lot of details due to subsequent processing and noise filtering, and the problem that the raw radio frequency (RF) data containing rich information is not accessible to all ultrasound systems. For the convenience of preprocessing, all ultrasound images were resampled to a fixed 0.3mm isotropic resolution, matching the speckle scale of the image.

[0067] A typical ultrasound scanning video contains rich contextual information of more than two adjacent ultrasound image frames. Using only two consecutive ultrasound image frames may lose temporal information, resulting in more inaccurate reconstruction results. Therefore, a two-dimensional ultrasound image sequence is needed to provide a more general and representative motion trajectory of the ultrasound probe, so as to more accurately assist surgical guidance.

[0068] In step S1, the real space position label corresponding to the ultrasound image is tracked and recorded. The method is as follows:

[0069] To ensure that the training data matches the acquisition conditions of the clinical trial, a handheld ultrasound probe equipped with a high-precision external positioning device is used for scanning. Since the electromagnetic positioning system will produce drift and severe jitter, the present invention uses the Stryker NAV3 originally used for surgical navigation. TM The optical positioning system of the camera, after all spatial images are calibrated by the sensor, records the absolute positioning accuracy of the real spatial position tags within a translation range of about 0.2mm. Due to the digital interface and appropriate clock synchronization of the ultrasound research system, the time calibration is guaranteed not to cause jitter and drift, so the real position tags in adjacent ultrasound frames have high accuracy. Due to the different acquisition rates of the ultrasound and tracking systems, linear interpolation and spherical interpolation are performed for translation and rotation operations respectively.

[0070] In step S1, the inertial measurement unit IMU integrated hardware acquires auxiliary data T IMU The method is as follows:

[0071] An inertial measurement unit (IMU) refers to a chip that integrates a three-dimensional magnetometer, a rate gyroscope, and an accelerometer in a size of less than 1 cm. It can be installed in a mobile phone or other handheld device independent of external hardware, and its small size advantage will not reduce the user experience. The IMU can provide the movement direction and acceleration information of the ultrasonic probe. Due to the low-speed movement of the probe during the scanning and acquisition process, there is a large amount of noise in the acceleration information. Therefore, the present invention only uses the direction information collected by the IMU. A sensor with an inertial measurement unit is used to estimate the orientation of the ultrasonic probe, and rotation measurement data can be obtained. Due to the extremely low signal-to-noise ratio and the need for double numerical integration, the data in the accelerometer is insufficient to calculate the scanning trajectory. Therefore, the data captured by the inertial measurement unit is used as auxiliary data of the method of the present invention to reconstruct the trajectory of the ultrasonic probe.

[0072] Using sensors and optically positioned real-space position tags, various freehand ultrasonic movements are recorded. The inertial measurement unit (IMU) used in the present invention can directly return useful representations of the Euler direction. The angle data is recorded in the 3×3 rotation matrix T IMU In which: Respectively represent the rotation angle of the IMU device around the x, y, and z coordinate axes, and no fusion with the sensor and additional post-processing are required. The camera coordinate system is established with the optical tracking scanning camera as the origin of the coordinate system, and the auxiliary data T IMU The data is collected in the world coordinate system, where the spatial tags recorded by optical tracking have been calibrated with the ultrasound image. Therefore, the auxiliary data T obtained by the IMU device is IMU The real label information collected After time and space calibration, it can be used in the method of reconstructing ultrasound three-dimensional structure of this patent. Formula (101) maximizes the use of the real spatial position label value Auxiliary data T detected by IMU IMU Time calibration is performed by correlation between:

[0073]

[0074] Where: t IMU→GT Represents the real space position label at time t With auxiliary data T IMU The correlation between them is the largest, so the best calibration effect is achieved in the time dimension; ω IMU Represents the measured angular velocity of the gyroscope sensor in the IMU, ω GT is the angular velocity of the real spatial position tag; in order to align the directions of the two tracking coordinate systems, the auxiliary data T IMU Real-space labeling with optical tracking Perform spatial calibration according to formula (102):

[0075]

[0076] Where: For the kth interval of a certain ultrasound sequence: matrix T GT contains the rotation part of the real space position label, and the matrix T IMU Contains the Euler orientation parameters captured by the IMU. C and R represent the constant calibration matrix and registration matrix respectively: C describes the local transformation between the IMU sensor and the optical tracking target, and R describes the global transformation between the world coordinate system referenced by the IMU and the optical tracking camera; the two matrices C and R are minimized by using the MLSL algorithm T GT The absolute Euler residual between the right side of equation (2) is used to perform numerical global optimization.

[0077] In order to make the 3D structure reconstruction system in the present invention interactive, a graphical interface that supports ultrasound image acquisition and 3D structure reconstruction of different parts of the human body will be designed. At the same time, the interface can provide buttons for calculating relative motion components using the optical flow method and IMU-assisted data acquisition to improve the quality of the reconstructed structure. Figure 2 As shown, reconstructing the three-dimensional structure of different tissues and organs of the human body to assist in surgical guidance mainly includes three parts: data acquisition, training model and reconstructed structure display.

[0078] S2: The spatial positioning information obtained in step S1 is Two-dimensional ultrasound image sequence (I n ) n Reconstruction into three-dimensional ultrasound structure by a method that does not rely on external magnetic positioning or optical positioning equipment;

[0079] The nth ultrasound sequence (I n ) n Two adjacent frames of images (I n-1 ,I n ) as units as input, multiple groups of inputs are trained by the long short-term memory network method of the present invention to obtain output result parameters and thus reconstruct the three-dimensional ultrasound structure.

[0080] In step S2, computer image processing technology is first used to superimpose the collected ultrasound sequence with two consecutive frames of images and the optical flow motion components generated between them as units, and the long short-term memory network is used as the basic framework. After fusing the IMU auxiliary data, the relative motion transformation parameters between consecutive frames are directly estimated, and the self-defined loss function is calculated by comparing with the real space position label. The loss function is minimized to obtain the optimal three-dimensional reconstruction effect, thereby constructing a model of the three-dimensional ultrasound structure of the target tissue corresponding to the input data. Then the external positioning device is removed, and the real-time data collected by the two-dimensional ultrasound probe for disease diagnosis is based on the sensorless reconstruction method designed above. The doctor can quickly obtain the three-dimensional ultrasound structure of the target tissue to achieve surgical guidance, while allowing him to move the ultrasound probe with fewer constraints to avoid blocking the tracking signal and cost issues.

[0081] The reconstruction method in this paper includes the preprocessing of ultrasound images and optical flow, and the acquisition of auxiliary data by IMU sensors, which makes the modeling parameters more robust and accurate, so that more realistic scanning motions can be targeted, including important rotations or translations in or out of the plane, which can meet the requirements of clinical trials. At the same time, ordinary two-dimensional ultrasound probes for disease diagnosis are used in surgical guidance, and the acquired data is reconstructed into a three-dimensional structure.

[0082] Example 2

[0083] The method of the present invention is based on inter-frame motion estimation, and a network model is trained to estimate the translation and rotation of the probe between consecutive frames. At the same time, a small inertial measurement unit installed on the probe can be used to assist in modeling, such as Figure 3 The method of the ultrasonic three-dimensional structure rapid acquisition system which does not rely on external positioning equipment comprises the following steps:

[0084] Step S21: The image sequence set (I n ) n According to the two adjacent ultrasound slices (I n-1 ,I n ) as a unit to segment each ultrasound image sequence, then each ultrasound image sequence can be divided into N segments, where I n-1 and I n They correspond to the n-1th and nth frames in a single image sequence respectively;

[0085] Step S22: Input the unit divided in step S21 (I n-1 ,I n ) are processed by optical flow algorithm to obtain the relative motion components d corresponding to adjacent ultrasound slices. flow ;

[0086] Step S23: The motion component d obtained in step S22 isflow The unit input (I n-1 ,I n ) are connected and used together as the superposition input of the long short-term memory network model LSTM based on the sequence method;

[0087] Step S24: Send the superimposed input in step S23 to the long short-term memory network model LSTM to obtain the hidden state h at the corresponding time t t After that, the auxiliary data T collected by IMU in Example 1 IMU Connect, and then output 6 value result parameters after linear transformation To represent the rigid motion between adjacent frames, where: T IMU Depend on It consists of three parameters, which represent the rotation angle of the IMU device around the x, y, and z axes at time t. They are the three degrees of freedom of translation along the x, y, and z coordinate axes and the three degrees of freedom of rotation around the coordinate axes; the N-1 adjacent slices remaining in step 1 are used as the second, third, ..., N-1 inputs of the long short-term memory network model LSTM in sequence, and the corresponding output parameters are obtained in turn to reconstruct the three-dimensional structure;

[0088] Step S25: for the adjacent frames (I) of the unit ultrasound image in step S24, n-1 ,I n ) Calculate the mean absolute parameter error, maximum center error, and length error for the nth ordered ultrasound image sequence (I n ) n The overall loss function Loss value of the entire network model is calculated; the model of the present invention is trained by the loss function and the optimization method, and the trained model is directly used to process two-dimensional ultrasound slices, thereby being used for three-dimensional trajectory reconstruction of the ultrasound probe, wherein Figure 4 The model structure diagram of the present invention is shown.

[0089] The present invention reconstructs a two-dimensional ultrasonic image sequence into a three-dimensional image according to the trajectory of the ultrasonic probe, replaces the traditional convolutional neural network that processes data with a long short-term memory network that is good at extracting temporal features, uses an optical flow algorithm to extract the relative motion component between the two frames of images before taking adjacent ultrasonic frames as the network input, fuses the image data and motion information to construct a four-channel network input, connects the network hidden state obtained at the corresponding moment input with the auxiliary data collected by the IMU, and then performs a linear transformation, finally obtaining a matrix output encoding rigid motion at the corresponding moment, and at the same time, the hidden state will be transferred to the next moment, and the corresponding unit output will be obtained in turn on the network input at the next moment. Combined with the average absolute parameter error, maximum center error, length error and overall loss function of the ultrasonic image sequence between unit continuous frames, the long short-term memory network is trained using the adjacent ultrasonic frame set, the optical flow field motion component and the IMU auxiliary data, and a three-dimensional image reconstructed by the ultrasonic probe trajectory is generated after obtaining a series of matrix outputs. Compared with a series of reconstruction methods with the help of external positioning equipment, the present invention uses a two-dimensional ultrasonic probe for disease diagnosis to obtain a three-dimensional ultrasonic structure, assists in surgical guidance, is low-cost, easy to operate and improves the accuracy of positioning, and is more suitable for real clinical environments.

[0090] As a preferred solution, in step S21, the two-dimensional ultrasound image sequence (I n ) n The present invention attempts to reconstruct the nth frame of ultrasound image I n The corresponding time t n The trajectory of the ultrasound probe is represented by a matrix sequence (T n ) n Encoding: The trajectory can be encoded into a homogeneous matrix and then through a 6-value parameter vector = represents the relative transformation corresponding to the trajectory, where 3 degrees of freedom represent translation along the x, y, and z axes, and 3 degrees of freedom represent rotation around the x, y, and z axes. Since there is no absolute coordinate system, the present invention only considers relative changes: it estimates that each frame I n Compared with the previous frame ultrasound image I n-1 Relative transformation of T n-1→n represents the n-1th transformation T of the ultrasound image sequence n-1 With the nth conversion T n A rigid relative transformation, T n-1→n Available 6-value vector Parameterization, Represents the matrix T n-1 The inverse matrix of n-1 ,I n), train a suitable long short-term memory network LSTM to act on adjacent frames, through LSTM (I n-1 ,I n ) process estimates the outcome parameters that represent relative changes:

[0091]

[0092] Design a long short-term memory network model LSTM, iterate all pairs of consecutive adjacent frame ultrasound images, and use the chain rule to estimate T using the previous relative transformation motion 2→3 ,T 3→4 ,...,T k-1→k ,....Calculate any matrix T n-1→n , where: T 1→2 Refers to the relative transformation matrix from the first frame to the second frame of the ultrasound sequence, T 2→3 ,T 3→4 ,...,T k-1→k ,....with T 1→2 The meaning is consistent; the ultrasound image sequence (I n ) n The units of two consecutive frames are divided into N segments, and the rigid motion is divided into t x ,t y ,t z The three parameters represent the translational motion and θ x ,θ y ,θ z The 3 Euler angles represent the rotation components:

[0093] As a preferred solution, in step S22, it is known from practice that providing additional information as other input is beneficial to the reconstruction task, so the present invention uses the optical flow field algorithm to process the continuous adjacent frames (I n-1 ,I n ). The continuous adjacent frames (I ) divided in step S21 are processed using the optical flow algorithm. n-1 ,I n ). Assume that the previous frame I n-1 A pixel u=[u x ,u y ] and the next frame I n The pixel v = [u x +d x ,u y +d y ] matches, that is, the grayscale difference between pixels u and v is the smallest, and u and v correspond to each other one by one, where: [u x ,u y ] represents the x-axis and y-axis coordinates of pixel u in the two-dimensional plane of the image. Then d flow =[d x ,dy ] is I n-1 Pixel u moves to I n The relative motion component of pixel v, [d x ,d y ] represents the movement length of pixel u along the x and y axes of the two-dimensional plane when it moves to v. n (u x ,u y ) means that the coordinates of the point in the nth frame are (u x ,u y ) pixel value size, H n-1 (x,y) means the coordinates of the point in the n-1 frame are (u x +d x ,u y +d y ) pixel value, then I n-1 with I n The relationship between the pixel values ​​of the image frame after optical flow processing is as shown in formula (2):

[0094] H n (u x ,u y )=H n-1 (u x +d x ,u y +d y ) (2)

[0095] As a preferred solution, in step S23, each ultrasound image sequence (I n ) n Divide into N segments according to the adjacent frames and process them sequentially: divide the adjacent frames (I n-1 ,I n ) and the corresponding relative motion component d extracted in step S22 flow After fusion according to equation (3), we get the input X n-1 , directly fed into the long short-term memory network model LSTM:

[0096] X n-1 =[I n-1 ,I n ,d flow ] (3)

[0097] The LSTM network has three control gates f at time t t 、i t and t Each control gate essentially uses a nonlinear function Sigmoid, which is determined by the input x corresponding to the current time t t And the hidden state h output at time t-1t-1 The weighted sum of is activated; where: the forget gate f t , input gate i t and output gate o t Defined as formula (4)-formula (6):

[0098] f t =Sigmoid(W f ·h t-1 +W f ·x t +b f ) (4)

[0099] i t =Sigmoid(W i ·h t-1 +W i ·x t +b i ) (5)

[0100] o t =Sigmoid(W o ·h t-1 +W o ·x t +b o ) (6)

[0101] Where: W f ,W i ,W o are the weight matrices of the forget gate, input gate, and output gate, respectively, and b f ,b i ,b o is its corresponding bias. Sigmoid represents a smooth, easy-to-derive activation function commonly used in neural networks, which can map the input of the function to the interval [0,1]. The forget gate f t Decide whether to keep the state c of the LSTM network at the previous moment t-1 , input gate i t Decide whether to use the current input to update the state of the long short-term memory network, output gate o t Decide whether to change the current state h t Passed to the next iteration; by the state c of the latest unit t The final output of the long short-term memory network is determined by the following definition:

[0102]

[0103]

[0104]

[0105] in: is the new gate input at time t, c t-1 is the gate state at the previous time t-1, W c ,b c are the state matrix and bias of the input unit, tanh is the activation function, Represents an element-wise multiplication operation.

[0106] As a preferred solution, in step S24, the hidden state h of the long short-term memory network LSTM obtained in step S23 is t Before performing linear transformation to obtain the final output, non-image information is incorporated into the long short-term memory network LSTM structure: Figure 4 As shown, the three Euler rotation angle data measured by the IMU device are The matrix T IMU Connect to the hidden state h output by the network at time t t The relative transformation matrix T is obtained from kt =[h t ,T IMU ], and finally the LSTM network outputs a 6-value vector containing 3 degrees of freedom for translation along the x, y, and z axes and 3 degrees of freedom for rotation around the x, y, and z axes It is obtained by the following formula (10):

[0107]

[0108] Among them: The hidden state h corresponding to time t contains the relative position transformation information of consecutive adjacent frames t After connecting with the additional directional information provided by the IMU, a linear transformation is performed, and then the nonlinear activation function ReLU is applied. The left saturation characteristic of the ReLU function with sparse activation can solve the problem of gradient disappearance during neural network training to a certain extent. are the weight matrix and bias parameters respectively.

[0109] As a preferred solution, in step S25, in order to make the three-dimensional reconstruction model of the two-dimensional ultrasound image sequence more accurate, the difference between the network output and the real space position label is minimized when training the network designed by the present invention. In order to achieve the purpose of adapting to clinical relevant data, it is also necessary to capture and quantify different types of error values. Therefore, the relative transformation matrix T of the kth interval frame is obtained by the ultrasound three-dimensional structure reconstruction method of the present invention in step S24. k Calculate the mean absolute parameter error, maximum center error, and length error to judge the reconstruction effect; at the same time, use the final 6-value result parameter obtained in step ⑷ Calculate the overall loss function value, update the model's training parameters by calculating the gradient, and continuously optimize to find the best ultrasound three-dimensional structure reconstruction effect.

[0110] The mean absolute parameter error is calculated by averaging the absolute difference between the relative transformation parameters estimated at the kth interval of adjacent frames in the scan and the true position label: where |||| F represents the Frobenius norm, It represents the inverse matrix of the relative transformation matrix of two adjacent frames of ultrasound images in the kth segment. represents the real position conversion label matrix corresponding to the kth interval, and E is the identity matrix. The final drift of each scan is calculated as the distance between the center position of the last frame image predicted by the probe trajectory and the real label in space: in represents the center position coordinates of the last frame image, i.e., the Mth frame, predicted by the present invention for reconstructing the three-dimensional structure, Indicates the center position coordinates of the last frame of the real spatial position label corresponding to the data acquisition; the maximum center error is the maximum distance between the predicted center and the real center of the kth frame of each scan: in Indicates the center position coordinates of the k-th frame image predicted during the reconstruction of the three-dimensional structure, Represents the center position coordinates of the kth frame of the corresponding real space position label in data acquisition. The length error is the difference between the distance from the head frame to the last frame in the probe trajectory estimation and the real trajectory: This difference is needed in clinical trials, such as one of the goals of reconstructing a 3D model is to draw the vascular tree and measure the length of each vessel. Represents the position coordinates of the center point of the first frame of ultrasound image in the prediction data of the reconstruction work, Refers to the coordinates of the center point of the head frame in the corresponding real space label.

[0111] A set of true spatial position labels for ordered ultrasound image sequences For each ultrasound image sequence, the average parameters are calculated according to formula (11):

[0112]

[0113] in: is the average value of the true relative transformation matrix label parameters of this ultrasound sequence, Represents the true relative transformation matrix label of the ultrasound image in the i-th segment of adjacent frame intervals. The average value of the true label parameters is used. The reason is that the motion between two frames is very small, so the average value can effectively smooth the noise in the probe motion.

[0114] In order to optimize the three-dimensional reconstruction results of ultrasound image sequences, the network model in the present invention is designed to include two loss functions, one of which is the mean square error (MSE) loss commonly used in deep regression problems: and The calculation process is consistent, and the processed output parameters of all adjacent frames of the ultrasound sequence are averaged to obtain However, if only the MSE loss is used to smoothly estimate the probe motion, the trained network tends to remember the general pattern of how the clinician moves the probe, that is, the average trajectory of the ultrasound probe, and cannot adapt to the rotation, drift, etc. that occur during scanning. Therefore, a case-related loss Loss_corr based on the Pearson correlation coefficient is introduced to emphasize the specific motion pattern of the probe scanning. The correlation coefficient between the motion estimate of each degree of freedom and the true spatial label is calculated and expressed as formula (12):

[0115]

[0116] Where: j represents a 6-value parameter or The corresponding value of the j-th degree of freedom, Cov and σ are respectively the covariance calculation and the standard deviation calculation. The overall loss function of the network model for rapid reconstruction of ultrasonic three-dimensional structure in the present invention is calculated by the mean square error MSE and the case-related loss Loss_corr: Loss = MSE + Loss_corr.

[0117] According to the above overall loss function, the parameters in the long short-term memory network with optical flow method and IMU auxiliary data are updated until the network reaches convergence, and the mean absolute parameter error, maximum center error, and length error are quantified and evaluated to obtain the best reconstruction effect. The method of the present invention can be used as a tool for establishing a three-dimensional stereo model of image sequence data collected by a two-dimensional ultrasound probe without the help of an external positioning device. The invention of this tool will promote the development of ultrasound imaging technology in the medical field, reduce the risk of surgery, reduce the trauma of patients and the labor intensity of doctors.

[0118] The method of the present invention is based on inter-frame motion estimation, training a network model to estimate the translation and rotation of the probe between consecutive frames, and optionally using a small inertial measurement unit installed on the probe, such as Figure 3 Shown is an overview diagram of the present invention.

[0119] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for a rapid ultrasonic three-dimensional structure acquisition system that does not rely on external positioning equipment, characterized in that: Using a common two-dimensional ultrasound probe to acquire image data to construct a three-dimensional model of the target organ includes the following steps: S1. Use ordinary ultrasound probe to collect spatial positioning information Two-dimensional ultrasound image sequence (I n ) n , and rigidly install an inertial measurement unit (IMU) on the probe to collect auxiliary data T IMU ; S2, the spatial positioning information obtained in step S1 Two-dimensional ultrasound image sequence (I n ) n Reconstruction into three-dimensional ultrasound structure by a method that does not rely on external magnetic positioning or optical positioning equipment; S21, the image sequence set (I n ) n According to the two adjacent ultrasound slices (I n-1 ,I n ) is used as a unit to segment each ultrasound image sequence, then each ultrasound image sequence is divided into N segments, where: n-1 and I n They correspond to the n-1th and nth frames in a single image sequence respectively; S22, the unit input after the division in step S21 (I n-1 ,I n ) are processed by optical flow algorithm to obtain the relative motion components d corresponding to adjacent ultrasound slices. flow ; S23, the motion component d obtained in step S22 flow The unit input (I n-1 ,I n ) are connected and used together as the superposition input of the long short-term memory network model LSTM based on the sequence method; S24, sending the superimposed input in step S23 into the long short-term memory network model LSTM to obtain the hidden state h corresponding to the time t t After that, the auxiliary data T collected by IMU in step S1 IMU Connect, and then output 6 value result parameters after linear transformation To represent the rigid motion between adjacent frames, where: T IMU Depend on It consists of three parameters: Respectively represent the angles of rotation of the IMU device around the x, y, and z axes at time t. The three degrees of freedom are respectively the three translation degrees of freedom along the x, y, and z coordinate axes and the three degrees of freedom of rotation around the coordinate axes; the N-1 adjacent slices remaining in step S21 are sequentially used as the second, third, ..., N-1 inputs of the long short-term memory network model LSTM, and the corresponding output parameters are obtained in turn to reconstruct the three-dimensional structure; S25, for the adjacent frames (I) of the unit ultrasound image in step S24, n-1 ,I n ) Calculate the mean absolute parameter error, maximum center error, and length error for the nth ordered ultrasound image sequence (I n ) n The overall loss function Loss value of the entire network model is calculated; the model of the present invention is trained by the loss function and the optimization method, and the trained model is directly used to process two-dimensional ultrasonic slices, thereby being used for three-dimensional trajectory reconstruction of the ultrasonic probe.

2. The method of the ultrasonic three-dimensional structure rapid acquisition system that does not rely on external positioning equipment according to claim 1 is characterized in that: In step S1, a two-dimensional ultrasound image sequence (I n ) n The method is as follows: A Cicada ultrasound machine was used to capture 3D optically tracked ultrasound scans, using a 128-element linear probe at 5 MHz to generate images consisting of 256 scan lines, capturing ultrasound images at 35 frames per second.

3. The method of the ultrasonic three-dimensional structure rapid acquisition system that does not rely on external positioning equipment according to claim 2 is characterized in that: In step S1, the real space position label corresponding to the ultrasound image is tracked and recorded. The method is as follows: Using Stryker NAV3 TM The optical positioning system of the camera records the real spatial position labels of all ultrasound images after sensor calibration The absolute positioning accuracy is within 0.2mm translation range.

4. The method of the ultrasonic three-dimensional structure rapid acquisition system that does not rely on external positioning equipment according to claim 3 is characterized in that: In step S1, the inertial measurement unit IMU integrated hardware acquires auxiliary data T IMU The method is as follows: Inertial measurement unit (IMU) refers to a chip with a size of less than 1 cm that integrates a three-dimensional magnetometer, a rate gyroscope, and an accelerometer. It is installed in a mobile phone or other handheld device to provide the movement direction and acceleration information of the ultrasound probe, and only uses the collected direction information; a sensor with IMU is used to estimate the orientation of the ultrasound probe to obtain rotation measurement data T IMU Used as auxiliary data to reconstruct the trajectory of the ultrasound probe and obtain the ultrasound three-dimensional structure; IMU devices directly return useful representations of Euler directions The angle data is recorded in the 3×3 rotation matrix T IMU In which: Respectively represent the rotation angle of the IMU device around the x, y, and z coordinate axes; due to the real spatial position label The camera coordinate system is established with the optical tracking scanning camera as the origin of the coordinate system, and the auxiliary data T IMU The data is collected in the world coordinate system, where the spatial tags recorded by optical tracking have been calibrated with the ultrasound image. Therefore, the auxiliary data T obtained by the IMU device is IMU The real label information collected After time and space calibration, it can be used in the method of reconstructing ultrasound three-dimensional structure; Formula (101) maximizes the use of the real spatial position label value Auxiliary data T detected by IMU IMU Time calibration is performed by correlation between: Where: t IMU→GT Represents the real space position label at time t With auxiliary data T IMU The correlation between them is the largest, so the best calibration effect is achieved in the time dimension; ω IMU Represents the measured angular velocity of the gyroscope sensor in the IMU, ω GT is the angular velocity of the real spatial position tag; in order to align the directions of the two tracking coordinate systems, the auxiliary data T IMU Real-space labeling with optical tracking Perform spatial calibration according to formula (102): Among them: For the kth interval of a certain ultrasound sequence: matrix contains the rotation part of the real space position label, and the matrix Contains the Euler direction parameters captured by the IMU; C, R represent the constant calibration matrix and registration matrix respectively: C describes the local transformation between the IMU sensor and the optical tracking target, and R describes the global transformation between the world coordinate system referenced by the IMU and the optical tracking camera; the two matrices C and R are minimized by using the MLSL algorithm The absolute Euler residual between the right side of equation (102) and the right side of equation (102) is used to perform numerical global optimization.

5. The method of the ultrasonic three-dimensional structure rapid acquisition system that does not rely on external positioning equipment according to claim 4 is characterized in that: The step S21 uses the two-dimensional ultrasound image sequence (I n ) n , trying to reconstruct the nth frame of ultrasound image I n The corresponding time t n The trajectory of the ultrasound probe is represented by a matrix sequence (T n ) n Encoding: Encode the trajectory into a homogeneous matrix and then pass the 6-value result vector Represents the relative transformation corresponding to the trajectory, where 6 values: 3 degrees of freedom represent translation along the x, y, and z axes, and 3 degrees of freedom represent rotation around the x, y, and z axes; since there is no absolute coordinate system, only relative changes are considered: it is estimated that each frame I n Compared with the previous frame ultrasound image I n-1 Relative transformation of T n-1→n represents the n-1th transformation T of the ultrasound image sequence n-1 With the nth conversion T n A rigid relative transformation, T n-1→n With 6-value vector Parameterization, Represents the matrix T n-1 The inverse matrix of a given ultrasound sequence is n-1 ,I n ), train a suitable long short-term memory network LSTM to act on adjacent frames, through LSTM (I n-1 ,I n ) process estimates the outcome parameters that represent relative changes: Design a long short-term memory network model LSTM, iterate all pairs of consecutive adjacent frame ultrasound images, and use the chain rule to estimate T using the previous relative transformation motion 2→3 ,T 3→4 ,...,T k-1→k ,....Calculate any matrix T n-1→n , where: T 1→2 Refers to the relative transformation matrix from the first frame to the second frame of the ultrasound sequence, T 2→3 ,T 3→4 ,...,T k-1→k ,....with T 1→2 The meaning is consistent; the ultrasound image sequence (I n ) n The units of two consecutive frames are divided into N segments, and the rigid motion is divided into t x ,t y ,t z The three parameters represent the translational motion and θ x ,θ y ,θ z The 3 Euler angles represent the rotation components:

6. The method of the ultrasonic three-dimensional structure rapid acquisition system independent of external positioning equipment according to claim 5, characterized in that: In step S22, the continuous adjacent frames (I n-1 ,I n ); Assume that the previous frame I n-1 A pixel u=[u x ,u y ] and the next frame I n The pixel v = [u x +d x ,u y +d y ] matches, that is, the grayscale difference between pixels u and v is the smallest, and u and v correspond to each other one by one, where: [u x ,u y ] represents the x and y axis coordinates of pixel u in the two-dimensional plane of the image; d flow =[d x ,d y ] is I n-1 Pixel u moves to I n The relative motion component of pixel v, [d x ,d y ] represents the movement length of pixel u along the x and y axes of the two-dimensional plane when it moves to v; H n (u x ,u y ) means that the coordinates of the point in the nth frame are (u x ,u y ) pixel value size, H n-1 (x,y) means the coordinates of the point in the n-1 frame are (u x +d x ,u y +d y ) pixel value, then I n-1 with I n The relationship between the pixel values ​​of the image frame after optical flow processing is as shown in formula (2): H n (u x ,u y )=H n-1 (u x +d x ,u y +d y ) (2)。 7. The method of the ultrasonic three-dimensional structure rapid acquisition system independent of external positioning equipment according to claim 6, characterized in that: In step S23, each ultrasound image sequence (I n ) n Divide into N segments according to the adjacent frames and process them sequentially: divide the adjacent frames (I n-1 ,I n ) and the corresponding relative motion component d extracted in step S22 flow After fusion according to formula (3), we get the input X n-1 , directly fed into the long short-term memory network model LSTM: X n-1 =[I n-1 ,I n ,d flow ] (3) The LSTM network has three control gates f at time t t 、i t and t Each control gate essentially uses a nonlinear function Sigmoid, which is determined by the input x corresponding to the current time t t And the hidden state h output at time t-1 t-1 The weighted sum of is activated; where: the forget gate f t , input gate i t and output gate o t Defined as formula (4)-formula (6): f t =Sigmoid(W f ·h t-1 +W f ·x t +b f ) (4) i t =Sigmoid(W i ·h t-1 +W i ·x t +b i ) (5) o t =Sigmoid(W o ·h t-1 +W o ·x t +b o ) (6) Where: W f ,W i ,W o are the weight matrices of the forget gate, input gate, and output gate, respectively, and b f ,b i ,b o is its corresponding bias. Sigmoid represents a smooth, easy-to-derive activation function commonly used in neural networks, mapping the input of the function to the interval [0,1]. The forget gate f t Decide whether to keep the state c of the LSTM network at the previous moment t-1 , input gate i t Decide whether to use the current input to update the state of the long short-term memory network, output gate o t Decide whether to change the current state h t Passed to the next iteration; by the state c of the latest unit t The final output of the long short-term memory network is determined by the following definition: in: is the new gate input at time t, c t-1 is the gate state at the previous time t-1, W c ,b c are the state matrix and bias of the input unit, tanh is the activation function, Represents an element-wise multiplication operation.

8. The method of the ultrasonic three-dimensional structure rapid acquisition system independent of external positioning equipment according to claim 7, characterized in that: In step S24, the hidden state h of the long short-term memory network LSTM obtained in step S23 is t Before performing linear transformation to obtain the final output, non-image information is incorporated into the long short-term memory network LSTM structure: the three Euler rotation angle data measured by the IMU device are included The matrix T IMU Connect to the hidden state h output by the network at time t t The relative transformation matrix T is obtained from kt =[h t ,T IMU ], and finally the LSTM network outputs a 6-value vector containing 3 degrees of freedom for translation along the x, y, and z axes and 3 degrees of freedom for rotation around the x, y, and z axes According to formula (10), we can get: Among them: The hidden state h corresponding to time t contains the relative position transformation information of consecutive adjacent frames t After connecting with the additional directional information provided by the IMU, a linear transformation is performed, and then the nonlinear activation function ReLU is applied. The left saturation characteristic of the ReLU function with sparse activation can solve the problem of gradient disappearance during neural network training. are the weight matrix and bias parameters respectively.

9. The method of the ultrasonic three-dimensional structure rapid acquisition system independent of external positioning equipment according to claim 8, characterized in that: In step S25, the relative transformation matrix T of the kth interval frame is obtained by using the ultrasound three-dimensional structure reconstruction method of the present invention in step S24. k Calculate the mean absolute parameter error, maximum center error, and length error to judge the reconstruction effect; at the same time, use the final 6-value result parameter obtained in step S24 Calculate the overall loss function value, update the model training parameters by calculating the gradient, and continuously optimize to find the best ultrasound 3D structure reconstruction effect; The mean absolute parameter error is calculated by averaging the absolute difference between the relative transformation parameters estimated at the kth interval of adjacent frames in the scan and the true position label: Where: || || F represents the Frobenius norm, It represents the inverse matrix of the relative transformation matrix of two adjacent frames of ultrasound images in the kth segment. represents the real position conversion label matrix corresponding to the kth interval, E is the unit matrix; the final drift of each scan is calculated as the distance between the center position of the last frame image predicted by the probe trajectory and the real label in space: in: It represents the center position coordinates of the last frame image predicted for the reconstruction of the three-dimensional structure, i.e. the Mth frame. Indicates the center position coordinates of the last frame of the real spatial position label corresponding to the data acquisition; the maximum center error is the maximum distance between the predicted center and the real center of the kth frame of each scan: in: Indicates the center position coordinates of the k-th frame image predicted during the reconstruction of the three-dimensional structure, It represents the center position coordinates of the kth frame of the corresponding real space position label in data acquisition; the length error is the difference between the distance from the head frame to the last frame in the probe trajectory estimation and the real trajectory: in: Represents the position coordinates of the center point of the first frame of ultrasound image in the prediction data of the reconstruction work, Refers to the coordinates of the center point of the head frame in the corresponding real space label; A set of true spatial position labels for ordered ultrasound image sequences For each ultrasound image sequence, the average parameters are calculated according to formula (11): in: is the average value of the true relative transformation matrix label parameters of this ultrasound sequence, The true relative transformation matrix label of the ultrasound image between the i-th adjacent frame intervals; In order to optimize the 3D reconstruction results, the overall loss function Loss of the network model is designed, which contains two loss functions, namely the mean square error MSE and the case correlation loss Loss_corr; the mean square error MSE loss is defined as: and The calculation process is consistent, and the processed output parameters of all adjacent frames of the ultrasound sequence are averaged to obtain The case correlation loss Loss_corr calculates the correlation coefficient between the motion estimate of each degree of freedom and the spatial true label, expressed as formula (12): Where: j represents a 6-value parameter or The corresponding value of the j-th degree of freedom, Cov and σ are the covariance calculation and standard deviation calculation respectively; the overall loss function of the network model for reconstructing the three-dimensional structure of ultrasound is calculated by the mean square error MSE and the individual correlation loss Loss_corr: Loss = MSE + Loss_corr.

Images