Ultrasonic imaging assisted puncture method and system based on mixed reality

Abstract

The invention relates to the field of medical digital, and discloses an ultrasonic imaging auxiliary puncture method and system based on mixed reality, and the method comprises the steps: carrying out the segmentation and reconstruction of a region of interest in CT image data, obtaining a three-dimensional virtual model, importing the three-dimensional virtual model into a three-dimensional target platform, and obtaining a three-dimensional model holographic image; carrying out registration on the real-time ultrasonic image; carrying out coordinate conversion on the target point, and displaying the converted mark point on the holographic image of the three-dimensional model; a puncture simulation experiment is carried out, and a puncture association relationship is established based on the deep neural network; and taking the target spot as an actual puncture point, calculating a virtual target spot based on the puncture association relationship, and performing puncture by taking the virtual target spot as a target. According to the method, the three-dimensional model holographic image is obtained through conversion of the CT image data, the image of the to-be-punctured part can be directly observed through the wearable device, medical staff can conveniently and directly puncture an entity according to the three-dimensional model holographic image, and the puncture success rate is increased.

Description

Technical Field

[0001] The present invention relates to the field of digital medical technologies, and particularly to a method and system for ultrasound imaging-assisted puncture based on mixed reality. Background Art

[0002] In puncture medical surgeries, such as breast intervention surgeries and prostate puncture surgeries, accurately positioning the puncture point is crucial. Currently, it generally relies on medical personnel to directly judge with the naked eye for puncture. Due to individual differences, medical personnel need to perform high-precision operations during the puncture surgery, which requires high operating requirements for medical personnel and has high surgical risks. To improve surgical safety, a computer-assisted surgery system can also be used, such as relying on images such as CT or MRI to provide medical personnel with general information about the patient's lesion before the surgery. However, such images still cannot provide sufficient and accurate puncture navigation information.

[0003] To improve the success rate of puncture surgeries, existing technologies have methods of registering ultrasound images and CT images and using an assisted puncture system to achieve automatic puncture. First, the target puncture point is selected, the coordinate transformation between the target puncture point and the ultrasound image is performed using navigation software, the positional relationship between the puncture needle and the local lesion position in the image is calculated, and the assisted puncture system performs puncture according to a specific puncture path with the target puncture point as the target.

[0004] In the prior art, there may be multiple possible puncture paths for different target puncture points, and different puncture paths correspond to different needle insertion directions and needle insertion positions. Due to factors such as soft tissue compression and body structure during the puncture process, different needle insertion directions and needle insertion positions may have different effects on the final puncture endpoint, resulting in the puncture position deviating from the target puncture point and low puncture accuracy. Summary of the Invention

[0005] Therefore, the present invention provides a method and system for ultrasound imaging-assisted puncture based on mixed reality, effectively solving the technical problem in the prior art that due to factors such as soft tissue compression and body structure during the puncture process, different needle insertion directions and needle insertion positions may have different effects on the final puncture endpoint, resulting in the puncture position deviating from the target puncture point and low puncture accuracy.

[0006] To solve the above technical problem, the present invention specifically provides the following technical solution: A method for ultrasound imaging-assisted puncture based on mixed reality, comprising the following steps:

[0007] Obtain CT images and CT image data, use 3D modeling software to segment and reconstruct the region of interest in the CT image data to obtain a 3D virtual model, and import the 3D virtual model into a 3D target platform to obtain a 3D model holographic image;

[0008] Obtain real-time ultrasound images, register the real-time ultrasound images based on CT images, and construct the coordinate transformation relationship between the ultrasound space and the puncture needle space;

[0009] Select the target range, take the center point of the target range as the target point, perform coordinate transformation of the target point according to the coordinate transformation relationship, import the transformed marked point into the three-dimensional target platform and display it in real time on the three-dimensional model holographic image;

[0010] Based on different puncture parameters and the target point, repeat the simulated puncture experiment with the three-dimensional model holographic image, construct a deep neural network, use the experimental data of the simulated puncture experiment as input, train the deep neural network, and establish the puncture correlation relationship between the puncture parameters, the target point and the actual puncture point based on the deep neural network;

[0011] Take the target point as the actual puncture point, calculate the corresponding puncture parameters based on the puncture correlation relationship, inversely calculate the ideal target point coordinates from the puncture parameters and the actual puncture point to obtain the virtual target point, and perform puncture with the virtual target point as the target

[0012] Furthermore, after importing the three-dimensional virtual model into the three-dimensional target platform, perform color rendering and display the three-dimensional model holographic image in the display window;

[0013] Establish a network connection between the wearable device and the three-dimensional target platform, calibrate the three-dimensional model holographic image to the corresponding position, and display the three-dimensional model holographic image in the wearable device.

[0014] Furthermore, constructing the coordinate transformation relationship between the ultrasound space and the puncture needle space includes the following steps:

[0015] Create a puncture needle space A, an ultrasound image space B, an electromagnetic space C, an ultrasound probe space D, and a CT space E, and unify the puncture needle space A, the ultrasound image space B, the electromagnetic space C, the ultrasound probe space D, and the CT space E into a spatial coordinate system;

[0016] Establish the transformation relationship between the electromagnetic space C and the puncture needle space A:

[0017] A = M1 * C;

[0018] Wherein, M1 is the transformation matrix for transforming the electromagnetic space C to the puncture needle space A;

[0019] Establish the transformation relationship between the ultrasound probe space D and the electromagnetic space C:

[0020] C = M2 * D;

[0021] Wherein, M2 is the transformation matrix for transforming the ultrasound probe space D to the electromagnetic space C;

[0022] Establish the transformation relationship between the ultrasonic image space B and the ultrasonic probe space D:

[0023] D = M3 * B;

[0024] Wherein, M3 is the transformation matrix for transforming the ultrasonic image space B to the ultrasonic probe space D;

[0025] Establish the transformation relationship between the ultrasonic image space B and the CT space E:

[0026] B = M4 * E;

[0027] Wherein, M3 is the transformation matrix for transforming the CT space E to the ultrasonic image space B;

[0028] Combining the above transformation relationships, obtain the coordinate transformation relationship between the puncture needle space A and the CT space E:

[0029] A = M1 * M2 * M3 * M4 * E.

[0030] Furthermore, select a target range within the puncture needle space A, take the center point of the target range as the target target point, calculate the coordinate position of the target target point in the CT space according to the coordinate transformation relationship, use the transformed target target point as a marking point, import the updated three-dimensional virtual model into the three-dimensional target platform, and after color rendering the marking point, display it in real time on the three-dimensional model holographic image.

[0031] Furthermore, train the deep neural network, and establish a puncture correlation relationship between the puncture parameters, the target target point, and the actual puncture point based on the deep neural network, including the following steps:

[0032] Establish a coordinate linear relationship between the target target point and the actual puncture point:

[0033]

[0034] Wherein, (x1, y1, z1) are the coordinates of the actual puncture point in the puncture needle space, (x2, y2, z2) are the coordinates of the target target point in the puncture needle space, and a, b, c, d, e, f are all correlation parameters;

[0035] Use the experimental data of the simulated puncture experiment as the input, and obtain the first correlation relationship between the puncture parameters and the correlation parameters, and the second correlation relationship between the puncture parameters and the actual puncture point through the training of the deep neural network.

[0036] Furthermore, the puncture parameters include the puncture point (x j , y j , z j ), the puncture angle θ, and the puncture path distance s;

[0037] wherein, the distance s of the needle insertion path is (x1, y1, z1) are the coordinates of the actual puncture point in the puncture needle space.

[0038] Further, set the target range during the actual puncture process, calculate the center point of the target range as the target target point, use the target target point as the actual puncture point, calculate the feasible puncture parameters based on the second correlation relationship between the puncture parameters and the actual puncture point, calculate the correlation parameters based on the first correlation relationship between the puncture parameters and the correlation parameters, substitute the correlation parameters and update the coordinate linear relationship between the puncture target target point and the actual puncture point;

[0039] Substitute the actual puncture point into the updated coordinate linear relationship between the puncture target target point and the actual puncture point, and reversely calculate to obtain the ideal target point coordinates (x2, y2, z2), and use the ideal target point coordinates as the virtual target;

[0040] After coordinate transformation of the virtual target, it is updated in real time on the three-dimensional model holographic image.

[0041] Further, register the real-time ultrasound image based on the CT image, including the following steps:

[0042] Use point registration to establish a one-to-one correspondence between the CT image and the body structure;

[0043] Based on the data of the real-time ultrasound image, obtain the position coordinates and orientation data of the body structure during the operation;

[0044] Import the real-time ultrasound image and CT image data into the image processing module for image preprocessing respectively, and export the preprocessed standard data;

[0045] Align and merge the standard data based on the Dldp algorithm.

[0046] To solve the above technical problems, the present invention further provides the following technical solution: A mixed reality-based ultrasound imaging-assisted puncture system, comprising:

[0047] An ultrasound imaging device, which uses an ultrasound probe to obtain a real-time ultrasound image;

[0048] A CT scanner, which performs a pre-puncture scan to obtain a CT image and CT image data;

[0049] An image processing module, which receives the CT image and CT image data, segments and reconstructs the region of interest in the CT image data to obtain a three-dimensional virtual model, and converts the three-dimensional virtual model into a three-dimensional model holographic image;

[0050] A wearable device is network - connected to an image - processing module. The wearable device receives and displays a three - dimensional model holographic image;

[0051] A puncture - assisting device, with a puncture needle installed at its execution end. The puncture needle performs a puncture action after adjusting its position and puncture angle under the drive of an execution module;

[0052] A data - processing module, connected to the execution module, and a deep neural network is constructed within the data - processing module;

[0053] Among them, the execution module drives the puncture needle to conduct a simulated puncture experiment based on different puncture parameters and target points. The execution module shares the experimental data measured in the simulated puncture experiment with the data - processing module in real - time to train the deep neural network.

[0054] Furthermore, in the data - processing module, a puncture correlation relationship between puncture parameters, target points, and actual puncture points is established based on the deep neural network;

[0055] Before the puncture, a target point is set, the target point is used as the actual puncture point, and the corresponding puncture parameters are calculated based on the puncture correlation relationship. The ideal target point coordinates are calculated in reverse from the puncture parameters and the actual puncture point to obtain a virtual target point;

[0056] The execution module performs puncture with the virtual target point as the target according to the puncture parameters.

[0057] The present invention has the following beneficial effects compared with the prior art:

[0058] In the present invention, a three - dimensional model holographic image is obtained by converting CT image data. Through the wearable device, the image of the part to be punctured can be directly observed, which is convenient for medical staff to directly perform puncture on the entity according to the three - dimensional model holographic image, improving the puncture success rate;

[0059] Furthermore, considering the influence of different puncture parameters on the final puncture point, a puncture correlation relationship between puncture parameters, target points, and actual puncture points is established through the deep neural network. When the target point is known, the target point is used as the actual puncture point, and the ideal target point coordinates are calculated in reverse according to the puncture parameters. Using this coordinate as the virtual target point and performing puncture on the virtual target point greatly improves the puncture success rate and puncture accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are merely exemplary, and for those of ordinary skill in the art, without creative efforts, other implementation drawings can be obtained by extending the provided drawings.

[0061] Figure 1 Flowchart of a method for ultrasound imaging-assisted puncture based on mixed reality provided by an embodiment of the present invention;

[0062] Figure 2 Structural block diagram of a system for ultrasound imaging-assisted puncture based on mixed reality provided by an embodiment of the present invention. Specific implementation manners

[0063] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0064] As Figure 1 shown, the present invention provides a method for ultrasound imaging-assisted puncture based on mixed reality, including the following steps:

[0065] Obtain CT images and CT image data, use 3D modeling software to segment and reconstruct the region of interest in the CT image data to obtain a 3D virtual model, and import the 3D virtual model into a 3D target platform to obtain a 3D model holographic image;

[0066] Obtain real-time ultrasound images, register the real-time ultrasound images based on the CT images, and construct the coordinate transformation relationship between the ultrasound space and the puncture needle space;

[0067] Select a target range, use the center point of the target range as the target target, perform coordinate transformation of the target target according to the coordinate transformation relationship, and import the transformed marked point into the 3D target platform and display it in real time on the 3D model holographic image;

[0068] Based on different puncture parameters and the target target, repeat the simulated puncture experiment in combination with the 3D model holographic image, construct a deep neural network, use the experimental data of the simulated puncture experiment as input, train the deep neural network, and establish a puncture correlation relationship between the puncture parameters, the target target, and the actual puncture point based on the deep neural network;

[0069] Take the target target point as the actual puncture point, calculate the corresponding puncture parameters based on the puncture correlation relationship, and inversely calculate the ideal target point coordinates from the puncture parameters and the actual puncture point to obtain a virtual target point, and perform puncture with the virtual target point as the target.

[0070] In the present invention, a three-dimensional model holographic image is obtained by converting CT image data. Through a wearable device, the image of the part to be punctured can be directly observed, which is convenient for medical staff to directly perform puncture on the entity according to the three-dimensional model holographic image, improving the puncture success rate.

[0071] Furthermore, considering the influence of different puncture parameters on the final puncture point, a puncture correlation relationship between the puncture parameters, the target target point, and the actual puncture point is established through a deep neural network. When the target target point is known, the target target point is used as the actual puncture point, and the ideal target point coordinates are inversely calculated based on the puncture parameters. Using this coordinate as the virtual target point, puncture is performed on the virtual target point, greatly improving the puncture success rate and puncture accuracy.

[0072] In the present invention, a three-dimensional virtual model is constructed using CT images and CT image data. Specifically, after importing the three-dimensional virtual model into a three-dimensional target platform, color rendering is performed, and the three-dimensional model holographic image is displayed in a display window.

[0073] Establish a network connection between the wearable device and the three-dimensional target platform, calibrate the three-dimensional model holographic image to the corresponding position, and display the three-dimensional model holographic image in the wearable device.

[0074] In actual operation, first, the body model to be punctured is scanned by CT to obtain CT image data in DICOM format. The 3D Slicer software is used to segment and reconstruct the region of interest in the CT image data to obtain an obj format file. The obtained obj file is imported into the Unity3D platform, and the general window platform is selected as the three-dimensional target platform. The model is rendered with a specific color, and the three-dimensional virtual model is displayed in the display window. The three-dimensional virtual model is imported into the wearable device, and the three-dimensional model holographic image is calibrated to a specific position, and the three-dimensional model holographic image is displayed in the wearable device.

[0075] CT images can only provide preoperative image data. To understand the relevant data during the operation, in addition to CT images and CT image data, the present invention also needs to obtain ultrasonic images, obtain real-time ultrasonic images, and register the real-time ultrasonic images based on the CT images.

[0076] Specifically, registering the real-time ultrasonic image based on the CT image includes the following steps:

[0077] Use point registration to establish a one-to-one correspondence between the CT image and the body structure.

[0078] Obtain the position coordinates and orientation data of the intraoperative body structure based on the data of real-time ultrasound images;

[0079] Import the real-time ultrasound images and CT image data into the image processing module for image preprocessing respectively, and export the preprocessed standard data;

[0080] Align and merge the standard data based on the Dldp algorithm.

[0081] In addition to point registration, surface registration can also be used to establish a one-to-one correspondence between the CT image and the body structure. Use the image processing module to obtain the position coordinates of the registration marker points or surfaces in the ultrasound space or CT space, and also calculate the position coordinates of the registration marker points or surfaces in the preoperative image. Use the relevant registration algorithm to overlap the two spatial coordinate systems into the same coordinate system;

[0082] Obtain real-time ultrasound images during the operation, import the image data into the image processing module, and through the image processing module for data processing, obtain the spatial position coordinates and orientation data of the patient's intraoperative body structure;

[0083] Use the position information of the two-dimensional image for three-dimensional reconstruction. Import the data of the ultrasound images and CT images into the image processing software MIPAV for image preprocessing respectively, and finally export the preprocessed standard data;

[0084] Finally, based on the Dldp algorithm for the fusion of ultrasound images and CT images, align and merge the standard data. The Dldp algorithm is an existing image fusion algorithm;

[0085] In the actual operation process, fuse the intraoperative real-time ultrasound image data into the preoperative CT image, and perform repeated registration. Each time a real-time ultrasound image is obtained, a data fusion is performed.

[0086] After the above data fusion, the real-time update of the three-dimensional model holographic image can be performed based on the updated image data. Thus, a dynamic three-dimensional model holographic image can be observed during the operation.

[0087] The above steps achieve the real-time display of the body structure on the wearable device side. To further display the images related to puncture, it is necessary to construct the coordinate transformation relationship between the ultrasound space and the puncture needle space in advance. Specifically, constructing the coordinate transformation relationship between the ultrasound space and the puncture needle space includes the following steps:

[0088] Create a puncture needle space A, an ultrasound image space B, an electromagnetic space C, an ultrasound probe space D, and a CT space E, and unify the puncture needle space A, the ultrasound image space B, the electromagnetic space C, the ultrasound probe space D, and the CT space E into a spatial coordinate system;

[0089] Establish the transformation relationship between the electromagnetic space C and the puncture needle space A:

[0090] A = M1 * C;

[0091] Wherein, M1 is the transformation matrix for transforming the electromagnetic space C to the puncture needle space A;

[0092] Establish the transformation relationship between the ultrasound probe space D and the electromagnetic space C:

[0093] C = M2 * D;

[0094] Wherein, M2 is the transformation matrix for transforming the ultrasound probe space D to the electromagnetic space C;

[0095] Establish the transformation relationship between the ultrasound image space B and the ultrasound probe space D:

[0096] D = M3 * B;

[0097] Wherein, M3 is the transformation matrix for transforming the ultrasound image space B to the ultrasound probe space D;

[0098] Establish the transformation relationship between the ultrasound image space B and the CT space E:

[0099] B = M4 * E;

[0100] Wherein, M3 is the transformation matrix for transforming the CT space E to the ultrasound image space B;

[0101] Combining the above transformation relationships, obtain the coordinate transformation relationship between the puncture needle space A and the CT space E:

[0102] A = M1 * M2 * M3 * M4 * E.

[0103] Select a target range within the puncture needle space A, take the center point of the target range as the target target point, calculate the coordinate position of the target target point in the CT space according to the coordinate transformation relationship, use the transformed target target point as a marker point, import the marker point into the three-dimensional virtual model, import the updated three-dimensional virtual model into the three-dimensional target platform, and after color rendering the marker point, it is displayed in real time on the three-dimensional model holographic image.

[0104] In the above embodiments, according to the coordinate transformation relationship between the puncture needle space A and the CT space E, the coordinate conversion of the target target point can be realized, and after converting the target target point into a marker point, it is displayed in real time on the three-dimensional model holographic image. Medical personnel can perform a simulated puncture experiment by means of the position of the marker point in the wearable device. In the case of using manual simulation for the puncture experiment, the puncture needle has a higher degree of freedom, more puncture parameters can be adjusted, and more comprehensive experimental data can be obtained;

[0105] Under the same target, the puncture parameters are continuously adjusted to conduct simulated puncture experiments to obtain different actual puncture points. Under different targets, the puncture parameters are repeatedly adjusted to conduct simulated puncture experiments to obtain different actual puncture points, so as to obtain multiple sets of experimental data.

[0106] In the present invention, the puncture parameters include but are not limited to the needle entry point (x j , y j , z j ), the needle entry angle θ, and the needle entry path distance s;

[0107] Among them, (x1, y1, z1) are the coordinates of the actual puncture point in the puncture needle space.

[0108] In practical applications, other puncture parameters related to the actual puncture point can be added according to the actual situation.

[0109] After multiple sets of simulated puncture experiments are completed, the experimental data of the simulated puncture experiments are used as input to train the deep neural network. Specifically, when training the deep neural network, a puncture correlation relationship between the puncture parameters, the target, and the actual puncture point is established based on the deep neural network, including the following steps:

[0110] Establish a coordinate linear relationship between the target and the actual puncture point:

[0111]

[0112] In the formula, (x1, y1, z1) are the coordinates of the actual puncture point in the puncture needle space, (x2, y2, z2) are the coordinates of the target in the puncture needle space, and a, b, c, d, e, f are all correlation parameters;

[0113] Using the experimental data of the simulated puncture experiments as input, a first correlation relationship between the puncture parameters and the correlation parameters, and a second correlation relationship between the puncture parameters and the actual puncture point are obtained through training of the deep neural network.

[0114] The training using the deep neural network specifically includes the following steps:

[0115] The sample set composed of the experimental data of the simulated puncture experiments, the target, and the actual puncture point is divided into a training set and a test set;

[0116] The training set is propagated layer by layer from the input layer through the hidden layer to the output layer;

[0117] Calculate the error value between the expected output and the actual output of the deep neural network, and propagate the error value layer by layer from the output layer through the hidden layer to the input layer to correct the connection weights;

[0118] Forward propagation and error backpropagation are alternately repeated for memory training.

[0119] Through the above training steps of the deep neural network, the first correlation relationship between the puncture parameters and the correlation parameters, and the second correlation relationship between the puncture parameters and the actual puncture points can be obtained.

[0120] During the actual puncture operation, first set the target range during the actual puncture process, calculate the center point of the target range as the target target point, use the target target point as the actual puncture point, substitute the actual puncture point into the second correlation relationship between the known puncture parameters and the actual puncture point to calculate the feasible puncture parameters, substitute the calculated puncture parameters into the first correlation relationship between the known puncture parameters and the correlation parameters to calculate the correlation parameters, and substitute the correlation parameters into the coordinate linear relationship between the puncture target target point and the actual puncture point to obtain a new coordinate linear relationship between the puncture target target point and the actual puncture point;

[0121] Substitute the actual puncture point coordinates into the updated coordinate linear relationship between the puncture target target point and the actual puncture point, and reversely calculate to obtain the ideal target point coordinates (x2, y2, z2). Use the ideal target point coordinates as the virtual target point, and update the virtual target point on the three-dimensional model holographic image in real time after coordinate transformation.

[0122] In addition, the calculated puncture parameters can also be updated on the three-dimensional model holographic image after coordinate transformation, presenting a new marking structure, such as the needle insertion point, needle insertion angle, etc. Medical staff can perform the puncture operation according to the position of the needle insertion point and the position of the virtual target point with the help of the marking structure in the wearable device. In addition, the target target point set before the puncture operation is hidden to avoid confusion during the puncture operation.

[0123] As Figure 2 shown, the present invention also provides a mixed reality-based ultrasonic imaging-assisted puncture system, which includes:

[0124] An ultrasonic imaging device that uses an ultrasonic probe to obtain real-time ultrasonic images;

[0125] A CT scanner that performs pre-puncture scanning to obtain CT images and CT image data;

[0126] An image processing module that receives the CT images and CT image data, segments and reconstructs the region of interest in the CT image data to obtain a three-dimensional virtual model, and converts the three-dimensional virtual model into a three-dimensional model holographic image;

[0127] A wearable device that is network-connected to the image processing module. The wearable device receives and displays the three-dimensional model holographic image;

[0128] A puncture assistance device, with a puncture needle installed at its execution end. The puncture needle performs a puncture action after being adjusted to the right position and puncture angle by the execution module;

[0129] A data processing module, connected to the execution module, with a deep neural network built therein;

[0130] Among them, the execution module drives the puncture needle to conduct a simulated puncture experiment based on different puncture parameters and target points. The execution module shares the experimental data measured in the simulated puncture experiment with the data processing module in real time to train the deep neural network.

[0131] Under this system, using a machine to conduct a simulated puncture experiment, compared with the simulated puncture experiment conducted manually, the data obtained from this simulated puncture experiment has a better training effect and is more accurate in constructing the puncture correlation relationship. In addition, during a puncture operation, a medical staff wears a wearable device. After the execution module adjusts the puncture parameters, the medical staff can also fine-tune the puncture parameters of the execution module according to their own experience to make the puncture more accurate.

[0132] In the data processing module, a puncture correlation relationship between the puncture parameters, the target point, and the actual puncture point is established based on the deep neural network;

[0133] Before the puncture is performed, a target point is set, the target point is used as the actual puncture point, and the corresponding puncture parameters are calculated based on the puncture correlation relationship. The ideal target point coordinates are calculated in reverse from the puncture parameters and the actual puncture point to obtain a virtual target point. The execution module performs the puncture with the virtual target point as the target according to the puncture parameters.

[0134] Using the execution module to drive the puncture needle to perform the puncture action can more accurately perform the puncture with the virtual target point as the target according to the puncture parameters, making the puncture process more accurate.

[0135] The above embodiments are only exemplary embodiments of the present application and are not used to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art can make various modifications or equivalent replacements within the essence and protection scope of the present application, and such modifications or equivalent replacements should also be regarded as falling within the protection scope of the present application.

Claims

1. An ultrasound imaging-assisted puncture method based on mixed reality, characterized in that: The following steps are involved: Acquire CT images and CT image data, segment and reconstruct the region of interest in the CT image data using 3D modeling software to obtain a 3D virtual model, import the 3D virtual model into a 3D target platform, and obtain a 3D model holographic image; Acquire real-time ultrasound images, register them based on CT images, and construct the coordinate transformation relationship between ultrasound space and puncture needle space; Select the target range, take the center point of the target range as the target point, transform the coordinates of the target point according to the coordinate transformation relationship, import the transformed mark point into the three-dimensional target platform and display it in real time on the three-dimensional model holographic image; Based on different puncture parameters, target points and three-dimensional model holographic images, simulated puncture experiments are repeatedly carried out to construct a deep neural network. The experimental data of the simulated puncture experiment is used as input to train the deep neural network. The puncture correlation relationship between the puncture parameters and the target points and the actual puncture points is established based on the deep neural network. The target point is taken as the actual puncture point, and the corresponding puncture parameters are calculated based on the puncture association relationship. The ideal target coordinates are reversely calculated from the puncture parameters and the actual puncture point to obtain the virtual target point, and puncture is performed with the virtual target point as the target.

2. The ultrasound imaging-assisted puncture method based on mixed reality according to claim 1, characterized in that: After the 3D virtual model is imported into the 3D target platform, color rendering is performed and a holographic image of the 3D model is displayed in a display window; Establish a network connection between the wearable device and the three-dimensional target platform, calibrate the three-dimensional model holographic image to the corresponding position, and display the three-dimensional model holographic image in the wearable device.

3. The ultrasound imaging-assisted puncture method based on mixed reality according to claim 1, characterized in that: Constructing the coordinate transformation relationship between the ultrasound space and the puncture needle space includes the following steps: Create a puncture needle space A, an ultrasound image space B, an electromagnetic space C, an ultrasound probe space D, and a CT space E, and unify the puncture needle space A, the ultrasound image space B, the electromagnetic space C, the ultrasound probe space D, and the CT space E into a spatial coordinate system; Establish the transformation relationship between electromagnetic space C and puncture needle space A: A=M1*C; Where M1 is the transformation matrix from electromagnetic space C to puncture needle space A; Establish the transformation relationship between the ultrasonic probe space D and the electromagnetic space C: C = M2*D; Where M2 is the transformation matrix from the ultrasonic probe space D to the electromagnetic space C; Establish the transformation relationship between the ultrasound image space B and the ultrasound probe space D: D = M3 * B; Wherein, M3 is the transformation matrix from ultrasound image space B to ultrasound probe space D; Establish the transformation relationship between ultrasound image space B and CT space E: B = M4*E; Where M3 is the transformation matrix from CT space E to ultrasound image space B; Combining the above transformation relationships, the coordinate transformation relationship between the puncture needle space A and the CT space E is obtained: A=M1*M2*M3*M4*E.

4. The ultrasound imaging-assisted puncture method based on mixed reality according to claim 3, characterized in that: A target range is selected in the puncture needle space A, and the center point of the target range is used as the target point. The coordinate position of the target point in the CT space is calculated based on the coordinate transformation relationship. The converted target point is used as a marking point, and the marking point is imported into the three-dimensional virtual model. The updated three-dimensional virtual model is imported into the three-dimensional target platform, and the marking point is rendered in color and displayed in real time on the three-dimensional model holographic image.

5. The ultrasound imaging-assisted puncture method based on mixed reality according to claim 1, characterized in that: The deep neural network is trained, and the puncture correlation relationship between the puncture parameters and the target point and the actual puncture point is established based on the deep neural network, including the following steps: Establish the coordinate linear relationship between the target point and the actual puncture point: Where, (x1, y1, z1) is the coordinate of the actual puncture point in the puncture needle space, (x2, y2, z2) is the coordinate of the target point in the puncture needle space, and a, b, c, d, e, and f are all associated parameters; Taking the experimental data of the simulated puncture experiment as input, the first correlation relationship between the puncture parameters and the associated parameters and the second correlation relationship between the puncture parameters and the actual puncture points are obtained through deep neural network training.

6. The ultrasound imaging-assisted puncture method based on mixed reality according to claim 5, characterized in that: The puncture parameters include the needle insertion point (x j ,y j , z j ), needle insertion angle θ, needle insertion path distance s; Among them, the needle insertion path distance (x1, y1, z1) are the coordinates of the actual puncture point in the puncture needle space.

7. The ultrasound imaging-assisted puncture method based on mixed reality according to claim 5, characterized in that: The target range in the actual puncture process is set, the center point of the target range is calculated as the target point, the target point is used as the actual puncture point, and the feasible puncture parameters are calculated based on the actual puncture point by using the second correlation relationship between the puncture parameters and the actual puncture point, and the correlation parameters are calculated based on the puncture parameters by using the first correlation relationship between the puncture parameters and the correlation parameters, and the correlation parameters are substituted into and updated into the coordinate linear relationship between the puncture target point and the actual puncture point; Substitute the actual puncture point into the updated linear relationship between the puncture target point and the actual puncture point, calculate the ideal target point coordinates (x2, y2, z2) in reverse, and use the ideal target point coordinates as the virtual target point; The coordinates of the virtual target are changed and updated in real time on the 3D model holographic image.

8. The ultrasound imaging-assisted puncture method based on mixed reality according to claim 1, characterized in that: The real-time ultrasound image is registered based on the CT image, including the following steps: Using point registration, a one-to-one correspondence between CT images and body structures is established; Acquire the position coordinates and orientation data of the body structure during surgery based on real-time ultrasound image data; Importing real-time ultrasound images and CT image data into the image processing module for image preprocessing, and exporting the preprocessed standard data; The standard data are aligned and merged based on the Dldp algorithm.

9. A system using the ultrasound imaging-assisted puncture method based on mixed reality according to any one of claims 1 to 8, characterized in that: have: Ultrasonic imaging equipment, which uses an ultrasonic probe to obtain real-time ultrasonic images; CT scanner, for pre-puncture scanning to obtain CT images and CT image data; An image processing module receives CT images and CT image data, segments and reconstructs the region of interest in the CT image data, obtains a three-dimensional virtual model, and converts the three-dimensional virtual model into a three-dimensional model holographic image; A wearable device is connected to the image processing module network, and the wearable device receives and displays the 3D model holographic image; The puncture auxiliary device has a puncture needle installed at its execution end. The puncture needle performs the puncture action after adjusting the position and the needle insertion angle under the drive of the execution module; A data processing module, connected to the execution module, wherein a deep neural network is constructed in the data processing module; Among them, the execution module drives the puncture needle to perform a simulated puncture experiment based on different puncture parameters and target points, and the execution module shares the experimental data measured in the simulated puncture experiment to the data processing module in real time to train the deep neural network.

10. The ultrasound imaging-assisted puncture system based on mixed reality according to claim 9, characterized in that: In the data processing module, the puncture correlation relationship between the puncture parameters and the target point and the actual puncture point is established based on the deep neural network; Before puncture, the target point is set and used as the actual puncture point. The corresponding puncture parameters are calculated based on the puncture association relationship. The ideal target point coordinates are reversely calculated from the puncture parameters and the actual puncture point to obtain the virtual target point. The execution module takes the virtual target point as the target and performs puncture according to the puncture parameters.

Images