Puncture imaging system and method based on POCUS and MR imaging technologies

Abstract

The invention relates to the technical field of digital medical treatment, in particular to a puncture imaging system and method based on POCUS and MR imaging technologies, and the system comprises POCUS equipment which is used for carrying out instant ultrasonic examination on a puncture object to obtain an instant ultrasonic image of the puncture object; the image processing equipment is used for obtaining an ultrasonic image for guiding puncture catheterization; and the MR head-mounted display equipment is used for superposing the ultrasonic image for guiding the puncture catheterization onto a puncture object through a mixed display technology to obtain a puncture operation guiding view. The wireless mixed reality MR ultrasonic imaging technology and the POCUS technology are combined, ultrasonic images are transmitted to mobile equipment of a doctor in real time through the wireless transmission technology, the ultrasonic images and the body of a patient are fused, a visual three-dimensional view is provided, the visual immediacy sense is guaranteed, meanwhile, an automatic tracking and positioning algorithm is further arranged, and the system is convenient to use. Therefore, doctors are assisted to accurately identify blood vessel position information and provide operation guidance information.

Description

Technical Field

[0001] The present invention relates to the field of digital medical technology, and particularly to a puncture imaging system and method based on POCUS and MR imaging technologies. Background Art

[0002] Pediatric arterial puncture and catheterization is a common medical procedure used to treat various diseases such as heart failure, shock, etc. Due to the small diameter and deep location of pediatric arteries, and the fact that children often cannot cooperate, this procedure has certain difficulties and risks.

[0003] Currently, pediatric arterial puncture and catheterization usually rely on preoperative ultrasound images to provide vascular position information for the operating doctor. However, the preoperative images cannot fully match the intraoperative arterial position, so it cannot provide the operating doctor with intraoperative visual presence, nor can it provide puncture guidance. The puncture and catheterization operation can only rely on the skills and experience of the operating doctor, resulting in a low success rate.

[0004] Therefore, the existing technology has the defects of lacking visual presence and puncture guidance in pediatric arterial puncture and catheterization, making it difficult to ensure the puncture success rate. Summary of the Invention

[0005] The purpose of the present invention is to provide a puncture imaging system and method based on POCUS and MR imaging technologies to solve the technical problem that in the existing technology, pediatric arterial puncture and catheterization lack visual presence and puncture guidance, making it difficult to ensure the puncture success rate.

[0006] To solve the above technical problem, the present invention specifically provides the following technical solutions:

[0007] A puncture imaging system based on POCUS and MR imaging technologies, including the following steps:

[0008] A POCUS device for performing real-time ultrasound examination on a puncture object, obtaining a real-time ultrasound image of the puncture object, and transmitting the real-time ultrasound image to an image processing device;

[0009] An image processing device for performing arterial tracking and positioning, puncture and catheterization path planning, and catheterization depth measurement on the real-time ultrasound image to obtain an ultrasound image for puncture and catheterization guidance, and transmitting the ultrasound image for puncture and catheterization guidance to an MR head-mounted display device;

[0010] An MR head-mounted display device for superimposing the ultrasound image for puncture and catheterization guidance on the puncture object through a mixed display technology to obtain a puncture operation guidance view for providing the real-time position of the artery, the puncture and catheterization path, and the catheterization depth on the puncture object entity.

[0011] As a preferred embodiment of the present invention, the POCUS device includes a POCUS device body and a wireless transmission module, and the wireless transmission module is disposed on the POCUS device body.

[0012] As a preferred embodiment of the present invention, the image processing device includes a processor module, a memory module, and a wireless transmission module. Among them, the processor module is used to perform differential analysis between real-time ultrasound images at consecutive times, and based on the differential analysis results, use the Unet network to perform tracking segmentation of arterial tissue between real-time ultrasound images at consecutive times to obtain an ultrasound image containing real-time arterial localization, and use the A* algorithm to plan a puncture and catheterization path on the ultrasound image containing real-time arterial localization, and measure the length of the puncture and catheterization path as the catheterization depth;

[0013] The memory is used to store real-time ultrasound images at each time, and ultrasound images for puncture and catheterization reference at each time.

[0014] As a preferred embodiment of the present invention, the MR head-mounted display device further includes an MR head-mounted device body, a gesture or voice command module, a puncture and catheterization guidance operation interface, and a wireless transmission module. The gesture or voice command module, the wireless transmission module, and the puncture and catheterization guidance operation interface are all disposed on the MR head-mounted device body. Among them, the puncture and catheterization guidance operation interface is used to display guidance views of the real-time arterial localization function, the puncture and catheterization path planning function, and the catheterization depth measurement function. The gesture or voice command module is used to receive operation instructions from the operation object for the puncture and catheterization guidance operation interface, and act on the puncture and catheterization guidance operation interface to execute the functions selected by the operation object;

[0015] The guidance view of the real-time arterial localization function is a puncture operation guidance view that provides the real-time position of the artery on the puncture object entity displayed on the MR head-mounted device body;

[0016] The guidance view of the puncture and catheterization path planning function is a puncture operation guidance view that provides the puncture and catheterization path on the puncture object entity displayed on the MR head-mounted device body;

[0017] The guidance view of the catheterization depth measurement function is a puncture operation guidance view that provides the catheterization depth on the puncture object entity displayed on the MR head-mounted device body.

[0018] As a preferred embodiment of the present invention, the guidance views of the real-time arterial localization function, the puncture and catheterization path planning function, and the catheterization depth measurement function can be superimposed and displayed on the puncture and catheterization guidance operation interface.

[0019] As a preferred embodiment of the present invention, the wireless transmission module is used for data transmission among the POCUS device, the image processing device, and the MR head-mounted display device.

[0020] As a preferred embodiment of the present invention, the present invention provides a puncture imaging method based on POCUS and MR imaging technologies, which is applied to a puncture imaging system based on POCUS and MR imaging technologies. The method includes the following steps:

[0021] Perform real-time ultrasound examination on the puncture object through the POCUS device to obtain a real-time ultrasound image of the puncture object, and transmit the real-time ultrasound image to the image processing device;

[0022] Perform artery tracking and positioning, puncture catheterization path planning, and catheterization depth measurement on the real-time ultrasound image through the image processing device to obtain an ultrasound image for puncture catheterization guidance, and transmit the ultrasound image for puncture catheterization guidance to the MR head-mounted display device;

[0023] Overlay the ultrasound image for puncture catheterization guidance on the puncture object through the MR head-mounted display device by using the mixed display technology to obtain a puncture operation guidance view for providing the immediate position of the artery, the puncture catheterization path, and the catheterization depth on the puncture object entity.

[0024] As a preferred embodiment of the present invention, the method for obtaining an ultrasound image including immediate artery positioning includes:

[0025] Before the puncture catheterization operation starts:

[0026] First step, perform real-time ultrasound examination on the puncture object through the POCUS device to obtain a preoperative real-time ultrasound image, which is marked as the frame image;

[0027] Second step, use the first Unet network to perform artery tissue segmentation on the frame image to obtain an artery segmentation image of the frame image;

[0028] After the puncture catheterization operation starts:

[0029] Third step, perform real-time ultrasound examination on the puncture object through the POCUS device at the current puncture moment to obtain a real-time ultrasound image at the current puncture moment, which is marked as the detail image;

[0030] Fourth step, perform differential analysis on the detail image and the frame image by using the Siamese network to obtain the different part between the detail image and the frame image;

[0031] Fifth step, use the second Unet network to perform artery tissue segmentation on the different part in the detail image to obtain an artery segmentation image of the different part;

[0032] Step 6: Replace the arterial segmentation image of the difference part with the arterial segmentation image corresponding to the difference part in the arterial segmentation image of the frame image, so as to obtain an immediate ultrasound image containing immediate arterial positioning at the current puncture moment;

[0033] Step 7: Replace the detail image with the frame image, and replace the immediate ultrasound image containing immediate arterial positioning at the current puncture moment with the arterial segmentation image of the frame image;

[0034] Step 8: Replace the next puncture moment with the current puncture moment, and go back to Step 3 until the puncture and catheterization operation is completed.

[0035] As a preferred solution of the present invention, the loss function of the first Unet network is:

[0036] L total1 = MSE(mask frame(out) , mask frame(real) ) + MSE(mask detail(out) , mask detail(real) );

[0037] In the formula, L total1 is the total loss of the first Unet network, mask frame(out) is the arterial segmentation image of the frame image obtained by the first Unet network, mask frame(real) is the ground truth of the arterial segmentation image of the frame image, mask detail(out) is the arterial segmentation image of the detail image obtained by the first Unet network, mask detail(real) is the ground truth of the arterial segmentation image of the detail image, and MSE is the mean square error operator;

[0038] The loss function of the second Unet network is:

[0039] L total2 = (1 - α)MSE(mask diff(out) , mask diff(real) ) + αMSE((mask diff(out) + mask same(out) ), mask detail(out) );

[0040]

[0041] In the formula, L total2 is the total loss of the second Unet network, mask diff(out) is the arterial segmentation image of the difference part in the detail image obtained by the second Unet network, mask diff(real) is the ground truth of the arterial segmentation image of the difference part in the detail image, mask diff(out)It is the arterial segmentation image obtained by removing the different parts from the frame image, and α is a hyperparameter.

[0042] As a preferred embodiment of the present invention, the display method of the puncture operation guidance view includes:

[0043] When the gesture or voice command module receives an operation command for displaying the instant arterial positioning function of the operation object, the puncture catheterization guidance operation interface will synchronously display the guidance view of the instant arterial positioning function;

[0044] When the gesture or voice command module receives an operation command for displaying the puncture catheterization path planning function of the operation object, the puncture catheterization guidance operation interface will synchronously display the guidance view of the puncture catheterization path planning function;

[0045] When the gesture or voice command module receives an operation command for displaying the catheterization depth measurement function of the operation object, the puncture catheterization guidance operation interface will synchronously display the guidance view of the catheterization depth measurement function;

[0046] When the gesture or voice command module receives an operation command for simultaneously displaying the instant arterial positioning function and the puncture catheterization path planning function of the operation object, the puncture catheterization guidance operation interface will superimpose and display the guidance view of the instant arterial positioning function and the guidance view of the puncture catheterization path planning function;

[0047] When the gesture or voice command module receives an operation command for simultaneously displaying the instant arterial positioning function and the catheterization depth measurement function of the operation object, the puncture catheterization guidance operation interface will superimpose and display the guidance view of the instant arterial positioning function and the guidance view of the catheterization depth measurement function;

[0048] When the gesture or voice command module receives an operation command for simultaneously displaying the catheterization depth measurement function and the puncture catheterization path planning function of the operation object, the puncture catheterization guidance operation interface will superimpose and display the guidance view of the catheterization depth measurement function and the guidance view of the puncture catheterization path planning function;

[0049] When the gesture or voice command module receives an operation command for simultaneously displaying the instant arterial positioning function, the puncture catheterization path planning function, and the catheterization depth measurement function of the operation object, the puncture catheterization guidance operation interface will superimpose and display the guidance view of the instant arterial positioning function, the guidance view of the puncture catheterization path planning function, and the guidance view of the catheterization depth measurement function.

[0050] The present invention has the following beneficial effects compared with the prior art:

[0051] The present invention combines wireless mixed reality MR ultrasound imaging technology with POCUS technology, transmits ultrasound images to the doctor's mobile device in real time through wireless transmission technology, and uses MR technology to fuse the ultrasound images with the patient's body to provide an intuitive three-dimensional view, ensuring visual presence. At the same time, it is also equipped with an automatic tracking and positioning algorithm to assist the doctor in accurately identifying the vascular position information, providing operation guidance information, and improving the success rate of puncture and catheterization. Description of the Drawings

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only exemplary, and for those of ordinary skill in the art, without creative efforts, other implementation drawings can also be obtained according to the provided drawings.

[0053] Figure 1 It is a block diagram of a puncture imaging system based on POCUS and MR imaging technologies provided by an embodiment of the present invention;

[0054] Figure 2 It is a flowchart of a puncture imaging method based on POCUS and MR imaging technologies provided by an embodiment of the present invention;

[0055] Figure 3 It is a puncture operation guidance view provided by an embodiment of the present invention. Detailed Embodiments

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0057] As Figure 1 shown, the present invention provides a puncture imaging system based on POCUS and MR imaging technologies, including the following steps:

[0058] A POCUS device for performing immediate ultrasound examination on the puncture object, obtaining the immediate ultrasound image of the puncture object, and transmitting the immediate ultrasound image to the image processing device;

[0059] An image processing device for performing arterial tracking and positioning, puncture and catheterization path planning, and catheterization depth measurement on the immediate ultrasound image to obtain the ultrasound image for puncture and catheterization guidance, and transmitting the ultrasound image for puncture and catheterization guidance to the MR head-mounted display device;

[0060] An MR head-mounted display device is used to superimpose the ultrasound image for puncture and catheterization guidance onto the puncture object through a mixed display technology, obtaining a puncture operation guidance view for providing the immediate position of the artery, the puncture and catheterization path, and the catheterization depth on the puncture object entity.

[0061] The present invention combines POCUS and MR imaging technologies, transmits the ultrasound image to the doctor's mobile device in real time through wireless transmission technology, and uses MR technology to fuse the ultrasound image with the patient's body, providing an intuitive three-dimensional view and ensuring a visual presence.

[0062] In the present invention, the POCUS (Point-of-Care Ultrasound) device is a portable ultrasound imaging device designed specifically for performing immediate ultrasound examinations at the patient's bedside or in the clinical field, with the advantages of convenience and immediacy. Therefore, it is used to obtain the immediate ultrasound image of the puncture object in real time during the pediatric puncture and catheterization process, so that real-time arterial information can be obtained from the immediate ultrasound image.

[0063] In the present invention, when obtaining the immediate ultrasound image during the pediatric puncture and catheterization process, a three-dimensional view that fuses the ultrasound image with the patient's body can be generated through mixed reality MR imaging technology. Thus, the real-time arterial information inside the patient can be seen through the MR head-mounted display device. These arterial information are seamlessly integrated with the patient's body parts, providing an immersive visual experience and achieving a high visual presence, which helps the operating doctor to observe the internal structure of the patient more clearly, contributing to improving the accuracy of diagnosis, especially in complex anatomical regions.

[0064] Furthermore, the present invention is also equipped with an image processing device (such as a computer device) for processing the immediate ultrasound image, which can mark the operation guidance information in the immediate ultrasound image through an automatic tracking and positioning algorithm, such as the real-time position information of the artery, the puncture and catheterization path, and the catheterization depth, etc. Thus, it is displayed on the three-dimensional view generated by MR technology, which can assist the doctor in identifying and tracking the target structure, reducing the difficulty and error of manual positioning.

[0065] Due to the dynamic characteristics of arterial pulsation, the artery will have different position information at different times during the puncture catheterization process. To ensure the accuracy of puncture catheterization, it is necessary to locate and track the artery position to obtain the immediate artery position information. Therefore, it is necessary to perform arterial tissue segmentation on the immediate ultrasound image (implemented through the Unet network). However, during the puncture catheterization process, the timeliness needs to be considered to quickly locate the immediate artery position information as much as possible. The present invention utilizes the continuity of artery positions between adjacent times, that is, between adjacent times, some arteries will pulsate, and some arteries will not pulsate. There is a certain invariance in the artery position information between the two. Therefore, only the different parts of the ultrasound images between adjacent times (the changing artery information formed by the pulsating arteries) need to be identified, the position information of the non-pulsating arteries (the unchanged artery information) is retained in the ultrasound image of the previous time, and only the different parts are segmented for arterial tissue in the ultrasound image of the later time. The segmentation result of the different parts is superimposed and combined with the segmentation result of the ultrasound image of the previous time excluding the different parts, and the artery position information of the later time can be obtained. Thus, the positioning of the entire immediate artery position information only needs to perform segmentation operations on the real-time different parts, without performing segmentation operations on the entire immediate ultrasound image, reducing the data calculation amount, and realizing the rapid positioning and tracking of artery position information in the processing of immediate ultrasound images during puncture catheterization, meeting the intraoperative requirements for timeliness.

[0066] After the present invention obtains the artery position information, it also plans the puncture catheterization path and measures the catheterization depth according to the artery information position, obtaining more information to guide puncture catheterization and enhancing the comprehensiveness of operation assistance.

[0067] The POCUS device includes a POCUS device body and a wireless transmission module, and the wireless transmission module is arranged on the POCUS device body.

[0068] The image processing device includes a processor module, a memory module, and a wireless transmission module. Among them, the processor module is used to perform differential analysis between the immediate ultrasound images at consecutive times, and based on the differential analysis result, use the Unet network to perform tracking segmentation of arterial tissue between the immediate ultrasound images at consecutive times to obtain an ultrasound image containing immediate artery localization, and use the A* algorithm to plan the puncture catheterization path on the ultrasound image containing immediate artery localization and measure the length of the puncture catheterization path as the catheterization depth;

[0069] The memory is used to store the immediate ultrasound images at each time and the ultrasound images for puncture catheterization guidance at each time.

[0070] The MR head-mounted display device also includes an MR head-mounted device body, a gesture or voice command module, a puncture and catheterization guidance operation interface, and a wireless transmission module. The gesture or voice command module, the wireless transmission module, and the puncture and catheterization guidance operation interface are all arranged on the MR head-mounted device body. Among them, the puncture and catheterization guidance operation interface is used to display the guidance views of the real-time artery positioning function, the puncture and catheterization path planning function, and the catheterization depth measurement function. The gesture or voice command module is used to receive the operation commands of the operation object on the puncture and catheterization guidance operation interface and execute the functions selected by the operation object on the puncture and catheterization guidance operation interface;

[0071] The guidance view of the real-time artery positioning function is a puncture operation guidance view that provides the real-time position of the artery on the puncture object entity displayed on the MR head-mounted device body;

[0072] The guidance view of the puncture and catheterization path planning function is a puncture operation guidance view that provides the puncture and catheterization path on the puncture object entity displayed on the MR head-mounted device body;

[0073] The guidance view of the catheterization depth measurement function is a puncture operation guidance view that provides the catheterization depth on the puncture object entity displayed on the MR head-mounted device body.

[0074] The guidance views of the real-time artery positioning function, the puncture and catheterization path planning function, and the catheterization depth measurement function can be superimposed and displayed on the puncture and catheterization guidance operation interface.

[0075] The wireless transmission module is used for data transmission between the POCUS device, the image processing device, and the MR head-mounted display device.

[0076] As Figure 2 shown, the present invention provides a puncture imaging method based on POCUS and MR imaging technologies, which is applied to a puncture imaging system based on POCUS and MR imaging technologies. The method includes the following steps:

[0077] Perform real-time ultrasound examination on the puncture object through the POCUS device to obtain the real-time ultrasound image of the puncture object, and transmit the real-time ultrasound image to the image processing device;

[0078] Perform artery tracking and positioning, puncture and catheterization path planning, and catheterization depth measurement on the real-time ultrasound image through the image processing device to obtain the ultrasound image for puncture and catheterization guidance, and transmit the ultrasound image for puncture and catheterization guidance to the MR head-mounted display device;

[0079] Through the MR head-mounted display device, superimpose the ultrasound image for puncture and catheterization guidance on the puncture object through the hybrid display technology to obtain a puncture operation guidance view that provides the real-time position of the artery, the puncture and catheterization path, and the catheterization depth on the puncture object entity (asFigure 3 as shown

[0080] The present invention utilizes the continuity of artery positions between adjacent moments, that is, between adjacent moments, there are some arteries that pulsate and some arteries that do not pulsate. There is a certain invariance in the artery position information between the two. Therefore, only the different parts of the ultrasound images between adjacent moments (the changing artery information formed by the pulsating arteries) need to be identified. The position information of the non-pulsating arteries (the unchanged artery information) is retained in the ultrasound image of the previous moment. In the ultrasound image of the subsequent moment, only the different parts are segmented for artery tissue. The segmentation result of the different parts is superimposed and combined with the segmentation result of the ultrasound image of the previous moment excluding the different parts, and the artery position information of the subsequent moment can be obtained. Thus, the positioning of the entire instantaneous artery position information only needs to perform segmentation operations on the real-time different parts, without performing segmentation operations on the entire instantaneous ultrasound image, reducing the data calculation amount, and achieving rapid positioning and tracking of artery position information in the real-time ultrasound image processing during the puncture and catheterization process, meeting the intraoperative requirements for timeliness, as follows:

[0081] The method for obtaining an ultrasound image including instantaneous artery positioning in the present invention includes:

[0082] Before the puncture and catheterization operation starts:

[0083] First step, perform an instantaneous ultrasound examination on the puncture object through a POCUS device to obtain a preoperative instantaneous ultrasound image, marked as the frame image;

[0084] Second step, utilize the first Unet network to segment artery tissue in the frame image to obtain the artery segmentation image of the frame image;

[0085] After the puncture and catheterization operation starts:

[0086] Third step, at the current puncture moment, perform an instantaneous ultrasound examination on the puncture object through a POCUS device to obtain the instantaneous ultrasound image at the current puncture moment, marked as the detail image;

[0087] Fourth step, perform differential analysis on the detail image and the frame image using a siamese network to obtain the different parts of the detail image and the frame image;

[0088] Fifth step, utilize the second Unet network to segment artery tissue in the different parts of the detail image to obtain the artery segmentation image of the different parts;

[0089] Sixth step, replace the artery segmentation image of the different parts in the artery segmentation image of the frame image corresponding to the different parts with the artery segmentation image of the different parts to obtain the instantaneous ultrasound image including instantaneous artery positioning at the current puncture moment;

[0090] Step 7: Replace the detail image with the frame image, and replace the real-time ultrasound image including the instant artery positioning at the current puncture moment with the artery segmentation image of the frame image;

[0091] Step 8: Replace the next puncture moment with the current puncture moment, and go back to Step 3 until the puncture catheterization operation is completed.

[0092] The frame image of the present invention contains the unchanged artery position information in the previous moment, and uses it as the frame. The detail image contains the changing artery position information in the current moment, and uses it as the detail. Filling the detail into the frame can obtain the complete artery position information at the current moment.

[0093] The present invention uses the preoperative ultrasound image as the first frame image, and subsequently uses the real-time ultrasound image of the previous moment of the current moment as the frame image, and the real-time ultrasound image of the current moment as the detail image.

[0094] In order to ensure the accuracy of the complete artery position information obtained by the combination of the frame and the detail at the current moment, the present invention sets the loss functions of the first Unet network and the second Unet network as follows:

[0095] The loss function of the first Unet network is:

[0096] L total1 = MSE(mask frame(out) , mask frame(real) ) + MSE(mask detail(out) , mask detail(real) );

[0097] In the formula, L total1 is the total loss of the first Unet network, mask frame(out) is the artery segmentation image of the frame image obtained by the first Unet network, mask frame(real) is the ground truth of the artery segmentation image of the frame image, mask detail(out) is the artery segmentation image of the detail image obtained by the first Unet network, mask detail(real) is the ground truth of the artery segmentation image of the detail image, and MSE is the mean square error operator;

[0098] In the present invention, the first Unet network is used to segment the artery tissue of the whole real-time ultrasound image. Therefore, it is necessary to ensure that the overall artery segmentation results of the frame image and the detail image have the smallest error from the real artery tissue, corresponding to the mean square error between mask frame(out) and mask frame(real) , and the mean square error between mask detail(out) and mask detail(real)The mean square error between them is minimized, thus ensuring the highest accuracy in the overall arterial segmentation of the first Unet network and enabling the acquisition of the most accurate arterial position information.

[0099] The loss function of the second Unet network is:

[0100] L total2 =(1 - α)MSE(mask diff(out) , mask diff(real) ) + ɑMSE((mask diff(out) + mask same(out) ), maxk detail(out) ) ;

[0101]

[0102] In the formula, L total2 is the total loss of the second Unet network, mask diff(out) is the arterial segmentation image of the different part in the detailed image obtained by the second Unet network, mask diff(real) is the ground truth of the arterial segmentation image of the different part in the detailed image, mask diff(out) is the arterial segmentation image of the frame image excluding the different part, and α is a hyperparameter.

[0103] In the present invention, the second Unet network is used for arterial tissue segmentation of the different part images in the real-time ultrasound images. Therefore, it is necessary to ensure that the error between the arterial segmentation result of the different part in the detailed image and the real arterial tissue of the different part image is minimized, corresponding to the minimum mean square error between mask diff(out) and mask diff(real) . It is also necessary to ensure that the error between the combined result of the arterial segmentation result of the different part in the detailed image and the arterial segmentation image of the frame image excluding the different part and the overall arterial segmentation result of the detailed image is minimized, corresponding to the minimum mean square error between (mask diff(out) + mask same(out) ) and mask detail(out) . In this way, it can be ensured that only the different part is subjected to arterial segmentation operation, and by adding a simple superposition combination operation with the arterial segmentation image of the frame image excluding the different part, a result consistent with the original overall arterial segmentation result on the detailed image can be obtained. During the arterial position information tracking process, compared with the overall arterial segmentation on the detailed image, only the different part needs to be segmented, ensuring a reduction in the segmentation operation while still maintaining the original segmentation accuracy.

[0104] Furthermore, the present invention also adds weights to MSE(mask diff(out) , mask diff(real) ) and MSE((mask diff(out)+mask same(out) ), mask detail(out) ) between which a hyperparameter α related to MSE (mask detail(out) , mask detail(real) ) is set to balance the two parts in L total2 , where:

[0105] When MSE (mask detail(out) , mask detail(real) ) is smaller, it indicates that the artery segmentation of the first Unet network for the detail image shows high-precision performance, and a segmentation result approximating the true value will be obtained. That is, the artery segmentation image of the detail image obtained by the first Unet network is approximately the true distribution of the artery tissue in the detail image. At this time, only by giving priority to ensuring the minimization of MSE ((mask diff(out) +mask same(out) ), mask detail(out) ), can it be ensured that the combined segmentation result of the different part and the non-different part is approximately the artery segmentation image of the detail image. That is, by giving priority to ensuring that the combined segmentation result of the different part and the non-different part is approximately the true distribution of the artery tissue in the detail image, so that the local segmentation of the different part can obtain a high-precision result similar to the overall segmentation of the detail image after superposition, ensuring the high-precision performance of the second Unet network for the artery segmentation of the different part of the detail image. The segmentation performance of the second Unet network does not require the supplement of the MSE (mask diff(out) , mask diff(real) ) part. Therefore, in this case, focus on optimizing the MSE ((mask diff(out) +mask same(out) ), mask detail(out) ). The larger α is, the higher the weight given to it, which can meet the expectation of only running the segmentation operation of the different part and the superposition operation with the non-different part, while reducing the segmentation operation of the non-different part and maintaining the high-precision segmentation performance of the second Unet network.

[0106] When MSE (mask detail(out) , mask detail(real) ) is larger, it indicates that the artery segmentation accuracy of the first Unet network for the detail image is low. At this time, even if it is ensured that the combined segmentation result of the different part and the non-different part is approximately the artery segmentation image of the detail image, it is still impossible to ensure that the combined segmentation result of the different part and the non-different part is approximately the true distribution of the artery tissue in the detail image. In this case, only focus on optimizing MSE ((mask diff(out) +mask same(out) ), mask detail(out)) part can no longer meet the high-performance segmentation expectations of the second Unet network. When the segmentation accuracy of the different part can no longer be improved, it is necessary to optimize the MSE (mask diff(out) , mask diff(real) ) part to supplement the segmentation performance of the second Unet network. It is necessary to shift the focus of optimizing the performance of the second Unet network from MSE((mask diff(out) + mask same(out) ), mask detail(out) ) to MSE((mask diff(out) + mask same(out) ), mask detail(out) ). The smaller α is, the weight of MSE((mask diff(out) + mask same(out) ), mask detail(out) ) is reduced, and the weight of MSE(mask diff(out) , mask diff(real) ) is increased, which can meet the need to only run the segmentation operation of the different part and the superposition operation with the non-different part, while reducing the segmentation operation of the non-different part and maintaining the expectation of the high-precision segmentation performance of the second Unet network.

[0107] Therefore, the present invention can achieve maintaining the high-precision segmentation performance of the second Unet network in various situations, achieving only running the segmentation operation of the different part and the superposition operation with the non-different part, while reducing the segmentation operation of the non-different part, and achieving rapid artery positioning during puncture catheterization while ensuring positioning accuracy.

[0108] The display method of the puncture operation guidance view includes:

[0109] When the gesture or voice command module receives an operation command for displaying the instant artery positioning function of the operation object, the puncture catheterization guidance operation interface will synchronously display the guidance view of the instant artery positioning function;

[0110] When the gesture or voice command module receives an operation command for displaying the puncture catheterization path planning function of the operation object, the puncture catheterization guidance operation interface will synchronously display the guidance view of the puncture catheterization path planning function;

[0111] When the gesture or voice command module receives an operation command for displaying the catheterization depth measurement function of the operation object, the puncture catheterization guidance operation interface will synchronously display the guidance view of the catheterization depth measurement function;

[0112] When the gesture or voice command module receives an operation command for simultaneously displaying the instant arterial localization function and the puncture and catheterization path planning function, the puncture and catheterization guidance operation interface will superimpose and display the guidance view of the instant arterial localization function and the guidance view of the puncture and catheterization path planning function;

[0113] When the gesture or voice command module receives an operation command for simultaneously displaying the instant arterial localization function and the catheterization depth measurement function, the puncture and catheterization guidance operation interface will superimpose and display the guidance view of the instant arterial localization function and the guidance view of the catheterization depth measurement function;

[0114] When the gesture or voice command module receives an operation command for simultaneously displaying the catheterization depth measurement function and the puncture and catheterization path planning function, the puncture and catheterization guidance operation interface will superimpose and display the guidance view of the catheterization depth measurement function and the guidance view of the puncture and catheterization path planning function;

[0115] When the gesture or voice command module receives an operation command for simultaneously displaying the instant arterial localization function, the puncture and catheterization path planning function, and the catheterization depth measurement function, the puncture and catheterization guidance operation interface will superimpose and display the guidance view of the instant arterial localization function, the guidance view of the puncture and catheterization path planning function, and the guidance view of the catheterization depth measurement function.

[0116] The present invention combines the wireless hybrid reality MR ultrasound imaging technology with the POCUS technology, transmits the ultrasound image to the doctor's mobile device in real time through the wireless transmission technology, and uses the MR technology to fuse the ultrasound image with the patient's body to provide an intuitive three-dimensional view, ensuring the visual presence. At the same time, it is also equipped with an automatic tracking and positioning algorithm to assist the doctor in accurately identifying the blood vessel position information, providing operation guidance information, and improving the success rate of puncture and catheterization.

[0117] The above embodiments are only exemplary embodiments of the present application and are not used to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art can make various modifications or equivalent replacements within the essence and protection scope of the present application, and such modifications or equivalent replacements should also be regarded as falling within the protection scope of the present application.

Claims

1. A puncture imaging system based on POCUS and MR imaging technologies, characterized in that, Including: A POCUS device for performing immediate ultrasound examination on a puncture object, obtaining an immediate ultrasound image of the puncture object, and transmitting the immediate ultrasound image to an image processing device; An image processing device for performing arterial tracking and positioning, puncture catheterization path planning, and catheterization depth measurement on the immediate ultrasound image, obtaining an ultrasound image for puncture catheterization reference, and transmitting the ultrasound image for puncture catheterization reference to an MR head-mounted display device; An MR head-mounted display device for superimposing the ultrasound image for puncture catheterization reference on the puncture object through a mixed display technology to obtain a puncture operation guidance view for providing the immediate position of the artery, the puncture catheterization path, and the catheterization depth on the puncture object entity.

2. The puncture imaging system based on POCUS and MR imaging technology according to claim 1, wherein: The POCUS device includes a POCUS device body and a wireless transmission module, and the wireless transmission module is arranged on the POCUS device body.

3. The POCUS system based on wireless MR imaging technology according to claim 2, wherein: The image processing device includes a processor module, a memory module, and a wireless transmission module. Among them, the processor module is used to perform differential analysis between the immediate ultrasound images at consecutive moments, and based on the differential analysis result, use the Unet network to perform tracking segmentation of the arterial tissue between the immediate ultrasound images at consecutive moments to obtain an ultrasound image including the immediate artery positioning, and use the A* algorithm to plan a puncture catheterization path on the ultrasound image including the immediate artery positioning, and measure the length of the puncture catheterization path as the catheterization depth; The memory is used to store the immediate ultrasound images at each moment and the ultrasound images for puncture catheterization reference at each moment.

4. A puncture imaging system based on POCUS and MR imaging techniques according to claim 3, characterized in that: The MR head-mounted display device includes an MR head-mounted device body, a gesture or voice instruction module, a puncture catheterization guidance operation interface, and a wireless transmission module. The gesture or voice instruction module, the wireless transmission module, and the puncture catheterization guidance operation interface are all arranged on the MR head-mounted device body. Among them, the puncture catheterization guidance operation interface is used to display the guidance views of the immediate artery positioning function, the puncture catheterization path planning function, and the catheterization depth measurement function. The gesture or voice instruction module is used to receive the operation instructions of the operation object on the puncture catheterization guidance operation interface and act on the puncture catheterization guidance operation interface to execute the functions selected by the operation object; The guidance view of the immediate artery positioning function is a puncture operation guidance view that provides the immediate position of the artery on the puncture object entity displayed on the MR head-mounted device body; The guidance view of the puncture catheterization path planning function is a puncture operation guidance view that provides the puncture catheterization path on the puncture object entity displayed on the MR head-mounted device body; The guidance view of the catheterization depth measurement function is a puncture operation guidance view that provides the catheterization depth on the puncture object entity displayed on the MR head-mounted device body.

5. The puncture imaging system based on POCUS and MR imaging technology according to claim 4, wherein: The guidance view of the immediate artery positioning function, the guidance view of the puncture catheterization path planning function, and the guidance view of the catheterization depth measurement function can be superimposed and displayed on the puncture catheterization guidance operation interface.

6. The puncture imaging system based on POCUS and MR imaging technology according to claim 5, characterized in that: The wireless transmission module is used for data transmission between the POCUS device, the image processing device, and the MR head-mounted display device.

7. A puncture imaging method based on POCUS and MR imaging technologies, characterized in that, Applied to a puncture imaging system based on POCUS and MR imaging techniques according to any one of claims 1-6, the method includes the following steps: Perform real-time ultrasound examination on the puncture object through a POCUS device, obtain the real-time ultrasound image of the puncture object, and transmit the real-time ultrasound image to an image processing device; Perform arterial tracking and positioning, puncture catheterization path planning, and catheterization depth measurement on the real-time ultrasound image through the image processing device to obtain the ultrasound image for puncture catheterization reference, and transmit the ultrasound image for puncture catheterization reference to the MR head-mounted display device; Overlay the ultrasound image for puncture catheterization reference on the puncture object through the MR head-mounted display device by using mixed display technology to obtain a puncture operation guidance view for providing the immediate position of the artery, puncture catheterization path, and catheterization depth on the puncture object entity.

8. A puncture imaging method based on POCUS and MR imaging techniques according to claim 7, characterized in that: The method for obtaining an ultrasound image including immediate artery positioning includes: Before the start of the puncture catheterization operation: First step, perform real-time ultrasound examination on the puncture object through a POCUS device to obtain a preoperative real-time ultrasound image, marked as a frame image; Second step, use the first Unet network to perform arterial tissue segmentation on the frame image to obtain the arterial segmentation image of the frame image; After the start of the puncture catheterization operation: Third step, perform real-time ultrasound examination on the puncture object through a POCUS device at the current puncture moment to obtain the real-time ultrasound image at the current puncture moment, marked as a detail image; Fourth step, perform differential analysis on the detail image and the frame image using a siamese network to obtain the different part between the detail image and the frame image; Fifth step, use the second Unet network to perform arterial tissue segmentation on the different part in the detail image to obtain the arterial segmentation image of the different part; Sixth step, replace the arterial segmentation image of the different part in the arterial segmentation image of the frame image with the arterial segmentation image of the different part to obtain the real-time ultrasound image including immediate artery positioning at the current puncture moment; Seventh step, replace the detail image with the frame image, and replace the real-time ultrasound image including immediate artery positioning at the current puncture moment with the arterial segmentation image of the frame image; Eighth step, replace the next puncture moment with the current puncture moment, and go to the third step until the puncture catheterization operation is completed.

9. A puncture imaging method based on POCUS and MR imaging techniques according to claim 8, characterized in that: The loss function of the first Unet network is: L total1 = MSE(mask frame(out) , mask frame(real) ) + MSE; (mask detail( out ) ,mask detail(real) ) where L total1 is the total loss of the first Unet network, mask frame(out) is the arterial segmentation image of the framework image obtained by the first Unet network, mask frame(real) is the ground truth of the arterial segmentation image of the framework image, mask detail(out) is the arterial segmentation image of the detail image obtained by the first Unet network, mask detail(real) is the ground truth of the arterial segmentation image of the detail image, and MSE is the mean square error operator; The loss function of the second Unet network is: L total2 =(1 - α)MSE(mask diff(out) , mask diff(real) ) + αMSE((mask diff(out) + mask same(out) ), mask detail(out) ); () Where L total2 is the total loss of the second Unet network, mask diff(out) is the arterial segmentation image of the different part in the detail image obtained by the second Unet network, mask diff(real) is the ground truth of the arterial segmentation image of the different part in the detail image, mask diff(out) is the arterial segmentation image obtained by removing the different part in the frame image, and α is a hyperparameter.

10. A puncture imaging method based on POCUS and MR imaging techniques according to claim 9, characterized in that: The display method of the puncture operation guidance view includes: When the gesture or voice command module receives an operation command for displaying the immediate artery positioning function of the operation object, the puncture catheterization guidance operation interface will synchronously display the guidance view of the immediate artery positioning function; When the gesture or voice command module receives an operation command for displaying the puncture catheterization path planning function of the operation object, the puncture catheterization guidance operation interface will synchronously display the guidance view of the puncture catheterization path planning function; When the gesture or voice command module receives an operation command for displaying the catheterization depth measurement function of the operation object, the puncture catheterization guidance operation interface will synchronously display the guidance view of the catheterization depth measurement function; When the gesture or voice command module receives an operation command for simultaneously displaying the instant artery positioning function and the puncture and catheterization path planning function while receiving the operation object, the puncture and catheterization guiding operation interface will superimpose and display the guiding view of the instant artery positioning function and the guiding view of the puncture and catheterization path planning function; When the gesture or voice command module receives an operation command for simultaneously displaying the instant artery positioning function and the catheterization depth measurement function while receiving the operation object, the puncture and catheterization guiding operation interface will superimpose and display the guiding view of the instant artery positioning function and the guiding view of the catheterization depth measurement function; When the gesture or voice command module receives an operation command for simultaneously displaying the catheterization depth measurement function and the puncture and catheterization path planning function while receiving the operation object, the puncture and catheterization guiding operation interface will superimpose and display the guiding view of the catheterization depth measurement function and the guiding view of the puncture and catheterization path planning function; When the gesture or voice command module receives an operation command for simultaneously displaying the instant artery positioning function, the puncture and catheterization path planning function, and the catheterization depth measurement function while receiving the operation object, the puncture and catheterization guiding operation interface will superimpose and display the guiding view of the instant artery positioning function, the guiding view of the puncture and catheterization path planning function, and the guiding view of the catheterization depth measurement function.

Images