Puncture guiding method and device based on multi-modal image

Abstract

The invention provides a puncture guiding method and device based on a multi-modal image. The method comprises the steps of determining target image acquisition equipment based on lung tumor case data of a current patient, and performing image acquisition on the current patient on a non-magnetization examination bed by adopting the target image acquisition equipment to obtain a first medical image; the target image acquisition equipment is CT equipment or MRI equipment; analyzing the first medical image to obtain a focus area and an organ area of the organ structure; based on the focus area and the organ area, a puncture path is planned to control a puncture instrument to conduct puncture. According to the method, the focus can be automatically positioned and punctured, the defect that the radiation dose is obviously increased due to repeated scanning during conventional CT guidance and evaluation of thermal ablation of the large focus of lung cancer is overcome, and radiation damage caused by repeated CT evaluation is reduced.

Description

Technical Field

[0001] This application relates to the technical field of medical intelligent devices. Specifically, it relates to a puncture guidance method and device based on multimodal imaging. Background Art

[0002] Lung cancer thermal ablation therapy, as a local treatment method, was first applied in the treatment of lung cancer. It mainly includes techniques such as microwave, radiofrequency ablation, and cryoablation under image guidance, and is applicable to patients with primary lung cancer who are not suitable for surgery, as well as patients with oligometastasis or local recurrence. Radiofrequency ablation (RFA) and microwave ablation (MWA) are the two most commonly used thermal ablation methods for lung cancer patients who are not suitable for surgery at present. Currently, CT is still the most important imaging guidance and evaluation method for lung tumor thermal ablation therapy. CT has defects in guiding and evaluating thermal ablation therapy, such as inaccurate positioning, inability to accurately evaluate the ablation effect in real time, and inability to apply multi-parameter functional imaging for evaluation. In addition, CT guidance and evaluation also have the risk of radiation damage. The above deficiencies have affected the reliability and safety of the treatment effect of lung cancer thermal ablation therapy to a certain extent. Summary of the Invention

[0003] The purpose of the embodiments of this application is to provide a puncture guidance method and device based on multimodal imaging, which uses a combined strategy of mainly MRI guidance for large lesions and mainly CT guidance for small lesions by applying image multimodal technology, solving the drawback of significantly increased radiation dose due to repeated scans during conventional CT guidance and evaluation of thermal ablation of large lung cancer lesions; and using CT for guiding and positioning small lesions and then using MRI for guiding real-time thermal ablation evaluation, which not only solves the problems of poor display effect and inaccurate positioning of small lesions by MRI, but also reduces the radiation damage caused by repeated CT evaluations.

[0004] In a first aspect, a puncture guidance method based on multimodal imaging is provided, which is applied to a puncture guidance system including a processor. The puncture guidance system further includes a non-magnetic examination bed equipped with a puncture instrument and different image acquisition devices. The method may include:

[0005] Based on the lung tumor case data of the current patient, determine the target image acquisition device, and use the target image acquisition device to acquire images of the current patient on the non-magnetic examination bed to obtain a first medical image; the target image acquisition device is a CT device or an MRI device;

[0006] Analyze the first medical image to obtain a lesion area and an organ area of the organ structure;

[0007] Based on the lesion area and the organ area, plan a puncture path to control the puncture instrument to perform puncture.

[0008] In a possible implementation, determining a target image acquisition device based on the lung tumor case data of the current patient includes:

[0009] If the lung tumor lesion in the case data meets the preset small lesion condition, determine the CT device as the target image acquisition device;

[0010] If the lung tumor lesion in the case data meets the preset large lesion condition, determine the MRI device as the target image acquisition device.

[0011] In a possible implementation, the organ region includes the skin region, the rib region, and the great vessels region of the heart;

[0012] Planning a puncture path based on the lesion region and the organ region includes:

[0013] Determining a puncture target: Determine the center point of the lesion region as the puncture target;

[0014] Determining a puncture entry point: Determine each point in the region of the skin region except the rib region and the great vessels region of the heart as an initial puncture entry point; and determine the current puncture entry point based on the distance between each initial puncture entry point and the puncture target;

[0015] Analyze the path between the puncture entry point and the puncture target to obtain the entry angle and entry depth at the puncture entry point, so as to control the puncture instrument to perform puncture and entry;

[0016] Use the target image acquisition device to perform image acquisition on the current patient who is being punctured and entered on the non-magnetized examination bed to obtain a second medical image;

[0017] Analyze the second medical image to obtain the positional relationship between the puncture direction of the puncture instrument and the puncture target;

[0018] If the positional relationship between the puncture direction of the second medical image and the puncture target meets the entry angle, continue to control the puncture instrument to perform puncture and entry;

[0019] If the positional relationship between the puncture direction of the second medical image and the puncture target does not meet the entry angle, determine the position after the puncture instrument is punctured and entered as a new puncture entry point, and return to execute the step: Analyze the path between the puncture entry point and the puncture target to obtain the entry angle and entry depth at the puncture entry point, so as to control the puncture instrument to perform puncture and entry.

[0020] In a possible implementation, before determining the puncture target and determining the puncture entry point, the method further includes:

[0021] If the area of the lesion region is greater than a preset segmentation region area threshold, an image segmentation algorithm is used to segment the lesion region to obtain multiple segmented sub-lesion regions;

[0022] Based on the configured region priority rule, determine the priority order of the multiple sub-lesion regions;

[0023] After determining the puncture target point and the puncture needle entry point of the current sub-lesion region according to the priority order, analyze the path between the puncture needle entry point and the puncture target point of the current sub-lesion region to obtain the needle entry angle and the needle entry depth at the puncture needle entry point, so as to control the puncture instrument to perform puncture and needle entry, where the current sub-lesion region is any one of the multiple sub-lesion regions.

[0024] In a possible implementation, the puncture instrument includes a non-magnetized positioning auxiliary bracket, a positioning robotic arm connected to the auxiliary bracket, and a puncture needle at the end of the positioning robotic arm; wherein, both the non-magnetized positioning auxiliary bracket and the positioning robotic arm are connected to the processor.

[0025] In a possible implementation, the puncture guidance system further includes a laser locator; the laser locator is used for positioning and marking on the skin area; the method further includes:

[0026] Based on the positioning mark, provide the user with a puncture path plan based on the skin area, rib area, and great cardiac vessels area.

[0027] In a possible implementation, the method further includes:

[0028] Obtain the CT images collected by the CT device and the MRI images collected by the MRI device of the current patient before and after treatment;

[0029] Perform image analysis on the CT images and MRI images before and after treatment to obtain the lesion region;

[0030] Based on the image data corresponding to the lesion region, determine the lesion quantification index;

[0031] Use the determined lesion quantification index as input data and input it into the trained ablation effect scoring model to obtain the output comprehensive lesion score;

[0032] Based on the comprehensive lesion score, determine the treatment plan.

[0033] In a second aspect, a puncture guidance device based on multi-modal imaging is provided, which is applied to a puncture guidance system including a processor. The puncture guidance system further includes a non-magnetized examination bed equipped with a puncture instrument and different image acquisition devices. The device may include:

[0034] A determination unit, configured to determine a target image acquisition device based on the lung tumor case data of the current patient;

[0035] An acquisition unit, configured to use the target image acquisition device to perform image acquisition on the current patient on the non-magnetized examination bed to obtain a first medical image;

[0036] An analysis unit, configured to analyze the first medical image to obtain a lesion region and an organ region of an organ structure;

[0037] A planning unit, configured to plan a puncture path based on the lesion region and the organ region to control the puncture instrument to perform puncture.

[0038] In a third aspect, an electronic device is provided. The electronic device includes a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete mutual communication through the communication bus;

[0039] The memory is used to store a computer program;

[0040] The processor is configured to implement the method steps described in any one of the first aspects when executing the program stored on the memory.

[0041] In a fourth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and the computer program implements the method steps described in any one of the first aspects when executed by a processor.

[0042] A puncture guidance method based on multi-modal imaging provided by an embodiment of the present application is applied to a puncture guidance system including a processor. The puncture guidance system further includes a non-magnetized examination bed equipped with a puncture instrument and different image acquisition devices. The method includes: determining a target image acquisition device based on the lung tumor case data of the current patient, and using the target image acquisition device to perform image acquisition on the current patient on the non-magnetized examination bed to obtain a first medical image; the target image acquisition device is a CT device or an MRI device; analyzing the first medical image to obtain a lesion region and an organ region of an organ structure; planning a puncture path based on the lesion region and the organ region to control the puncture instrument to perform puncture. This method can automatically locate and puncture the lesion, and in the process of locating and puncturing, a combined strategy of mainly using MRI guidance for large lesions and mainly using CT guidance for small lesions is adopted in the application of multi-modal imaging technology, which solves the drawback of the obvious increase in radiation dose due to repeated scans during conventional CT guidance and evaluation of thermal ablation of large lung cancer lesions; and for small lesions, CT guidance is used for positioning and then MRI is used to guide real-time thermal ablation evaluation, which not only solves the problems of poor display effect and inaccurate positioning of small lesions by MRI, but also reduces the radiation damage caused by repeated CT evaluations. Description of the Drawings

[0043] To more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0044] Figure 1 It is a schematic structural diagram of a puncture guidance system provided by an embodiment of the present application;

[0045] Figure 2 It is a schematic flowchart of a puncture guidance method based on multimodal images provided by an embodiment of the present application;

[0046] Figure 3 It is a schematic structural diagram of a puncture guidance device based on multimodal images provided by an embodiment of the present application;

[0047] Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed Embodiments

[0048] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all of them. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application. Unless otherwise defined, the technical terms or scientific terms used in the present application should have the ordinary meaning understood by those of ordinary skill in the art in the field to which the present invention belongs. The "first", "second" and similar terms used in the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. The terms such as "including" or "comprising" mean that the elements or objects appearing before this word cover the elements or objects listed after this word and their equivalents, without excluding other elements or objects. The terms such as "connected", "coupled" or "linked" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The terms such as "upper", "lower", "left" and "right" are only used to represent relative positional relationships, and when the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0049] Currently, CT has defects in guiding and evaluating thermal ablation therapy, such as inaccurate positioning, inability to accurately evaluate ablation effects in real time, and inability to use multi-parameter functional imaging for evaluation. In addition, CT guidance and evaluation also carry the risk of radiation damage. The above deficiencies have affected the reliability and safety of the thermal ablation treatment effect of lung cancer to a certain extent. When MRI is used to guide thermal ablation, it can image in multiple planes and multiple sequences at any angle. MRI images have high soft tissue resolution and clear anatomical relationships. The application of MRI fast imaging sequences enables MRI to perform real-time guidance for positioning and monitoring and evaluating ablation effects.

[0050] The inventors found that, compared with CT, MRI has no ionizing radiation damage, and its advantage of no radiation damage is more obvious, especially when performing multiple positioning ablations on large lesions. However, due to factors such as respiratory movement and pulsation of the large blood vessels of the heart in the lungs, MRI shows poor results for lung lesions with a diameter of about 1.0 cm or less. Therefore, for small lesions, MRI has deficiencies such as inaccurate lesion positioning or even inability to position due to the inability to display the lesions. CT has high temporal resolution, and the lung tissue has good contrast against the background of air, and can clearly and accurately display lesions with a diameter of less than 1.0 cm and is less affected by motion artifacts. Therefore, CT can be used as a guiding means for small lesions with a diameter ≤ 1.0 cm, which well solves the problem of inaccurate positioning of small lesions by magnetic resonance. By comprehensively applying the multi-modal imaging technology, a combined strategy with MRI guidance for large lesions (diameter > 1.0 cm) and CT guidance for small lesions (diameter ≤ 1.0 cm) well solves the drawback of the significantly increased radiation dose due to repeated scans during the conventional CT-guided and evaluated thermal ablation of large lung cancer lesions. In addition, for small lesions, CT is first used for guiding and positioning, and then MRI is used to guide real-time thermal ablation evaluation, which not only solves the problems of poor display effect and inaccurate positioning of small lesions by MRI, but also reduces the radiation damage caused by repeated CT evaluations. Therefore, a reasonable strategy for guiding ablation treatment is established.

[0051] The puncture guiding method based on multi-modal imaging provided by the embodiments of the present application combines the lesion position in the image through software, accurately calculates the angle and depth of the puncture needle insertion, and accurately determines the actual angle and depth of the needle insertion through software-assisted corresponding automated equipment, so as to more accurately position the lesion, avoid unnecessary operations of repeated punctures due to inaccurate manual punctures, and the damage caused to the patient by multiple repeated operations.

[0052] The puncture guiding method based on multi-modal imaging provided by the embodiments of the present application can be applied in Figure 1 the puncture guiding system shown in Figure 1As shown in the figure, the puncture guidance system may include: a processor, a non-magnetic inspection bed equipped with a puncture instrument and different image acquisition devices that are communicatively connected to the processor. Among them, the puncture instrument may include a non-magnetic positioning auxiliary bracket, a positioning robotic arm connected to the auxiliary bracket, and a puncture needle (such as an ablation needle) at the end of the positioning robotic arm; the non-magnetic positioning auxiliary bracket is used to support the positioning robotic arm, and the positioning robotic arm is used to drive the puncture needle to move. Among them, both the non-magnetic positioning auxiliary bracket and the positioning robotic arm are connected to the processor. The processor may be a server or a terminal. The image acquisition device may include a CT device and an MRI device.

[0053] In order to improve the accuracy of system positioning, the puncture guidance system may further include a laser locator; the laser locator is used to perform positioning marks on the skin area of the patient. Conventionally, when applying MRI-guided puncture, a self-made grid-like cod liver oil particle sticker is used as the skin positioning mark. When applying CT as the image guidance, an iodized oil particle sticker is used as the skin positioning mark. The positioning sticker is pasted on the skin surface near the approximate anatomical position where the lesion is located. Then, routine MRI or / and CT plain scans are performed. According to the spatial position relationship among the cod liver oil / iodized oil mark, the lesion, and the intercostal space shown in the MRI / CT image, the accurate puncture needle insertion position, angle, and insertion depth are calculated. After marking the skin puncture point, the skin is routinely disinfected, a sterile drape is laid, and 5-10 mL of 2% lidocaine is used for local anesthesia of the chest wall at the puncture point. After the anesthesia effect is satisfactory, the skin is incised about 2 mm at the puncture point, and then the ablation needle puncture procedure is performed.

[0054] The preferred embodiments of the present application will be described below in conjunction with the accompanying drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. And without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other.

[0055] Figure 2 It is a schematic flow chart of a puncture guidance method based on multi-modal imaging provided by an embodiment of the present application. As Figure 2 shown, the method may include:

[0056] Step S210: Based on the lung tumor case data of the current patient, determine the target image acquisition device, and use the target image acquisition device to perform image acquisition on the current patient on the non-magnetic inspection bed to obtain the first medical image.

[0057] In specific implementation, it is detected whether the lung tumor lesion in the lung tumor case data belongs to a small lesion or a large lesion; if the lung tumor lesion in the case data meets the preset small lesion conditions, the CT device is determined as the target image acquisition device; if the lung tumor lesion in the case data meets the preset large lesion conditions, the MRI device is determined as the target image acquisition device. A comprehensive application of various imaging examination and evaluation techniques such as fast T1WI, T2WI, DWI / ADC, DCE-MRI, and CT plain scan plus enhanced scan is carried out. Generally, fast T1WI, T2WI, DWI / ADC, and CT plain scan are used as guiding positioning images. For lesions with a diameter greater than 1.0 cm, MRI guidance is mainly used, and for lesions with a diameter less than 1.0 cm or those with poor MRI display, CT guidance is mainly used. Among them, fast T1WI and T2WI can clearly show the location of the lesion and the intercostal space. As for which sequence is mainly used, it is determined by the image results at that time. Functional imaging sequences such as DWI / ADC have great value in clarifying the solid components of the lesion and the approximate boundary of the malignant components of the lesion. Combining with the preoperative dynamic enhanced scan images, the ablation range can be well defined, which has important guiding value for ablation treatment.

[0058] The target image acquisition device is used to perform image acquisition on the current patient on the non-magnetized examination bed, and a first medical image with a visual positioning grid is obtained.

[0059] Step S220: Analyze the first medical image to obtain the lesion area and the organ area of the organ structure.

[0060] The organ area may include the skin area, the rib area, and the great cardiac vessels area.

[0061] Step S230: Based on the lesion area and the organ area, plan the puncture path to control the puncture instrument to perform puncture.

[0062] In specific implementation, the preliminary planning of the puncture path is first carried out: determining the puncture target point: the center point of the lesion area is determined as the puncture target point; and, determining the puncture entry point: each point in the area on the skin area except the rib area and the great cardiac vessels area is determined as the initial puncture entry point; and based on the distance between each initial puncture entry point and the puncture target point, the current puncture entry point is determined. Analyze the path between the puncture entry point and the puncture target point to obtain the entry angle and entry depth at the puncture entry point, so as to control the puncture instrument to perform puncture.

[0063] After that, the re-planning of the puncture path is carried out: the target image acquisition device is used to perform image acquisition on the current patient with puncture on the non-magnetized examination bed, and a second medical image is obtained;

[0064] Analyze the second medical image to obtain the positional relationship between the puncture direction of the puncture instrument and the puncture target; if the positional relationship between the puncture direction and the puncture target shown in the second medical image satisfies the determined needle insertion angle, continue to control the puncture instrument to perform puncture needle insertion; if the positional relationship between the puncture direction and the puncture target shown in the second medical image does not satisfy the determined needle insertion angle, determine the position after the puncture instrument performs puncture needle insertion as the new puncture needle insertion point, and return to execute the steps: analyze the path between the puncture needle insertion point and the puncture target to obtain the needle insertion angle and needle insertion depth at the puncture needle insertion point, so as to control the puncture instrument to perform puncture needle insertion.

[0065] In some embodiments, before determining the puncture target and the puncture needle insertion point, the method further includes: if the regional area of the lesion area is greater than the preset segmentation regional area threshold, an image segmentation algorithm can be used to segment the lesion area to obtain multiple segmented sub-lesion areas; the image segmentation algorithm can be one or a combination of multiple of the snake method, the semi-automatic graph cut method, the partial differential equation based on the second derivative, and the convolutional neural network, or it can be a custom image area equal division method.

[0066] Based on the configured regional priority rule, determine the priority order of the multiple sub-lesion areas; wherein, the regional priority rule can be that the farther the sub-lesion area is from the rib area and the great cardiac vessels area, the higher the priority.

[0067] After determining the puncture target and the puncture needle insertion point of the current sub-lesion area according to the priority order, analyze the path between the puncture needle insertion point and the puncture target of the current sub-lesion area to obtain the needle insertion angle and needle insertion depth at the puncture needle insertion point, so as to control the puncture instrument to perform puncture needle insertion, and the current sub-lesion area is any one of the multiple sub-lesion areas.

[0068] Further, after executing step S230, obtain the CT images collected under the CT device and the MRI images collected by the MRI device of the current patient before and after treatment;

[0069] Image analysis is performed on CT images and MRI images before and after treatment to obtain the lesion area; based on the image data corresponding to the lesion area, lesion quantification indicators are determined; the lesion quantification indicators may include: the change rate of lesion volume, edge sharpness, the change in the restricted state of tissue water molecule diffusion (based on DWI / ADC imaging), and blood supply change (combined with DCE-MRI), etc. Among them, the change rate of lesion volume is used to evaluate the ablation effect. The lesion areas before and after treatment are determined using a lesion recognition model, and their respective volumes are calculated (which can be estimated by multiplying the number of pixels by the actual volume represented by each pixel). Then the formula: (pre-treatment volume - post-treatment volume) / pre-treatment volume * 100% is used to calculate the volume change rate. The boundary sharpness reflects the contrast between the ablation boundary and the surrounding normal tissue, which helps to evaluate whether the ablation is complete. Edge detection algorithms (such as Sobel operator, Canny edge detection, etc.) can be used to process at the lesion edge, and then the edge intensity or edge width is calculated. The higher the edge intensity or the narrower the edge, the clearer the boundary and the better the ablation effect. The change in the restricted state of tissue water molecule diffusion can reflect the changes in cell density and tissue structure by analyzing the changes in the restricted diffusion of water molecules. The ADC (Apparent Diffusion Coefficient) values of the lesion area before and after treatment are calculated using DWI / ADC images. The ADC value can reflect the diffusion ability of water molecules in tissue. Usually, as cells die, the ADC value increases. Calculate (post-treatment ADC mean - pre-treatment ADC mean) / pre-treatment ADC mean to measure the change in the ADC value. The blood supply change is used to evaluate tumor activity by analyzing the changes in blood perfusion parameters in the lesion area. Using DCE-MRI data, parameters such as Ktrans (vascular permeability) and Ve (extracellular fluid volume ratio) of the lesion area are extracted through a pharmacokinetic model (such as the Tofts model). The changes in the above parameters before and after treatment are compared to evaluate the impact of ablation on tumor blood supply.

[0070] Fast T1-TFE, mDIXON, T2-TSE, and DWI / ADC imaging in MRI images are applied to the dynamic real-time evaluation of thermal ablation of lung cancer to preliminarily judge liquefied necrotic tissue and tumor active tissue. After the conventional sequence indicates successful ablation, the DCE-MRI sequence is added for scanning, and combined with sequences such as DWI / ADC to further evaluate the ablation treatment of lung cancer. The conventional sequence is used to display the tumor ablation boundary, the edema zone around the ablation focus, tissue hemorrhage and necrosis, etc. Functional imaging sequences such as DWI / ADC can, at the cellular level, through the change in the restricted state of water molecule diffusion, display the degree of damage to tumor tissue by thermal ablation in real time. DWI / ADC combined with DCE-MRI further clarifies the ablation effect through the change in tumor tissue blood supply, achieving the purpose of accurately evaluating the ablation treatment effect. And the CT image data after surgery is used for supplementary evaluation, making up for the deficiency of MRI in monitoring complications such as pneumothorax.

[0071] After that, the determined lesion quantification index is used as input data and input into the trained ablation effect scoring model to obtain the comprehensive score of the output lesion. Based on the comprehensive score of the lesion, a treatment plan is determined, that is, according to the comprehensive score of the lesion and combined with the actual clinical situation (such as the patient's overall health status, tumor characteristics, and the patient's wishes and preferences, etc.), a scientific basis is provided for the subsequent treatment plan, such as whether further treatment or observation is required. Among them, high score (good response): If the ablation effect score of the patient is high, it means that the lesion area has been effectively treated and the tumor cells have been effectively destroyed. At this time, the doctor may recommend an observation and waiting strategy, and perform imaging examinations regularly to monitor the changes in the condition to ensure that there is no recurrence or new lesions. Medium score (partial response): For patients with a medium score, it indicates that although there is a certain therapeutic effect, there may be lesion tissues that have not been completely ablated. In this case, further local treatment may be required, such as performing thermal ablation again, surgical resection, or other forms of local treatment (such as radiotherapy), combined with other adjuvant treatment methods. Low score (adverse reaction): If the patient obtains a low score, it means that the thermal ablation treatment fails to achieve the expected effect, and the lesion may be hardly affected or has progressed. In this situation, a more aggressive treatment plan should be considered, including but not limited to surgical operations, systemic treatments (such as chemotherapy, targeted therapy, or immunotherapy), etc., and it may be necessary to discuss and formulate a personalized treatment strategy by a multidisciplinary team.

[0072] In some embodiments, image analysis is performed on CT images and MRI images before and after treatment to obtain the lesion area, including: extracting features from CT images and MRI images before and after treatment; inputting the extracted features into the trained lesion recognition model to obtain the output lesion area.

[0073] Among them, the training process of the lesion recognition model includes:

[0074] Collect CT images and MRI images of different historical patients before and after treatment;

[0075] For any image, it is labeled by a professional radiologist to obtain labeled data to clarify the location, size, and morphological characteristics of the lesion;

[0076] Use a pre-trained convolutional neural network to extract features from CT images and MRI images;

[0077] Fuse the extracted features through a shared fully-connected layer or an attention mechanism; among them, the feature fusion methods include: early fusion, late fusion, and intermediate fusion. Among them, early fusion: directly splice the features of CT images and MRI images as the input of a convolutional neural network (CNN). Late fusion: First, extract features from each modality of data separately, and then fuse them at the high-level feature level. For example, after extracting features through independent CNNs respectively, splice the feature maps of the last layer. Intermediate fusion: Perform feature fusion at different levels of the CNN. For example, perform feature map splicing or element-wise addition operations after several convolutional layers.

[0078] Based on the fused features and the labeled data, train the neural network model to be trained to obtain a trained lesion recognition model.

[0079] In some embodiments, before re-planning the puncture path, a real-time monitoring system can be introduced. Using technologies such as high-frame-rate ultrasound or fast-scanning MRI, obtain corresponding image data in real time, and perform analysis to identify the displacement changes of the lesion area caused by respiration or other physiological activities. According to this displacement change, automatically re-obtain the positional relationship between the puncture direction of the puncture instrument and the puncture target, so as to calculate the optimal puncture path. This may involve fine-tuning the angle and depth of the puncture needle or completely re-positioning the puncture point to ensure that the puncture needle always advances accurately towards the lesion area.

[0080] In some embodiments, the puncture guidance system may further include an augmented reality (AR) device and a voice receiving device.

[0081] The augmented reality (AR) device is used to provide a more intuitive operation interface for the doctor using AR technology, and can display the position information of the puncture target and the puncture point in the current planned puncture path, as well as the corresponding puncture angle and puncture depth.

[0082] The voice receiving device is used to receive the voice commands for the doctor to control the puncture instrument.

[0083] Corresponding to the above method, an embodiment of the present application further provides a puncture guidance device based on multi-modal images, as Figure 3 shown, the device includes:

[0084] A determination unit 310, configured to determine a target image acquisition device based on the lung tumor case data of the current patient;

[0085] An acquisition unit 320, configured to acquire an image of the current patient on the non-magnetized examination bed using the target image acquisition device to obtain a first medical image;

[0086] An analysis unit 330 for analyzing the first medical image to obtain a lesion region and an organ region of an organ structure;

[0087] A planning unit 340 for planning a puncture path based on the lesion region and the organ region to control the puncture instrument to perform puncture.

[0088] The functions of the functional units of the puncture guiding device based on multimodal images provided in the above embodiments of the present application can be implemented by the above method steps. Therefore, the specific working processes and beneficial effects of each unit in the puncture guiding device based on multimodal images provided in the embodiments of the present application will not be repeated here.

[0089] The embodiments of the present application also provide an electronic device, as Figure 4 shown, including a processor 410, a communication interface 420, a memory 430, and a communication bus 440. Among them, the processor 410, the communication interface 420, and the memory 430 complete communication with each other through the communication bus 440.

[0090] The memory 430 is used to store a computer program;

[0091] When the processor 410 is used to execute the program stored on the memory 430, the following steps are implemented:

[0092] Based on the lung tumor case data of the current patient, determine a target image acquisition device, and use the target image acquisition device to acquire an image of the current patient on the non-magnetized examination bed to obtain a first medical image;

[0093] Analyze the first medical image to obtain a lesion region and an organ region of an organ structure;

[0094] Based on the lesion region and the organ region, plan a puncture path to control the puncture instrument to perform puncture.

[0095] The above-mentioned communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity, only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0096] The communication interface is used for communication between the above electronic device and other devices.

[0097] The memory may include a Random Access Memory (RAM), or may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0098] The aforementioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0099] Since the implementation manners and beneficial effects of each component of the electronic device in the above embodiments for solving problems can be seen from Figure 2 the steps in the shown embodiments, therefore, the specific working process and beneficial effects of the electronic device provided in the embodiments of the present application will not be repeated here.

[0100] In another embodiment provided by the present application, there is also provided a computer-readable storage medium, in which instructions are stored, and when it runs on a computer, it causes the computer to execute the multi-modal image-based puncture guidance method described in any one of the above embodiments.

[0101] In another embodiment provided by the present application, there is also provided a computer program product containing instructions, and when it runs on a computer, it causes the computer to execute the multi-modal image-based puncture guidance method described in any one of the above embodiments.

[0102] Those skilled in the art should understand that the embodiments in the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the embodiments in the embodiments of the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments in the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0103] Embodiments of the present application are described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in multiple blocks.

[0104] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in multiple blocks.

[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in multiple blocks.

[0106] Although the preferred embodiments in the embodiments of the present application have been described, those skilled in the art can make additional changes and modifications once they know the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0107] Obviously, those skilled in the art can make various changes and variations to the embodiments in the embodiments of the present application without departing from the spirit and scope of the embodiments in the embodiments of the present application. Thus, if these modifications and variations of the embodiments in the embodiments of the present application fall within the scope of the claims of the embodiments of the present application and their equivalent technologies, the embodiments of the present application are also intended to include these changes and variations.

Claims

1. A puncture guidance method based on multimodal images, characterized in that, Applied to a puncture guidance system including a processor, the puncture guidance system further including a non-magnetic examination bed equipped with a puncture instrument and different image acquisition devices, the method includes: Based on the lung tumor case data of the current patient, determine the target image acquisition device, and use the target image acquisition device to perform image acquisition on the current patient on the non-magnetic examination bed to obtain a first medical image; the target image acquisition device is a CT device or an MRI device; Analyze the first medical image to obtain the lesion area and the organ area of the organ structure; Based on the lesion area and the organ area, plan a puncture path to control the puncture instrument to perform puncture.

2. The method according to claim 1, wherein Based on the lung tumor case data of the current patient, determining the target image acquisition device includes: If the lung tumor lesion in the case data meets the preset small lesion condition, determine the CT device as the target image acquisition device; If the lung tumor lesion in the case data meets the preset large lesion condition, determine the MRI device as the target image acquisition device.

3. The method according to claim 1, wherein The organ area includes the skin area, the rib area, and the great cardiac vessels area; Based on the lesion area and the organ area, planning the puncture path includes: Determine the puncture target: Determine the center point of the lesion area as the puncture target; Determine the puncture entry point: Determine each point in the area on the skin area except the rib area and the great cardiac vessels area as the initial puncture entry point; and based on the distance between each initial puncture entry point and the puncture target, determine the current puncture entry point; Analyze the path between the puncture entry point and the puncture target to obtain the entry angle and entry depth at the puncture entry point to control the puncture instrument to perform puncture; Use the target image acquisition device to perform image acquisition on the current patient with a puncture on the non-magnetic examination bed to obtain a second medical image; Analyze the second medical image to obtain the positional relationship between the puncture direction of the puncture instrument and the puncture target; If the positional relationship between the puncture direction of the second medical image and the puncture target meets the entry angle, continue to control the puncture instrument to perform puncture; If the positional relationship between the puncture direction of the second medical image and the puncture target does not meet the entry angle, determine the position after the puncture instrument is punctured as the new puncture entry point, and return to execute the step: Analyze the path between the puncture entry point and the puncture target to obtain the entry angle and entry depth at the puncture entry point to control the puncture instrument to perform puncture.

4. The method according to claim 3, characterized in that Before determining the puncture target and determining the puncture entry point, the method further includes: If the area of the lesion area is greater than the preset segmentation area threshold, use an image segmentation algorithm to segment the lesion area to obtain multiple segmented sub-lesion areas; Based on the configured area priority rule, determine the priority order of the multiple sub-lesion areas; According to the said priority order, after determining the puncture target point and the puncture needle insertion point of the current sub-lesion area, analyze the path between the puncture needle insertion point and the puncture target point of the current sub-lesion area to obtain the needle insertion angle and the needle insertion depth at the puncture needle insertion point, so as to control the puncture instrument to perform puncture needle insertion. The current sub-lesion area is any one of multiple sub-lesion areas.

5. The method according to claim 3, wherein The puncture instrument includes a non-magnetized positioning auxiliary bracket, a positioning robotic arm connected to the auxiliary bracket, and a puncture needle at the end of the positioning robotic arm; wherein, both the non-magnetized positioning auxiliary bracket and the positioning robotic arm are connected to the processor.

6. The method according to claim 5, wherein The puncture guidance system further includes a laser locator; the laser locator is used for positioning and marking on the skin area. The method further includes: Based on the positioning mark, provide the user with a puncture path plan based on the skin area, rib area, and great cardiac vessels area.

7. The method according to claim 1, wherein The method further includes: Obtain the CT images collected by the CT device and the MRI images collected by the MRI device of the current patient before and after treatment. Perform image analysis on the CT images and MRI images before and after treatment to obtain the lesion area. Based on the image data corresponding to the lesion area, determine the lesion quantification index. Take the determined lesion quantification index as input data and input it into the trained ablation effect scoring model to obtain the output comprehensive lesion score. Based on the comprehensive lesion score, determine the treatment plan.

8. A puncture guiding device based on multimodal imaging, characterized in that, Applied to a puncture guidance system including a processor, the puncture guidance system further includes a non-magnetized examination bed equipped with a puncture instrument and different image acquisition devices. The device includes: A determination unit, configured to determine the target image acquisition device based on the lung tumor case data of the current patient. An acquisition unit, configured to use the target image acquisition device to acquire images of the current patient on the non-magnetized examination bed to obtain the first medical image. An analysis unit, configured to analyze the first medical image to obtain the lesion area and the organ area of the organ structure. A planning unit, configured to plan a puncture path based on the lesion area and the organ area to control the puncture instrument to perform puncture.

9. An electronic device, characterized in that, The electronic device includes a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus. The memory is used for storing computer programs. The processor is configured to implement the method according to any one of claims 1-7 when executing the program stored on the memory.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the method according to any one of claims 1-7.

Images