Robotic navigation system and device for breast puncture surgery based on multimodal image fusion

Abstract

The present invention belongs to the field of medical robotics technology, and specifically relates to a robotic navigation system and device for breast puncture surgery based on multimodal image fusion. This invention achieves high-precision three-dimensional reconstruction of breast tissue by aligning and fusing multimodal image data, providing doctors with more intuitive and accurate surgical navigation information. By tracking the position of the puncture needle in real time and spatially aligning it with the image data, the accuracy and safety of the surgery are improved. At the same time, by collecting real-time feedback information on the puncture force of the puncture needle during the needle movement, the effectiveness of the puncture navigation path is evaluated. When risks are found in the puncture path, dynamic adjustments are immediately made to replan a new puncture path that bypasses obstacles and reaches the lesion. If a safe path cannot be obtained, the system automatically stops the needle insertion and gives an audible and visual warning prompt, thereby avoiding surgical risks caused by human operational errors in traditional surgery.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical robots, and in particular relates to a breast puncture surgery robot navigation system and device based on multimodal image fusion. Background Art

[0002] Traditional breast biopsy surgery has problems such as inaccurate positioning and complex operation, which has caused great trouble to doctors and patients. However, with the continuous advancement of medical imaging technology, more accurate and safe navigation methods have been provided for breast biopsy surgery. Based on this technology, the efficiency and success rate of breast biopsy surgery have been significantly improved.

[0003] In the existing technology, a single medical imaging technology is mostly used for navigation, such as ultrasound imaging, MRI or CT, but each imaging technology has its limitations. For example, ultrasound imaging is easily affected by tissue density and gas interference. Although MRI can provide high-resolution soft tissue images, the equipment is expensive and complicated to operate. Multiple CT scans pose the risk of excessive radiation. In addition, during the puncture process, there is a lack of force feedback information to conduct real-time evaluation of the interaction state between the puncture needle and the tissue, which will undoubtedly lead to increased uncertainty during the operation. Based on this, the present invention proposes a breast puncture surgery robot navigation system based on multimodal image fusion to solve the above problems. Summary of the Invention

[0004] The purpose of the present invention is to provide a breast puncture surgery robot navigation system and device based on multimodal image fusion, which can fuse ultrasound images, MRI images and CT images, and evaluate the effectiveness of the puncture navigation path in real time according to the puncture force feedback information, and dynamically adjust the puncture path to improve the accuracy and safety of the operation.

[0005] The technical solutions adopted by the present invention are as follows:

[0006] The robotic navigation system for breast biopsy surgery based on multimodal image fusion includes:

[0007] An image acquisition module is used to obtain multimodal image data of the patient's breast, wherein the multimodal image data includes ultrasound images, MRI images, and CT images;

[0008] The path pre-planning module is used to perform registration and fusion processing on multimodal image data to generate fused image data, and then pre-plan the puncture navigation path based on the fused image data;

[0009] Position registration module, used to collect the position information of the puncture needle in real time, and superimpose the position information of the puncture needle on the fused image data to generate a real-time navigation image;

[0010] The path evaluation module is used to collect the puncture force feedback information of the puncture needle in real time during the needle movement process, and evaluate the effectiveness of the implementation of the puncture navigation path based on the puncture force feedback information;

[0011] A path adjustment module is used to control the robot to perform the puncture surgery along the puncture navigation path when the puncture navigation path is implemented effectively, and vice versa, dynamically adjust the puncture navigation path according to the real-time navigation image;

[0012] The sound and light warning module is used to send out sound and light warning signals when the puncture operation cannot be continued.

[0013] In a preferred embodiment, the image acquisition module uses non-invasive image acquisition technology to perform image acquisition, wherein ultrasound images are used to provide high-resolution structural information of breast tissue, MRI images are used to display soft tissue details inside the breast, and CT images are used to display calcification information inside the breast.

[0014] In a preferred embodiment, the path pre-planning module includes a pre-processing unit and a fusion unit, wherein the pre-processing unit is used to pre-process the ultrasound image, the MRI image, and the CT image respectively, and the pre-processing includes denoising, contrast enhancement, and geometric distortion correction;

[0015] The fusion unit is used to perform grayscale normalization on the preprocessed ultrasound images, MRI images and CT images to eliminate the grayscale differences among the ultrasound images, MRI images and CT images;

[0016] The pre-processed ultrasound images, MRI images, and CT images are preliminarily aligned using a three-dimensional spatial coordinate system, and then fine-tuned and calibrated for local tissue deformation.

[0017] The elastic characteristics of ultrasound and the soft tissue characteristics of MRI are layered and fused, and the calcification characteristics of CT and the high-resolution structural information of breast tissue are superimposed and fused to generate fused image data;

[0018] The fused image data is reconstructed in three dimensions to generate a navigation reference map that includes the shape of blood vessels, the three-dimensional boundaries of lesions, and safe passage annotations.

[0019] In a preferred embodiment, the path pre-planning module further includes a pre-planning unit, which is used to plan an initial puncture path that avoids blood vessels and reaches the lesion based on the blood vessel distribution and the three-dimensional boundary of the lesion in the navigation reference image, wherein multiple initial puncture paths are provided;

[0020] Each initial puncture path is segmented to obtain multiple needle segments, and the puncture angle corresponding to each needle segment and the distance between the puncture needle and the surrounding tissue are collected;

[0021] Normalizing the puncture angle and the distance between the puncture needle and the surrounding tissue to obtain a first evaluation index corresponding to the puncture angle and a second evaluation index corresponding to the distance between the puncture needle and the surrounding tissue;

[0022] Comprehensively evaluate the first evaluation index and the second evaluation index under each needle movement segment, and simultaneously output the comprehensive evaluation score of each initial puncture path;

[0023] Sort the comprehensive evaluation scores from high to low, and select the initial puncture path with the highest comprehensive evaluation score as the pre-planned puncture navigation path;

[0024] Among them, if there are multiple initial puncture paths with the highest comprehensive evaluation scores and the same score, the initial puncture path closest to the lesion center is selected as the pre-planned puncture navigation path.

[0025] In a preferred embodiment, the position registration module includes a tracking unit and a correction unit;

[0026] The tracking unit tracks the position of the puncture needle in real time through the optical positioner installed on the robot arm, and synchronously outputs the 3D spatial coordinates and attitude angle of the puncture needle;

[0027] The three-dimensional spatial coordinates and posture angles of the puncture needle are spatially registered with the fused image data, and the position information of the puncture needle is superimposed on the fused image data to form a real-time navigation image;

[0028] The correction unit is used to compare the real-time needle path under the real-time navigation image with the pre-planned needle path, output needle deviation information, and generate a correction path including a needle bending compensation amount when the needle deviation information exceeds a preset deviation threshold;

[0029] The bending compensation amount is proportionally distributed to each segment of the puncture needle, and the puncture needle is controlled to puncture along the corrected path until the puncture needle reaches the target position.

[0030] In a preferred embodiment, the step of generating a correction path including a needle bending compensation amount includes:

[0031] The three-dimensional coordinates and posture angles of the puncture needle are acquired in real time, and the spatial vector is compared with the reference coordinates under the pre-planned needle path, and the needle deviation information is output. The needle deviation information includes the lateral offset, longitudinal depth deviation and needle tip direction angle deviation;

[0032] Compare the lateral offset, longitudinal depth deviation, and needle tip angular deviation with corresponding preset deviation thresholds, respectively. When any one of the lateral offset, longitudinal depth deviation, and needle tip angular deviation exceeds the corresponding preset deviation threshold, trigger a correction of the puncture path.

[0033] Obtain a compensation amount calculation function, input the lateral offset and the longitudinal depth deviation into the compensation amount calculation function, and output the lateral offset compensation amount and the depth direction compensation amount of the puncture needle;

[0034] According to the needle tip direction angle deviation, the bending shape adjustment parameters of the puncture needle are determined, and a correction path including the needle body bending compensation amount is generated.

[0035] In a preferred embodiment, the path assessment module includes a risk assessment unit, which is used to obtain the axial force, lateral force and torque of the puncture needle during the needle movement in real time, and output the change trend of the axial force, lateral force and torque in real time;

[0036] Compare the changing trends of the axial force, lateral force, and torque with the corresponding preset safety thresholds. If any of the changing trends of the axial force, lateral force, or torque exceeds the corresponding preset safety threshold, it is directly determined that the currently executed puncture navigation path is risky and the robot stops puncture.

[0037] On the contrary, the changing trends of axial force, lateral force and torque are fused and the fusion evaluation index is output synchronously;

[0038] The fusion evaluation index is compared with the preset comprehensive evaluation interval. When the fusion evaluation index is within the comprehensive evaluation interval, the currently executed puncture navigation path is determined to be valid, and the puncture operation is continued along the currently executed puncture navigation path. Otherwise, the currently executed puncture navigation path is determined to be risky, and the robot's puncture is stopped.

[0039] In a preferred embodiment, the path adjustment module includes a path adjustment unit, which immediately performs dynamic adjustment of the puncture navigation path when there is a risk in the currently executed puncture navigation path;

[0040] Re-acquire ultrasound images to detect displacement and deformation of breast tissue caused by puncture, and locally register them with MRI and CT images to update the fused image data of the unpunctured area;

[0041] In the updated fused image data, the obstacle avoidance area is delineated with the current needle travel point as the center point. The obstacle avoidance area is circular, and the radius of the obstacle avoidance area is twice the distance from the needle tip to the center point when the change trend of the axial force, lateral force, and torque changes suddenly.

[0042] Based on the updated fused image data and obstacle avoidance area, a new puncture path that bypasses the obstacle avoidance area and reaches the lesion is replanned, and subsequent puncture operations are performed according to the new puncture path.

[0043] In a preferred embodiment, the sound and light warning signal includes a sound signal and a light signal, the sound signal is a buzzing sound of a preset frequency, and the light signal is a flashing light of a preset color. The sound and light warning signal is issued by a sound and light alarm installed in the operating room.

[0044] The present invention also provides a breast puncture surgery robot navigation device based on multimodal image fusion, the navigation device comprising:

[0045] at least one processor;

[0046] and a memory communicatively coupled to the at least one processor;

[0047] In which, the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned breast puncture surgery robot navigation system based on multimodal image fusion.

[0048] The technical effects achieved by the present invention are:

[0049] The present invention achieves high-precision three-dimensional reconstruction of breast tissue by aligning and fusing multimodal imaging data, providing doctors with more intuitive and accurate surgical navigation information. By tracking the position of the puncture needle in real time and spatially aligning it with the fused imaging data, doctors can clearly see the specific position of the puncture needle in the breast tissue, thereby improving the accuracy and safety of the operation. At the same time, the present invention also evaluates the effectiveness of the implementation of the puncture navigation path by collecting the puncture force feedback information of the puncture needle in real time during the needle operation. When risks are found in the puncture path, dynamic adjustments will be made immediately to re-plan a new puncture path that bypasses the obstacle area and reaches the lesion. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a schematic diagram of the system modules of the present invention;

[0051] Figure 2 It is a structural schematic diagram of the navigation device of the present invention. DETAILED DESCRIPTION

[0052] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0053] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0054] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in a preferred embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it constitute a separate or selective embodiment that is mutually exclusive of other embodiments.

[0055] See also Figure 1 As shown, the present invention provides a robotic navigation system for breast puncture surgery based on multimodal image fusion, comprising:

[0056] An image acquisition module is used to obtain multimodal image data of the patient's breast, wherein the multimodal image data includes ultrasound images, MRI images, and CT images;

[0057] When breast puncture is required, the image acquisition module is executed first to comprehensively obtain multimodal image data of the patient's breast. The multimodal image data covers a variety of imaging methods, including ultrasound images, MRI images and CT images, to ensure that all-round information of breast tissue is obtained. Among them, the image acquisition module uses non-invasive image acquisition technology to perform image acquisition. Among them, ultrasound images are used to provide high-resolution structural information of breast tissue. Through the reflection and refraction of ultrasound, the fine structure of breast tissue can be clearly presented. MRI images are used to display the soft tissue details inside the breast. Using strong magnetic fields and radio frequency pulses, MRI can generate high-contrast images, showing in detail the soft tissue distribution and lesions inside the breast. CT images are used to display calcification information inside the breast. Through X-ray penetration and computer reconstruction, CT can accurately identify and locate calcification foci in the breast.

[0058] The path pre-planning module is used to perform registration and fusion processing on multimodal image data to generate fused image data, and then pre-plan the puncture navigation path based on the fused image data;

[0059] When the path pre-planning module is executed, it will perform corresponding registration and fusion processing on the acquired multimodal image data to output fused image data that can reflect breast tissue. On this basis, it will also pre-plan the puncture navigation path for subsequent puncture to ensure the safety of the subsequent puncture process. The path pre-planning module includes a pre-processing unit and a fusion unit. The pre-processing unit is used to pre-process the ultrasound image, MRI image and CT image respectively. The pre-processing includes denoising, contrast enhancement and correction of geometric distortion.

[0060] The fusion unit is used to perform grayscale normalization on the preprocessed ultrasound images, MRI images and CT images to eliminate the grayscale differences among the ultrasound images, MRI images and CT images;

[0061] The pre-processed ultrasound images, MRI images, and CT images are preliminarily aligned using a three-dimensional spatial coordinate system, and then fine-tuned and calibrated for local tissue deformation.

[0062] The elastic characteristics of ultrasound and the soft tissue characteristics of MRI are layered and fused, and the calcification characteristics of CT and the high-resolution structural information of breast tissue are superimposed and fused to generate fused image data;

[0063] Perform 3D reconstruction on the fused image data to generate a navigation reference map that includes the shape of blood vessels, lesion boundaries, and safe passage annotations;

[0064] Specifically, when performing registration and fusion processing on multimodal image data, first pre-processing operations are performed on image data of each type. For ultrasound images, since they are easily interfered by human tissues, artifacts will appear. Therefore, they need to be geometrically distorted to ensure the accuracy of the image. For MRI images and CT images, although the imaging effects are relatively clear, they still need to be denoised to further improve the image quality. After completing the pre-processing operation, in order to enable different types of image data to be compared and analyzed in the same coordinate system, they need to be grayscale normalized to eliminate the grayscale differences between different images, so that the grayscale values ​​of various images are consistent, laying the foundation for subsequent fusion processing. Then, the pre-processed ultrasound images, MRI images and The CT images are initially aligned, and on this basis, fine-tuning and calibration are performed according to the deformation of local tissues to ensure high consistency in the spatial positions of various images. Then, the elastic features in the ultrasound images and the soft tissue features in the MRI images are layered and fused to fully utilize the advantages of both information. At the same time, the calcification features in the CT images are superimposed and fused with the high-resolution structural information of the breast tissue to further enhance the details of the fused image, and ultimately generate fused image data that integrates multiple image information. Finally, the generated fused image data is reconstructed in three dimensions to construct a stereoscopic image model containing rich information, which not only includes the shape of the blood vessels, but also marks the three-dimensional boundaries of the lesions and the location of the safe passage. Finally, the corresponding navigation reference map is generated to provide a reliable navigation basis for subsequent puncture surgery.

[0065] Secondly, the path pre-planning module also includes a pre-planning unit, which is used to plan an initial puncture path that avoids blood vessels and reaches the lesion based on the blood vessel distribution and the three-dimensional boundary of the lesion in the navigation reference map. There are multiple initial puncture paths.

[0066] Each initial puncture path is segmented to obtain multiple needle segments, and the puncture angle corresponding to each needle segment and the distance between the puncture needle and the surrounding tissue are collected;

[0067] Normalizing the puncture angle and the distance between the puncture needle and the surrounding tissue to obtain a first evaluation index corresponding to the puncture angle and a second evaluation index corresponding to the distance between the puncture needle and the surrounding tissue;

[0068] Comprehensively evaluate the first evaluation index and the second evaluation index under each needle movement segment, and simultaneously output the comprehensive evaluation score of each initial puncture path;

[0069] Sort the comprehensive evaluation scores from high to low, and select the initial puncture path with the highest comprehensive evaluation score as the pre-planned puncture navigation path;

[0070] If there are multiple initial puncture paths with the highest and same comprehensive evaluation scores, the initial puncture path closest to the lesion center is selected as the pre-planned puncture navigation path;

[0071] In this embodiment, when pre-planning the puncture navigation path, first, based on the blood vessel distribution and the three-dimensional boundary information of the lesion presented in the navigation reference map, multiple initial puncture paths that can effectively avoid blood vessels and accurately reach the lesion are generated, and then each initial puncture path is segmented and divided into multiple needle segments, and the puncture angle data corresponding to each needle segment and the distance between the puncture needle and the surrounding tissue are collected one by one, and the collected puncture angle and distance data between the puncture needle and the surrounding tissue are normalized to obtain a first evaluation index corresponding to the puncture angle and a second evaluation index corresponding to the distance between the puncture needle and the surrounding tissue. Then, a comprehensive comprehensive evaluation is performed on the first evaluation index and the second evaluation index under each needle segment. The comprehensive evaluation score of each initial puncture path is evaluated and output synchronously during the evaluation process. Specifically, the first evaluation index and the second evaluation index can be weighted and summed by a fusion weighted algorithm to calculate the final comprehensive evaluation score. The higher the comprehensive evaluation score, the higher the feasibility of the corresponding initial puncture path. The comprehensive evaluation scores are then arranged in descending order, and the initial puncture path with the highest comprehensive evaluation score is selected as the pre-planned puncture navigation path. It should be noted that if multiple initial puncture paths with the highest and identical comprehensive evaluation scores are found during the sorting process, the initial puncture path closest to the center of the lesion should be further selected as the final pre-planned puncture navigation path to ensure the accuracy of the puncture operation.

[0072] Position registration module, used to collect the position information of the puncture needle in real time, and superimpose the position information of the puncture needle on the fused image data to generate a real-time navigation image;

[0073] When the position registration module is executed, it will collect the position information of the puncture needle in real time and superimpose the position information of the puncture needle on the fused image data in real time, thereby generating a dynamic real-time navigation image to provide intuitive visual guidance for doctors. The position registration module includes a tracking unit and a correction unit;

[0074] The tracking unit tracks the position of the puncture needle in real time through the optical positioner installed on the robot arm, and synchronously outputs the 3D spatial coordinates and attitude angle of the puncture needle;

[0075] The three-dimensional spatial coordinates and posture angles of the puncture needle are spatially registered with the fused image data, and the position information of the puncture needle is superimposed on the fused image data to form a real-time navigation image;

[0076] The correction unit is used to compare the real-time needle path under the real-time navigation image with the pre-planned needle path, output needle deviation information, and generate a correction path including a needle bending compensation amount when the needle deviation information exceeds a preset deviation threshold;

[0077] Proportionally distribute the bending compensation amount to each segment of the puncture needle, and control the puncture needle to puncture along the corrected path until the puncture needle reaches the target position;

[0078] Specifically, when generating a real-time navigation image, the specific position of the puncture needle is first tracked in real time by a high-precision optical positioner installed on the robot arm, and the coordinates of the puncture needle in three-dimensional space and its attitude angle information are synchronously output to ensure the real-time and accuracy of the puncture needle position. The three-dimensional spatial coordinates of the puncture needle and its attitude angle are then spatially aligned with the fused image data to seamlessly superimpose the position information of the puncture needle onto the fused image data to form a real-time updated real-time navigation image, which can intuitively display the real-time position and direction of the puncture needle in the breast. The actual needle path of the puncture needle under the real-time navigation image is then compared with the pre-planned needle path, and the needle deviation information is output in real time. When the needle deviation information exceeds the preset deviation threshold, a correction path including the needle body bending compensation amount is automatically generated to correct the deviation of the puncture needle and ensure the accuracy of the puncture process. Finally, the calculated bending compensation amount is reasonably distributed to each segment of the puncture needle in proportion ( , where Indicates the The compensation amount allocated to each segment, Indicates the The length of the segment, Represents the attenuation coefficient, which is used to reduce the terminal cumulative error. represents the total number of puncture needle segments, which are specifically segmented according to the stiffness of the needle body), and the robot control system accurately controls the puncture needle to perform the puncture operation along the corrected path until the puncture needle successfully reaches the predetermined target position.

[0079] Next, the steps of generating a correction path including a needle bending compensation amount include:

[0080] The three-dimensional coordinates and posture angles of the puncture needle are acquired in real time, and the spatial vector is compared with the reference coordinates under the pre-planned needle path, and the needle deviation information is output. The needle deviation information includes the lateral offset, longitudinal depth deviation and needle tip direction angle deviation;

[0081] Compare the lateral offset, longitudinal depth deviation, and needle tip angular deviation with corresponding preset deviation thresholds, respectively. When any one of the lateral offset, longitudinal depth deviation, and needle tip angular deviation exceeds the corresponding preset deviation threshold, trigger a correction of the puncture path.

[0082] Obtain a compensation amount calculation function, input the lateral offset and the longitudinal depth deviation into the compensation amount calculation function, and output the lateral offset compensation amount and the depth direction compensation amount of the puncture needle;

[0083] According to the needle tip direction angle deviation, the bending shape adjustment parameters of the puncture needle are determined, and a correction path including the needle body bending compensation amount is generated;

[0084] In this embodiment, under the guidance of the real-time navigation image, the actual trajectory of the puncture needle will be continuously tracked, and the real-time needle path will be compared and analyzed with the pre-planned needle path, so that the needle deviation information can be accurately output. The needle deviation information covers multiple dimensions such as lateral offset, longitudinal depth deviation and needle tip direction angle deviation. At the same time, the lateral offset, longitudinal depth deviation and needle tip direction angle deviation will be compared with their corresponding preset deviation thresholds respectively. Once it is found that any one of the lateral offset, longitudinal depth deviation or needle tip direction angle deviation exceeds its corresponding preset deviation threshold, the puncture path correction mechanism will be immediately triggered. After the correction mechanism is started, a special compensation amount calculation function will be called, and the lateral offset and longitudinal depth deviation will be passed as input parameters to the compensation amount calculation function to output the specific compensation amount of the puncture needle in the lateral and depth directions. The expression of the compensation amount calculation function is:

[0085] ;

[0086] Where, Indicates the lateral offset compensation amount, Indicates the depth compensation amount, Indicates the lateral offset, Indicates the depth deviation, represents the elastic modulus of the puncture needle, represents the moment of inertia of the needle section, Indicates the length of the current unbent needle body. represents the initial inclination angle of the pre-planned puncture navigation path, Indicates the needle tip angular deviation, Indicates the curvature radius of the needle body;

[0087] In addition, the bending shape adjustment parameters of the puncture needle will be determined based on the needle tip angular deviation (specifically, a matching search is performed in the preset bending shape parameter database to obtain the bending shape adjustment parameters that match the current needle tip angular deviation). Finally, a correction path containing the needle body bending compensation amount is generated to ensure that the puncture needle can move accurately along the corrected path, thereby improving the accuracy and safety of the entire puncture operation.

[0088] The path evaluation module is used to collect the puncture force feedback information of the puncture needle in real time during the needle movement process, and evaluate the effectiveness of the implementation of the puncture navigation path based on the puncture force feedback information;

[0089] When the path assessment module is executed, it will monitor and collect the puncture force feedback information of the puncture needle during the needle movement in real time, and evaluate the effectiveness of the puncture navigation path based on the puncture force feedback information to ensure that the puncture operation is within a controllable range. Among them, the path adjustment module includes a risk assessment unit, which is used to obtain the axial force, lateral force and torque of the puncture needle during the needle movement in real time and output the change trend of the axial force, lateral force and torque in real time;

[0090] Compare the changing trends of the axial force, lateral force, and torque with the corresponding preset safety thresholds. If any of the changing trends of the axial force, lateral force, or torque exceeds the corresponding preset safety threshold, it is directly determined that the currently executed puncture navigation path is risky and the robot stops puncture.

[0091] On the contrary, the changing trends of axial force, lateral force and torque are fused and the fusion evaluation index is output synchronously;

[0092] Compare the fusion evaluation index with the preset comprehensive evaluation interval. When the fusion evaluation index is within the comprehensive evaluation interval, the currently executed puncture navigation path is determined to be valid, and the puncture surgery is continued along the currently executed puncture navigation path. Otherwise, it is determined that there is a risk in the currently executed puncture navigation path, and the robot puncture is stopped.

[0093] Specifically, when evaluating the effectiveness of the puncture navigation path based on the puncture force feedback information, it is necessary to obtain the specific values ​​of the axial force, lateral force and torque exerted on the puncture needle during the needle movement in real time, and simultaneously output the magnitude of the axial force, lateral force and torque and their changing trends in the form of graphs or data in real time, so that medical staff can monitor the force changes during the puncture process at any time. At the same time, the changing trends of the axial force, lateral force and torque obtained in real time will be compared with the preset safety thresholds. If it is found that any value of the axial force, lateral force or torque changing trend exceeds its corresponding preset safety threshold, it will be immediately determined that the currently executed puncture navigation path is risky. At this time, the puncture operation of the robot will be stopped, and an alarm signal will be issued through the sound and light alarm system to remind medical staff to take timely countermeasures. On the contrary, if the changing trends of the axial force, lateral force and torque are all within the preset safety threshold Within the range, the corresponding comprehensive fusion processing will be performed to generate a comprehensive fusion evaluation index, and it will be outputted synchronously as a fusion evaluation index (specifically, different weight coefficients can be assigned to axial force, lateral force and torque, and the product of each force and its corresponding weight coefficient can be summed to obtain a fusion evaluation index) for further analysis. Finally, the generated fusion evaluation index will be compared with the preset comprehensive evaluation interval. When the fusion evaluation index falls within the comprehensive evaluation interval, the currently executed puncture navigation path is determined to be a valid path, and the puncture surgery can continue along the currently executed puncture navigation path to ensure the safety and accuracy of the surgery. On the contrary, if the fusion evaluation index exceeds the preset comprehensive evaluation interval, it is determined that the currently executed puncture navigation path is risky, the robot's puncture operation is stopped immediately, and a warning signal is issued through the alarm system to ensure the safety of the patient and the success of the surgery.

[0094] A path adjustment module is used to control the robot to perform the puncture surgery along the puncture navigation path when the puncture navigation path is implemented effectively, and vice versa, dynamically adjust the puncture navigation path according to the real-time navigation image;

[0095] When the path adjustment module is executed, if the evaluation result shows that the puncture navigation path is effectively implemented, the robot is controlled to steadily perform the puncture surgery along the pre-planned puncture navigation path to ensure the smooth progress of the surgery. Conversely, if the evaluation result shows that there is a deviation or risk in the path, the puncture navigation path is immediately and dynamically adjusted according to the real-time navigation image to ensure the safety and success rate of the surgery. The path adjustment module includes a path adjustment unit. When the currently executed puncture navigation path has a risk, the path adjustment unit immediately performs dynamic adjustment of the puncture navigation path.

[0096] Re-acquire ultrasound images to detect displacement and deformation of breast tissue caused by puncture, and locally register them with MRI and CT images to update the fused image data of the unpunctured area;

[0097] In the updated fused image data, the obstacle avoidance area is delineated with the current needle travel point as the center point. The obstacle avoidance area is circular, and the radius of the obstacle avoidance area is twice the distance from the needle tip to the center point when the change trend of the axial force, lateral force, and torque changes suddenly.

[0098] Based on the updated fusion image data and obstacle avoidance area, a new puncture path is replanned that bypasses the obstacle avoidance area and reaches the lesion, and subsequent puncture operations are performed according to the new puncture path;

[0099] Specifically, when the executed puncture navigation path is assessed to have risks, the dynamic adjustment mechanism of the puncture navigation path will be immediately activated to ensure the safety and accuracy of the entire puncture process. In order to accurately grasp the displacement and deformation of breast tissue caused by the puncture operation, the latest ultrasound image data will be re-collected and locally aligned with the previously acquired MRI and CT images, and then the fused image data of the unpunctured area will be updated to ensure the real-time and accuracy of the image data. Then, based on the updated fused image data, the current needle point of the puncture needle will be used as the center point to delineate an obstacle avoidance area. This obstacle avoidance area is preferably circular, but of course it can also be other shapes, which are specifically set according to actual needs. This embodiment takes a circular obstacle avoidance area as an example, and the radius length of the obstacle avoidance area is determined based on the axial force and lateral force. And when the torque change trend suddenly occurs, it is twice the actual distance from the tip of the puncture needle to the center point, so as to effectively avoid potential obstacle areas, improve the safety of puncture, and effectively avoid the risk of tissue damage caused by repeated adjustments to the puncture needle path. Finally, based on the updated fusion image data and the designated obstacle avoidance area, a new puncture path that can bypass the obstacle avoidance area and accurately reach the lesion will be re-planned. There may also be multiple new puncture paths. When there are multiple new puncture paths, refer to the above-mentioned initial puncture path screening process to select the optimal new puncture path as the execution path for subsequent puncture operations. After determining the new puncture path, the control system will again accurately control the robot to continue the puncture operation along the newly planned puncture navigation path to ensure that the entire puncture process is both safe and efficient.

[0100] The sound and light warning module is used to issue a sound and light warning signal when the puncture operation cannot be continued. The sound and light warning signal includes a sound signal and a light signal. The sound signal is a beeping sound of a preset frequency, and the light signal is a flashing light of a preset color. The sound and light warning signal is issued through the sound and light alarm installed in the operating room.

[0101] See also Figure 2 , a breast puncture surgery robot navigation device based on multimodal image fusion, the navigation device includes:

[0102] at least one processor;

[0103] and a memory communicatively coupled to the at least one processor;

[0104] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the above-mentioned breast puncture surgery robot navigation system based on multimodal image fusion.

[0105] The processor of the above-mentioned navigation device can be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or other hardware with data processing capabilities and / or instruction execution capabilities. The memory can include random access memory (RAM), and can also include read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disk, or read-only compact disk (CD-ROM), etc., for storing computer programs and data. The navigation device can also include an arithmetic unit and input and output devices used therewith. The arithmetic unit can be an arithmetic logic unit (ALU) or other components capable of performing mathematical and logical operations. The input device can include a keyboard, a mouse, a touch screen, etc., for receiving user input instructions. The output device can include a display, a printer, etc., for displaying processing results or printing relevant information.

[0106] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0107] The foregoing is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained herein shall, unless otherwise specified or limited, be implemented in accordance with conventional means in the art.

Claims

1. A robotic navigation system for breast biopsy surgery based on multimodal image fusion, characterized by: include: An image acquisition module is used to obtain multimodal image data of the patient's breast, wherein the multimodal image data includes ultrasound images, MRI images, and CT images; The path pre-planning module is used to perform registration and fusion processing on multimodal image data to generate fused image data, and then pre-plan the puncture navigation path based on the fused image data; Position registration module, used to collect the position information of the puncture needle in real time, and superimpose the position information of the puncture needle on the fused image data to generate a real-time navigation image; The path evaluation module is used to collect the puncture force feedback information of the puncture needle in real time during the needle movement process, and evaluate the effectiveness of the implementation of the puncture navigation path based on the puncture force feedback information; A path adjustment module is used to control the robot to perform the puncture surgery along the puncture navigation path when the puncture navigation path is implemented effectively, and vice versa, dynamically adjust the puncture navigation path according to the real-time navigation image; An audible and visual warning module is used to issue an audible and visual warning signal when the puncture operation cannot be continued; The path assessment module includes a risk assessment unit, which is used to obtain the axial force, lateral force and torque of the puncture needle in real time during the needle movement, and output the change trend of the axial force, lateral force and torque in real time; The path adjustment module includes a path adjustment unit, which immediately performs dynamic adjustment of the puncture navigation path when there is a risk in the currently executed puncture navigation path; The path pre-planning module further includes a pre-planning unit, which is used to plan an initial puncture path that avoids blood vessels and reaches the lesion based on the blood vessel distribution and the three-dimensional boundary of the lesion in the navigation reference map, wherein multiple initial puncture paths are set; Each initial puncture path is segmented to obtain multiple needle segments, and the puncture angle corresponding to each needle segment and the distance between the puncture needle and the surrounding tissue are collected; Normalizing the puncture angle and the distance between the puncture needle and the surrounding tissue to obtain a first evaluation index corresponding to the puncture angle and a second evaluation index corresponding to the distance between the puncture needle and the surrounding tissue; Comprehensively evaluate the first evaluation index and the second evaluation index under each needle movement segment, and simultaneously output the comprehensive evaluation score of each initial puncture path; Sort the comprehensive evaluation scores from high to low, and select the initial puncture path with the highest comprehensive evaluation score as the pre-planned puncture navigation path; Among them, if there are multiple initial puncture paths with the highest comprehensive evaluation scores and the same score, the initial puncture path closest to the lesion center is selected as the pre-planned puncture navigation path.

2. The robotic navigation system for breast puncture surgery based on multimodal image fusion according to claim 1, characterized in that: The image acquisition module uses non-invasive image acquisition technology to perform image acquisition, wherein ultrasound images are used to provide high-resolution structural information of breast tissue, MRI images are used to display soft tissue details inside the breast, and CT images are used to display calcification information inside the breast.

3. The robotic navigation system for breast puncture surgery based on multimodal image fusion according to claim 1, characterized in that: The path pre-planning module includes a pre-processing unit and a fusion unit. The pre-processing unit is used to pre-process the ultrasound image, the MRI image and the CT image respectively. The pre-processing includes denoising, contrast enhancement and geometric distortion correction. The fusion unit is used to perform grayscale normalization on the preprocessed ultrasound images, MRI images and CT images to eliminate the grayscale differences among the ultrasound images, MRI images and CT images; The pre-processed ultrasound images, MRI images, and CT images are preliminarily aligned using a three-dimensional spatial coordinate system, and then fine-tuned and calibrated for local tissue deformation. The elastic characteristics of ultrasound and the soft tissue characteristics of MRI are layered and fused, and the calcification characteristics of CT and the high-resolution structural information of breast tissue are superimposed and fused to generate fused image data; The fused image data is reconstructed in three dimensions to generate a navigation reference map that includes the shape of blood vessels, the three-dimensional boundaries of lesions, and safe passage annotations.

4. The robotic navigation system for breast puncture surgery based on multimodal image fusion according to claim 1, characterized in that: The position registration module includes a tracking unit and a correction unit; The tracking unit tracks the position of the puncture needle in real time through the optical positioner installed on the robot arm, and synchronously outputs the 3D spatial coordinates and attitude angle of the puncture needle; The three-dimensional spatial coordinates and posture angles of the puncture needle are spatially registered with the fused image data, and the position information of the puncture needle is superimposed on the fused image data to form a real-time navigation image; The correction unit is used to compare the real-time needle path under the real-time navigation image with the pre-planned needle path, output needle deviation information, and generate a correction path including a needle bending compensation amount when the needle deviation information exceeds a preset deviation threshold; The bending compensation amount is proportionally distributed to each segment of the puncture needle, and the puncture needle is controlled to puncture along the corrected path until the puncture needle reaches the target position.

5. The robotic navigation system for breast puncture surgery based on multimodal image fusion according to claim 4, characterized in that: The step of generating a correction path including a needle bending compensation amount includes: The three-dimensional coordinates and posture angles of the puncture needle are acquired in real time, and the spatial vector is compared with the reference coordinates under the pre-planned needle path, and the needle deviation information is output. The needle deviation information includes the lateral offset, longitudinal depth deviation and needle tip direction angle deviation; Compare the lateral offset, longitudinal depth deviation, and needle tip angular deviation with corresponding preset deviation thresholds, respectively. When any one of the lateral offset, longitudinal depth deviation, and needle tip angular deviation exceeds the corresponding preset deviation threshold, trigger a correction of the puncture path. Obtain a compensation amount calculation function, input the lateral offset and the longitudinal depth deviation into the compensation amount calculation function, and output the lateral offset compensation amount and the depth direction compensation amount of the puncture needle; According to the needle tip direction angle deviation, the bending shape adjustment parameters of the puncture needle are determined, and a correction path including the needle body bending compensation amount is generated.

6. The robotic navigation system for breast puncture surgery based on multimodal image fusion according to claim 1, characterized in that: When the risk assessment unit is executed, the change trends of the axial force, lateral force and torque are compared with the corresponding preset safety thresholds. If any of the axial force, lateral force or torque change trends exceeds the corresponding preset safety threshold, it is directly determined that the currently executed puncture navigation path is risky and the robot's puncture is stopped; On the contrary, the changing trends of axial force, lateral force and torque are fused and the fusion evaluation index is output synchronously; The fusion evaluation index is compared with the preset comprehensive evaluation interval. When the fusion evaluation index is within the comprehensive evaluation interval, the currently executed puncture navigation path is determined to be valid, and the puncture operation is continued along the currently executed puncture navigation path. Otherwise, the currently executed puncture navigation path is determined to be risky, and the robot's puncture is stopped.

7. The robotic navigation system for breast puncture surgery based on multimodal image fusion according to claim 6, characterized in that: When the path adjustment unit is executed, ultrasound images are re-collected to detect the displacement and deformation of breast tissue caused by puncture, and local registration is performed with MRI and CT images to update the fused image data of the non-punctured area; In the updated fused image data, the obstacle avoidance area is delineated with the current needle travel point as the center point. The obstacle avoidance area is circular, and the radius of the obstacle avoidance area is twice the distance from the needle tip to the center point when the change trend of the axial force, lateral force, and torque changes suddenly. Based on the updated fused image data and the obstacle avoidance area, a new puncture path that bypasses the obstacle avoidance area and reaches the lesion is replanned, and subsequent puncture operations are performed according to the new puncture path.

8. The robotic navigation system for breast puncture surgery based on multimodal image fusion according to claim 1, characterized in that: The sound and light warning signal includes a sound signal and a light signal. The sound signal is a buzzing sound of a preset frequency, and the light signal is a flashing light of a preset color. The sound and light warning signal is issued by a sound and light alarm installed in the operating room.

9. A robotic navigation device for breast puncture surgery based on multimodal image fusion, characterized by: The navigation device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; In which, the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the breast puncture surgery robot navigation system based on multimodal image fusion as described in any one of claims 1 to 8.

Images