Breast puncture surgery robot navigation system and device based on multi-modal image fusion

Abstract

The invention belongs to the technical field of medical robots, and particularly relates to a mammary gland puncture surgical robot navigation system and device based on multi-modal image fusion. The multi-mode image data are registered and fused, high-precision three-dimensional reconstruction of the mammary gland tissue is achieved, more visual and accurate surgical navigation information is provided for doctors, the position of the puncture needle is tracked in real time, space registration is conducted on the puncture needle and the image data, and therefore the precision and safety of the surgery are improved, and the operation efficiency is improved. Meanwhile, puncture force feedback information of a puncture needle in the needle moving process is collected in real time, the implementation effectiveness of a puncture navigation path is evaluated, when it is found that the puncture path has risks, dynamic adjustment is conducted immediately, a new puncture path bypassing an obstacle area and reaching a focus is planned again, and the risk of the puncture path is avoided. And if the safe path cannot be obtained, the system automatically stops needle insertion and gives an acousto-optic early warning prompt, so that the surgical risk caused by manual misoperation in the traditional surgery is avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical robots, and particularly relates to a breast puncture surgery robot navigation system and device based on multimodal image fusion. Background Art

[0002] Traditional breast puncture surgeries have problems such as inaccurate positioning and complex operations, which bring great troubles to doctors and patients. However, with the continuous progress of medical imaging technology, more accurate and safe navigation means are provided for breast puncture surgeries. Based on this technology, the surgical efficiency and success rate of breast puncture have been significantly improved.

[0003] In the prior art, a single medical imaging technology is mostly used for navigation, such as ultrasonic imaging, MRI or CT, etc. However, each imaging technology has its limitations. For example, ultrasonic imaging is easily affected by tissue density and gas interference. Although MRI can provide high-resolution soft tissue images, the equipment is expensive and the operation is complex. Multiple CT scans have the risk of excessive radiation. In addition, during the puncture process, the lack of force feedback information to evaluate the interaction state between the puncture needle and the tissue in real time will undoubtedly increase the uncertainty during the surgery. Based on this, the present invention proposes a breast puncture surgery robot navigation system based on multimodal image fusion to solve the above problems. Summary of the Invention

[0004] The purpose of the present invention is to provide a breast puncture surgery robot navigation system and device based on multimodal image fusion, which can fuse ultrasonic images, MRI images and CT images, and evaluate the effectiveness of the puncture navigation path in real time according to the puncture force feedback information, and dynamically adjust the puncture path to improve the accuracy and safety of the surgery.

[0005] The technical solutions adopted by the present invention are specifically as follows: A breast puncture surgery robot navigation system based on multimodal image fusion, comprising: An image acquisition module, configured to acquire multimodal image data of a patient's breast, wherein the multimodal image data includes ultrasonic images, MRI images and CT images; A path pre-planning module, configured to perform registration and fusion processing on the multimodal image data to generate fused image data, and then pre-plan a puncture navigation path based on the fused image data; A position registration module, configured to collect the position information of the puncture needle in real time, and superimpose the position information of the puncture needle on the fused image data to generate a real-time navigation image; A path evaluation module, configured to collect the puncture force feedback information of the puncture needle during the needle insertion process in real time, and evaluate the implementation effectiveness of the puncture navigation path according to the puncture force feedback information; A path adjustment module is used to control the robot to perform the puncture surgery along the puncture navigation path when the puncture navigation path is implemented effectively, and vice versa, dynamically adjust the puncture navigation path according to the real-time navigation image; The sound and light warning module is used to send out sound and light warning signals when the puncture operation cannot be continued.

[0006] In a preferred embodiment, the image acquisition module uses non-invasive image acquisition technology to perform image acquisition, wherein ultrasound images are used to provide high-resolution structural information of breast tissue, MRI images are used to display soft tissue details inside the breast, and CT images are used to display calcification information inside the breast.

[0007] In a preferred embodiment, the path pre-planning module includes a pre-processing unit and a fusion unit, the pre-processing unit is used to pre-process the ultrasound image, the MRI image and the CT image respectively, and the pre-processing includes denoising, contrast enhancement and geometric distortion correction; The fusion unit is used to normalize the grayscale of the preprocessed ultrasound images, MRI images and CT images, and eliminate the grayscale differences among the ultrasound images, MRI images and CT images; The pre-processed ultrasound images, MRI images and CT images are preliminarily aligned through a three-dimensional spatial coordinate system, and then the local tissue deformation is fine-tuned and calibrated; The elastic characteristics of ultrasound and the soft tissue characteristics of MRI are layered and fused, and the calcification characteristics of CT and the high-resolution structural information of breast tissue are superimposed and fused to generate fused image data; The fused image data is reconstructed in three dimensions to generate a navigation reference map that includes the shape of blood vessels, three-dimensional boundaries of lesions, and safe channel annotations.

[0008] In a preferred embodiment, the path pre-planning module further includes a pre-planning unit, which is used to plan an initial puncture path that avoids blood vessels and reaches the lesion according to the blood vessel distribution and the three-dimensional boundary of the lesion in the navigation reference map, wherein a plurality of initial puncture paths are provided; Each initial puncture path is segmented to obtain multiple needle segments, and the puncture angle corresponding to each needle segment and the distance between the puncture needle and the surrounding tissue are collected; Normalizing the puncture angle and the distance between the puncture needle and the surrounding tissue to obtain a first evaluation index corresponding to the puncture angle and a second evaluation index corresponding to the distance between the puncture needle and the surrounding tissue; Comprehensively evaluate the first evaluation index and the second evaluation index under each needle movement segment, and synchronously output the comprehensive evaluation score of each initial puncture path; The comprehensive evaluation scores are sorted from high to low, and the initial puncture path with the highest comprehensive evaluation score is selected as the pre-planned puncture navigation path; Among them, if there are multiple initial puncture paths with the highest and same comprehensive evaluation scores, the initial puncture path closest to the center of the lesion is selected as the pre-planned puncture navigation path.

[0009] In a preferred embodiment, the position registration module includes a tracking unit and a correction unit; The tracking unit real-time tracks the position of the puncture needle through an optical locator installed on the robotic arm, and synchronously outputs the three-dimensional spatial coordinates and attitude angles of the puncture needle; The three-dimensional spatial coordinates and attitude angles of the puncture needle are spatially registered with the fused image data, and the position information of the puncture needle is superimposed on the fused image data to form a real-time navigation image; The correction unit is used to compare the real-time needle insertion path under the real-time navigation image with the pre-planned needle insertion path, output the needle insertion deviation information, and generate a correction path including the needle body bending compensation amount when the needle insertion deviation information exceeds the preset deviation threshold; The bending compensation amount is proportionally allocated to each segment of the puncture needle, and the puncture needle is controlled to perform puncture according to the correction path until the puncture needle reaches the target position.

[0010] In a preferred embodiment, the step of generating a correction path including the needle body bending compensation amount includes: Real-time obtain the three-dimensional spatial coordinates and attitude angles of the puncture needle, and perform a spatial vector comparison with the reference coordinates under the pre-planned needle insertion path, output the needle insertion deviation information, and the needle insertion deviation information includes the lateral offset, the longitudinal depth deviation, and the tip direction angle deviation; The lateral offset, the longitudinal depth deviation, and the tip direction angle deviation are respectively compared with the corresponding preset deviation thresholds. When any one of the lateral offset, the longitudinal depth deviation, and the tip direction angle deviation exceeds the corresponding preset deviation threshold, the correction of the puncture path is triggered; Obtain the compensation amount calculation function, input the lateral offset and the longitudinal depth deviation into the compensation amount calculation function, and output the lateral offset compensation amount and the depth direction compensation amount of the puncture needle; According to the tip direction angle deviation, determine the bending shape adjustment parameter of the puncture needle, and generate a correction path including the needle body bending compensation amount.

[0011] In a preferred embodiment, the path evaluation module includes a risk assessment unit, and the risk assessment unit is used to real-time obtain the axial force, the lateral force, and the torque during the needle insertion process of the puncture needle, and real-time output the change trends of the axial force, the lateral force, and the torque; Compare the changing trends of the axial force, lateral force, and torque with the corresponding preset safety thresholds. If any one of the changing trends of the axial force, lateral force, or torque exceeds the corresponding preset safety threshold, directly determine that there is a risk in the currently executed puncture navigation path and stop the puncture of the robot. Conversely, fuse the changing trends of the axial force, lateral force, and torque, and synchronously output the fusion evaluation index. Compare the fusion evaluation index with the preset comprehensive evaluation interval. When the fusion evaluation index is within the comprehensive evaluation interval, determine that the currently executed puncture navigation path is effective and continue to perform the puncture surgery along the currently executed puncture navigation path. Otherwise, determine that there is a risk in the currently executed puncture navigation path and stop the puncture of the robot.

[0012] In a preferred solution, the path adjustment module includes a path adjustment unit. When there is a risk in the currently executed puncture navigation path, the path adjustment unit immediately performs dynamic adjustment of the puncture navigation path. Re-collect ultrasound images to detect the displacement and deformation of breast tissue caused by puncture, and perform local registration with MRI and CT images to update the fused image data of the unpunctured area. In the updated fused image data, with the current needle insertion point of the puncture needle as the center point, delimit an obstacle avoidance area. The obstacle avoidance area is circular, and the radius of the obstacle avoidance area is twice the distance from the tip of the puncture needle to the center point when the changing trends of the axial force, lateral force, and torque change abruptly. According to the updated fused image data and the obstacle avoidance area, re-plan a new puncture path that bypasses the obstacle avoidance area and reaches the lesion, and perform subsequent puncture operations according to the new puncture path.

[0013] In a preferred solution, the acoustic and optical warning signal includes an acoustic signal and an optical signal. The acoustic signal is a beeping sound of a preset frequency, and the optical signal is a flashing light of a preset color. The acoustic and optical warning signal is emitted by an acoustic and optical alarm installed in the operating room.

[0014] The present invention also provides a navigation device for a breast puncture surgery robot based on multi-modal image fusion. The navigation device includes: At least one processor; And a memory communicatively connected to the at least one processor; Wherein, the memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor, so that the at least one processor can execute the above-mentioned navigation system for a breast puncture surgery robot based on multi-modal image fusion.

[0015] The technical effects achieved by the present invention are: By registering and fusing multi-modal imaging data, the present invention achieves high-precision three-dimensional reconstruction of breast tissue, providing doctors with more intuitive and accurate surgical navigation information. By real-time tracking the position of the puncture needle and performing spatial registration with the fused imaging data, doctors can clearly see the specific position of the puncture needle in the breast tissue, thereby improving the precision and safety of the surgery. At the same time, the present invention also evaluates the effectiveness of the puncture navigation path by real-time collecting the feedback information of the puncture force during the needle insertion process. When it is found that there is a risk in the puncture path, it will immediately perform dynamic adjustment and re-plan a new puncture path that bypasses the obstacle area and reaches the lesion. Description of the Drawings

[0016] Figure 1 is a schematic diagram of the system modules of the present invention; Figure 2 is a schematic diagram of the structure of the navigation device of the present invention. Detailed Embodiments

[0017] To make the above objects, features, and advantages of the present invention more obvious and understandable, the following detailed description of the specific embodiments of the present invention will be given in conjunction with the accompanying drawings of the specification.

[0018] In the following description, many specific details are set forth in order to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar generalizations without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0019] Secondly, the so-called "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that can be included in at least one implementation manner of the present invention. The "in a preferred embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that excludes other embodiments.

[0020] Please refer to Figure 1 As shown, the present invention provides a breast puncture surgical robot navigation system based on multi-modal image fusion, including: An image acquisition module for acquiring multi-modal imaging data of a patient's breast, wherein the multi-modal imaging data includes ultrasonic images, MRI images, and CT images; When a breast puncture operation needs to be performed, first execute the image acquisition module to comprehensively obtain multi-modal image data of the patient's breast. The multi-modal image data covers a variety of imaging modalities, including ultrasound images, MRI images, and CT images, to ensure obtaining comprehensive information about breast tissue. Among them, the image acquisition module uses non-invasive image acquisition technology to perform image acquisition. Among them, ultrasound images are used to provide high-resolution structural information of breast tissue. Through the reflection and refraction of ultrasonic waves, the fine structure of breast tissue can be clearly presented. MRI images are used to show the soft tissue details inside the breast. Using a strong magnetic field and radiofrequency pulses, MRI can generate high-contrast images to detail the soft tissue distribution and lesion conditions inside the breast. CT images are used to show the calcification information inside the breast. Through the penetration of X-rays and computer reconstruction, CT can accurately identify and locate calcification foci in the breast.

[0021] The path pre-planning module is used to register and fuse the multi-modal image data to generate fused image data, and then pre-plan the puncture navigation path based on the fused image data; When the path pre-planning module is executed, it will perform corresponding registration and fusion processing on the obtained multi-modal image data to output fused image data that can reflect breast tissue. And on this basis, it will also pre-plan the puncture navigation path for subsequent puncture to ensure the safety of the subsequent puncture process. Among them, the path pre-planning module includes a pre-processing unit and a fusion unit. The pre-processing unit is used to perform pre-processing on ultrasound images, MRI images, and CT images respectively. The pre-processing includes denoising, enhancing contrast, and correcting geometric distortion; The fusion unit is used to perform gray-scale normalization on the pre-processed ultrasound images, MRI images, and CT images to eliminate the gray-scale differences between the ultrasound images, MRI images, and CT images; The pre-processed ultrasound images, MRI images, and CT images are initially aligned through a three-dimensional space coordinate system, and then the local tissue deformation is finely calibrated; The elastic characteristics of ultrasound are layered and fused with the soft tissue characteristics of MRI, and the calcification characteristics of CT are superimposed and fused with the high-resolution structural information of breast tissue to generate fused image data; Perform three-dimensional reconstruction on the fused image data to generate a navigation reference map including blood vessel trajectories, three-dimensional boundaries of lesions, and safety channel markings; Specifically, when registering and fusing multi-modal image data, preprocessing operations are first performed on the image data of each type separately. For ultrasound images, due to the easy interference of human tissues, artifacts will occur. Therefore, geometric distortion correction needs to be performed on them to ensure the accuracy of the images. For MRI images and CT images, although the imaging effects are relatively clear, denoising processing still needs to be performed on them to further improve the image quality. After the preprocessing operations are completed, in order to enable different types of image data to be compared and analyzed in the same coordinate system, gray normalization processing also needs to be performed on them to eliminate the gray differences existing between different images, so that the gray values of various images reach consistency, laying a foundation for subsequent fusion processing. Then, using a three-dimensional space coordinate system, the preprocessed ultrasound images, MRI images, and CT images are initially aligned. On this basis, fine-tuning and calibration are also performed for the deformation of local tissues to ensure a high degree of consistency in the spatial positions of various images. After that, the elastic features in the ultrasound images are layered and fused with the soft tissue features in the MRI images to make full use of the advantageous information of both. At the same time, the calcification features in the CT images are superimposed and fused with the high-resolution structural information of the breast tissue to further enhance the details of the fused images. Finally, fused image data integrating various image information is generated. Finally, three-dimensional reconstruction processing is performed on the generated fused image data, aiming to construct a three-dimensional image model containing rich information, which not only includes the walking paths of blood vessels, but also marks the three-dimensional boundaries of the lesions and the positions of the safe channels, and finally generates the corresponding navigation reference map, providing a reliable navigation basis for subsequent puncture surgeries.

[0022] Secondly, the path pre-planning module also includes a pre-planning unit. The pre-planning unit is used to plan an initial puncture path that avoids blood vessels and reaches the lesion according to the blood vessel distribution and the three-dimensional boundaries of the lesion in the navigation reference map. Among them, multiple initial puncture paths are set. Each initial puncture path is segmented to obtain multiple needle insertion segments, and the puncture angles corresponding to each needle insertion segment and the distances between the puncture needle and the surrounding tissues are collected. The puncture angles and the distances between the puncture needle and the surrounding tissues are normalized to obtain a first evaluation index corresponding to the puncture angle and a second evaluation index corresponding to the distance between the puncture needle and the surrounding tissues. The first evaluation index and the second evaluation index under each needle insertion segment are comprehensively evaluated, and the comprehensive evaluation scores of each initial puncture path are synchronously output. The comprehensive evaluation scores are sorted from high to low, and the initial puncture path with the highest comprehensive evaluation score is selected as the pre-planned puncture navigation path. Among them, if there are multiple initial puncture paths with the highest and equal comprehensive evaluation scores, the initial puncture path closest to the center of the lesion is selected as the pre-planned puncture navigation path; In this embodiment, when pre-planning the puncture navigation path, first, based on the blood vessel distribution and the three-dimensional boundary information of the lesion presented in the navigation reference map, multiple initial puncture paths that can effectively avoid blood vessels and accurately reach the lesion are generated. Then, each initial puncture path is segmented into multiple needle insertion segments, and the puncture angle data corresponding to each needle insertion segment and the distance between the puncture needle and the surrounding tissues are collected one by one. Moreover, the collected puncture angle and the distance data between the puncture needle and the surrounding tissues are normalized to obtain the first evaluation index corresponding to the puncture angle and the second evaluation index corresponding to the distance between the puncture needle and the surrounding tissues. Then, a comprehensive evaluation of the first evaluation index and the second evaluation index for each needle insertion segment is carried out, and the comprehensive evaluation score of each initial puncture path is output synchronously during the evaluation process. Specifically, through the fusion weighted algorithm, the first evaluation index and the second evaluation index are weighted and summed to calculate the final comprehensive evaluation score. The higher the comprehensive evaluation score, the higher the feasibility of the corresponding initial puncture path. Then, the comprehensive evaluation scores are arranged in descending order, and the initial puncture path with the highest comprehensive evaluation score is selected as the pre-planned puncture navigation path. It should be noted that if it is found during the sorting process that there are multiple initial puncture paths with the highest and equal comprehensive evaluation scores, the initial puncture path closest to the center of the lesion should be further selected as the final pre-planned puncture navigation path to ensure the accuracy of the puncture operation.

[0023] The position registration module is used to collect the position information of the puncture needle in real time and superimpose the position information of the puncture needle on the fused image data to generate a real-time navigation image; When the position registration module is executed, it will collect the position information of the puncture needle in real time and superimpose the position information of the puncture needle on the fused image data in real time, thereby generating a dynamic real-time navigation image to provide intuitive visual guidance for the doctor. Among them, the position registration module includes a tracking unit and a correction unit; The tracking unit real-time tracks the position of the puncture needle through an optical locator installed on the robotic arm and synchronously outputs the three-dimensional space coordinates and attitude angles of the puncture needle; The three-dimensional space coordinates and attitude angles of the puncture needle are spatially registered with the fused image data, and the position information of the puncture needle is superimposed into the fused image data to form a real-time navigation image; The correction unit is used to compare the real-time needle insertion path under the real-time navigation image with the pre-planned needle insertion path, output the needle insertion deviation information, and generate a correction path including the needle body bending compensation amount when the needle insertion deviation information exceeds the preset deviation threshold; Distribute the bending compensation amount proportionally to each segment of the puncture needle, and control the puncture needle to puncture along the corrected path until the puncture needle reaches the target position; Specifically, when generating the real-time navigation image, first, use the high-precision optical locator installed on the robotic arm to track the specific position of the puncture needle in real time, and synchronously output the coordinates of the puncture needle in the three-dimensional space and its attitude angle information to ensure the real-time and accuracy of the puncture needle position. Then, perform corresponding spatial registration on the three-dimensional space coordinates and attitude angle of the puncture needle with the fused image data, so as to seamlessly superimpose the position information of the puncture needle onto the fused image data to form a real-time updated real-time navigation image, thereby being able to intuitively display the real-time position and direction of the puncture needle in the breast. Then, compare the actual needle insertion path of the puncture needle under the real-time navigation image with the pre-planned needle insertion path, and output the needle insertion deviation information in real time. When the needle insertion deviation information exceeds the preset deviation threshold, a corrected path including the bending compensation amount of the needle body will be automatically generated to correct the deviation of the puncture needle and ensure the accuracy of the puncture process. Finally, rationally distribute the calculated bending compensation amount proportionally to each segment of the puncture needle ( , where represents the compensation amount allocated to the th segment, represents the length of the th segment, represents the attenuation coefficient, which is used to reduce the cumulative error at the end, represents the total number of segments of the puncture needle, and the puncture needle is segmented specifically according to the stiffness of the needle body), and precisely control the puncture needle to perform the puncture operation along the corrected path through the robot control system until the puncture needle successfully reaches the predetermined target position.

[0024] Secondly, the steps of generating a corrected path including the bending compensation amount of the needle body include: Obtain the three-dimensional space coordinates and attitude angle of the puncture needle in real time, and perform a spatial vector comparison with the reference coordinates under the pre-planned needle insertion path, and output the needle insertion deviation information, where the needle insertion deviation information includes the lateral offset amount, the longitudinal depth deviation amount, and the tip direction angle deviation amount; Compare the lateral offset amount, the longitudinal depth deviation amount, and the tip direction angle deviation amount with the corresponding preset deviation thresholds respectively. When any one of the lateral offset amount, the longitudinal depth deviation amount, and the tip direction angle deviation amount exceeds the corresponding preset deviation threshold, trigger the correction of the puncture path; Obtain the compensation amount calculation function, input the lateral offset amount and the longitudinal depth deviation amount into the compensation amount calculation function, and output the lateral offset compensation amount and the depth direction compensation amount of the puncture needle; Determine the bending shape adjustment parameters of the puncture needle according to the tip direction angle deviation amount, and generate a corrected path including the needle body bending compensation amount; In this embodiment, under the guidance of the real-time navigation image, the actual traveling trajectory of the puncture needle will be continuously tracked, and the real-time needle insertion path will be compared and analyzed with the pre-planned needle insertion path, so as to accurately output the needle insertion deviation information. The needle insertion deviation information covers multiple dimensions such as the lateral offset amount, the longitudinal depth deviation amount, and the tip direction angle deviation amount. At the same time, the lateral offset amount, the longitudinal depth deviation amount, and the tip direction angle deviation amount will be respectively compared with their corresponding preset deviation thresholds. Once it is found that any one of the lateral offset amount, the longitudinal depth deviation amount, or the tip direction angle deviation amount exceeds its corresponding preset deviation threshold, the correction mechanism of the puncture path will be immediately triggered. After the correction mechanism is started, a special compensation amount calculation function will be called, and the lateral offset amount and the longitudinal depth deviation amount will be used as input parameters and passed to the compensation amount calculation function, so as to output the specific compensation amounts of the puncture needle in the lateral and depth directions. Among them, the expression of the compensation amount calculation function is: ; In the formula, represents the lateral offset compensation amount, represents the depth direction compensation amount, represents the lateral offset amount, represents the depth deviation amount, represents the elastic modulus of the puncture needle, represents the moment of inertia of the needle body cross-section, represents the length of the current unbent section of the needle body, represents the initial inclination angle of the pre-planned puncture navigation path, represents the tip direction angle deviation, represents the bending curvature radius of the needle body; In addition, the bending shape adjustment parameters of the puncture needle will be determined according to the tip direction angle deviation amount (specifically, by matching and searching in the preset bending shape parameter database to obtain the bending shape adjustment parameters matching the current tip direction angle deviation amount), and finally a corrected path including the needle body bending compensation amount will be generated to ensure that the puncture needle can accurately travel according to the corrected path, thereby improving the accuracy and safety of the entire puncture operation.

[0025] A path evaluation module for real-time collecting the puncture force feedback information during the needle insertion process of the puncture needle and evaluating the implementation effectiveness of the puncture navigation path according to the puncture force feedback information; When the path evaluation module is executed, it will monitor and collect the feedback information of the puncture force received by the puncture needle in real time during the needle insertion process, and evaluate the implementation effectiveness of the puncture navigation path based on the puncture force feedback information to ensure that the puncture operation is within a controllable range. Among them, the path adjustment module includes a risk assessment unit, and the risk assessment unit is used to obtain the axial force, lateral force and torque of the puncture needle in real time during the needle insertion process, and output the change trends of the axial force, lateral force and torque in real time; Compare the change trends of the axial force, lateral force and torque with the corresponding preset safety thresholds. If any one of the change trends of the axial force, lateral force or torque exceeds the corresponding preset safety threshold, directly determine that the currently executed puncture navigation path is risky and stop the puncture of the robot; Otherwise, fuse the change trends of the axial force, lateral force and torque, and synchronously output the fusion evaluation index; Compare the fusion evaluation index with the preset comprehensive evaluation interval. When the fusion evaluation index is within the comprehensive evaluation interval, determine that the currently executed puncture navigation path is effective and continue to perform the puncture surgery along the currently executed puncture navigation path. Otherwise, determine that the currently executed puncture navigation path is risky and stop the puncture of the robot; Specifically, when evaluating the effectiveness of the puncture navigation path based on the puncture force feedback information, it is necessary to obtain the specific values of the axial force, lateral force, and torque exerted on the puncture needle during the needle insertion process in real time, and synchronously output the magnitudes and change trends of the axial force, lateral force, and torque in the form of charts or data in real time, so as to facilitate medical staff to monitor the force changes during the puncture process at any time. At the same time, the change trends of these real-time obtained axial force, lateral force, and torque will be compared with the pre-set safety thresholds. If it is found that any one of the values in the change trends of the axial force, lateral force, or torque exceeds its corresponding pre-set safety threshold, it will be immediately determined that the currently executed puncture navigation path is at risk. At this time, the puncture operation of the robot will be stopped, and an alarm signal will be sent through the sound and light alarm system to remind the medical staff to take corresponding measures in time. On the contrary, if the change trends of the axial force, lateral force, and torque are all within the pre-set safety threshold range, corresponding comprehensive fusion processing will be performed to generate a comprehensive fusion evaluation index, which will be synchronously output as the fusion evaluation index (specifically, different weight coefficients can be assigned to the axial force, lateral force, and torque, and the sum of the products of each force and its corresponding weight coefficient can be obtained to get the fusion evaluation index) for further analysis. Finally, the generated fusion evaluation index will be compared with the pre-set comprehensive evaluation interval. When the fusion evaluation index falls within the comprehensive evaluation interval, it is determined that the currently executed puncture navigation path is an effective path, and the puncture operation can continue along the currently executed puncture navigation path to ensure the safety and accuracy of the operation. On the contrary, if the fusion evaluation index exceeds the pre-set comprehensive evaluation interval, it is determined that the currently executed puncture navigation path is at risk, and the puncture operation of the robot will be immediately stopped, and a warning signal will be sent through the alarm system to ensure the safety of the patient and the success of the operation.

[0026] The path adjustment module is used to control the robot to perform the puncture operation along the puncture navigation path when the puncture navigation path is effectively implemented. Otherwise, it will dynamically adjust the puncture navigation path according to the real-time navigation image. When the path adjustment module is executed, when the evaluation result shows that the puncture navigation path is effectively implemented, it controls the robot to steadily perform the puncture operation along the pre-planned puncture navigation path to ensure the smooth progress of the operation. Otherwise, if the evaluation result shows that the path has deviations or risks, it will immediately dynamically adjust the puncture navigation path according to the real-time navigation image to ensure the safety and success rate of the operation. Among them, the path adjustment module includes a path adjustment unit, and when the currently executed puncture navigation path is at risk, the path adjustment unit immediately performs the dynamic adjustment of the puncture navigation path. Re-collect the ultrasound image to detect the displacement and deformation of the breast tissue caused by the puncture, and perform local registration with the MRI and CT images to update the fusion image data of the unpunctured area. In the updated fused image data, with the current needle insertion point of the puncture needle as the center point, an obstacle avoidance area is delimited. Among them, the obstacle avoidance area is circular, and the radius of the obstacle avoidance area is twice the distance from the tip of the puncture needle to the center point when the change trends of the axial force, lateral force, and torque undergo sudden changes; According to the updated fused image data and the obstacle avoidance area, a new puncture path that bypasses the obstacle avoidance area and reaches the lesion is re-planned, and subsequent puncture operations are performed according to the new puncture path; Specifically, when the executed puncture navigation path is evaluated to have risks, the dynamic adjustment mechanism of the puncture navigation path will be immediately activated to ensure the safety and accuracy of the entire puncture process. In order to accurately grasp the displacement and deformation of breast tissue caused by puncture operations, the latest ultrasound image data will be re-acquired and locally registered with the previously obtained MRI and CT images, and then the fused image data of the unpunctured area will be updated to ensure the timeliness and accuracy of the image data. Then, based on the updated fused image data, the current needle insertion point of the puncture needle will be used as the center point to delimit an obstacle avoidance area. This obstacle avoidance area is preferably circular, but it can also be other shapes, which are specifically set according to actual needs. In this embodiment, the circular obstacle avoidance area is taken as an example. The radius length of the obstacle avoidance area is twice the actual distance from the tip of the puncture needle to the center point when the change trends of the axial force, lateral force, and torque undergo sudden changes. This can effectively avoid potential obstacle areas, improve the safety of puncture, and also effectively avoid the risk of tissue damage caused by continuously adjusting the puncture needle path multiple times. Finally, based on the updated fused image data and the delimited obstacle avoidance area, a new puncture path that can bypass the obstacle avoidance area and accurately reach the lesion will be re-planned. There may be multiple new puncture paths. When there are multiple new puncture paths, referring to the above screening process of the initial puncture path, the optimal new puncture path will be selected as the execution path for the subsequent puncture surgery. After determining the new puncture path, the control system will precisely control the robot again to make it continue to perform the puncture surgery along the newly planned puncture navigation path, ensuring that the entire puncture process is both safe and efficient.

[0027] An acoustic and optical warning module, which is used to emit an acoustic and optical warning signal when the puncture surgery cannot be continued. The acoustic and optical warning signal includes an acoustic signal and an optical signal. The acoustic signal is a beeping sound of a preset frequency, and the optical signal is a flashing light of a preset color. The acoustic and optical warning signal is emitted through an acoustic and optical alarm installed in the operating room.

[0028] Please refer to Figure 2 , a navigation device for a breast puncture surgery robot based on multi-modal image fusion. The navigation device includes: At least one processor; And a memory communicatively connected to at least one processor; Among them, the memory stores a computer program executable by at least one processor. The computer program is executed by at least one processor, enabling the at least one processor to execute the above-mentioned breast puncture surgical robot navigation system based on multimodal image fusion.

[0029] The processor of the above navigation device can be a hardware component with data processing capabilities and / or instruction execution capabilities such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), etc. The memory can include a random access memory (RAM), and can also include a read only memory (ROM), a non-volatile random access memory (NVRAM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read only memory (CD-ROM), etc., for storing computer programs and data. The navigation device can also include an arithmetic unit, as well as input devices and output devices used in conjunction with it. The arithmetic unit can be an arithmetic logic unit (ALU) or other components capable of performing mathematical and logical operations. The input devices can include a keyboard, a mouse, a touch screen, etc., for receiving input instructions from users. The output devices can include a display, a printer, etc., for presenting processing results or printing relevant information.

[0030] It should be noted that in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, device, article or method. Without further limitations, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, device, article or method including that element.

[0031] The above are only the preferred embodiments of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention. The structures, devices, and operation methods not specifically described and explained in the present invention are implemented according to the conventional means in the art without special instructions and limitations.

Claims

1. A robotic navigation system for breast puncture surgery based on multimodal image fusion, characterized in that: include: An image acquisition module is used to obtain multimodal image data of a patient's breast, wherein the multimodal image data includes ultrasound images, MRI images, and CT images; The path pre-planning module is used to perform registration and fusion processing on multi-modal image data, generate fused image data, and then pre-plan the puncture navigation path based on the fused image data; The position registration module is used to collect the position information of the puncture needle in real time, and superimpose the position information of the puncture needle on the fused image data to generate a real-time navigation image; The path evaluation module is used to collect the puncture force feedback information of the puncture needle in real time during the needle movement process, and evaluate the effectiveness of the implementation of the puncture navigation path based on the puncture force feedback information; A path adjustment module is used to control the robot to perform the puncture surgery along the puncture navigation path when the puncture navigation path is implemented effectively, and vice versa, dynamically adjust the puncture navigation path according to the real-time navigation image; An acoustic and visual warning module is used to send an acoustic and visual warning signal when the puncture operation cannot be continued; The path assessment module includes a risk assessment unit, which is used to obtain the axial force, lateral force and torque of the puncture needle in real time during the needle movement, and output the change trend of the axial force, lateral force and torque in real time; The path adjustment module includes a path adjustment unit, which immediately performs dynamic adjustment of the puncture navigation path when there is a risk in the currently executed puncture navigation path.

2. The robotic navigation system for breast puncture surgery based on multimodal image fusion according to claim 1, characterized in that: The image acquisition module uses non-invasive image acquisition technology to perform image acquisition, wherein ultrasound images are used to provide high-resolution structural information of breast tissue, MRI images are used to display soft tissue details inside the breast, and CT images are used to display calcification information inside the breast.

3. The robotic navigation system for breast puncture surgery based on multimodal image fusion according to claim 1, characterized in that: The path pre-planning module includes a pre-processing unit and a fusion unit, wherein the pre-processing unit is used to pre-process the ultrasound image, the MRI image and the CT image respectively, and the pre-processing includes denoising, contrast enhancement and geometric distortion correction; The fusion unit is used to normalize the grayscale of the preprocessed ultrasound images, MRI images and CT images, and eliminate the grayscale differences among the ultrasound images, MRI images and CT images; The pre-processed ultrasound images, MRI images and CT images are preliminarily aligned through a three-dimensional spatial coordinate system, and then the local tissue deformation is fine-tuned and calibrated; The elastic characteristics of ultrasound and the soft tissue characteristics of MRI are layered and fused, and the calcification characteristics of CT and the high-resolution structural information of breast tissue are superimposed and fused to generate fused image data; The fused image data is reconstructed in three dimensions to generate a navigation reference map that includes the shape of blood vessels, three-dimensional boundaries of lesions, and safe channel annotations.

4. The robotic navigation system for breast puncture surgery based on multimodal image fusion according to claim 1, characterized in that: The path pre-planning module further includes a pre-planning unit, which is used to plan an initial puncture path that avoids blood vessels and reaches the lesion according to the blood vessel distribution and the three-dimensional boundary of the lesion in the navigation reference map, wherein a plurality of initial puncture paths are provided; Each initial puncture path is segmented to obtain multiple needle segments, and the puncture angle corresponding to each needle segment and the distance between the puncture needle and the surrounding tissue are collected; Normalize the puncture angle and the distance between the puncture needle and the surrounding tissues to obtain a first evaluation index corresponding to the puncture angle and a second evaluation index corresponding to the distance between the puncture needle and the surrounding tissues; Comprehensively evaluate the first evaluation index and the second evaluation index under each needle insertion segment, and synchronously output the comprehensive evaluation scores of each initial puncture path; Sort the comprehensive evaluation scores from high to low, and select the initial puncture path with the highest comprehensive evaluation score as the pre-planned puncture navigation path; Among them, if there are multiple initial puncture paths with the same highest comprehensive evaluation score, select the initial puncture path closest to the center of the lesion as the pre-planned puncture navigation path.

5. The robotic navigation system for breast puncture surgery based on multimodal image fusion according to claim 1, characterized in that: The position registration module includes a tracking unit and a correction unit; The tracking unit real-time tracks the position of the puncture needle through an optical locator installed on the robotic arm, and synchronously outputs the three-dimensional spatial coordinates and attitude angles of the puncture needle; Perform spatial registration on the three-dimensional spatial coordinates and attitude angles of the puncture needle and the fused image data, and superimpose the position information of the puncture needle on the fused image data to form a real-time navigation image; The correction unit is used to compare the real-time needle insertion path under the real-time navigation image with the pre-planned needle insertion path, output the needle insertion deviation information, and generate a correction path including the needle body bending compensation amount when the needle insertion deviation information exceeds the preset deviation threshold; Allocate the bending compensation amount proportionally to each segment of the puncture needle, and control the puncture needle to puncture according to the correction path until the puncture needle reaches the target position.

6. The robotic navigation system for breast puncture surgery based on multimodal image fusion according to claim 5, characterized in that: The step of generating the correction path including the needle body bending compensation amount includes: Real-time obtain the three-dimensional spatial coordinates and attitude angles of the puncture needle, and perform a spatial vector comparison with the reference coordinates under the pre-planned needle insertion path, and output the needle insertion deviation information, where the needle insertion deviation information includes the lateral offset, the longitudinal depth deviation, and the tip direction angle deviation; Compare the lateral offset, the longitudinal depth deviation, and the tip direction angle deviation with the corresponding preset deviation thresholds respectively. When any one of the lateral offset, the longitudinal depth deviation, and the tip direction angle deviation exceeds the corresponding preset deviation threshold, trigger the correction of the puncture path; Obtain the compensation amount calculation function, input the lateral offset and the longitudinal depth deviation into the compensation amount calculation function, and output the lateral offset compensation amount and the depth direction compensation amount of the puncture needle; Determine the bending form adjustment parameters of the puncture needle according to the tip direction angle deviation, and generate a correction path including the needle body bending compensation amount.

7. The breast puncture surgical robot navigation system based on multi-modal image fusion according to claim 1, wherein: When the risk assessment unit is executed, compare the change trends of the axial force, the lateral force, and the torque with the corresponding preset safety thresholds. If any one of the change trends of the axial force, the lateral force, or the torque exceeds the corresponding preset safety threshold, directly determine that the currently executed puncture navigation path is risky, and stop the puncture of the robot; Otherwise, perform a fusion process on the change trends of the axial force, the lateral force, and the torque, and synchronously output the fusion evaluation index; Compare the fusion evaluation index with the preset comprehensive evaluation range. When the fusion evaluation index is within the comprehensive evaluation range, it is determined that the currently executed puncture navigation path is effective, and the puncture surgery is continued along the currently executed puncture navigation path. Otherwise, it is determined that there is a risk in the currently executed puncture navigation path, and the puncture of the robot is stopped.

8. The breast puncture surgical robot navigation system based on multi-modal image fusion according to claim 7, wherein: When the path adjustment unit is executed, ultrasonic images are re-acquired to detect the displacement and deformation of the breast tissue caused by the puncture, and local registration is performed with the MRI and CT images to update the fusion image data of the unpunctured area. In the updated fusion image data, a collision avoidance area is delimited with the current needle insertion point of the puncture needle as the center point. Among them, the collision avoidance area is circular, and the radius of the collision avoidance area is twice the distance from the tip of the puncture needle to the center point when the change trends of the axial force, lateral force, and torque mutate. According to the updated fusion image data and the collision avoidance area, a new puncture path that bypasses the collision avoidance area and reaches the lesion is re-planned, and the subsequent puncture operation is performed according to the new puncture path.

9. The breast puncture surgical robot navigation system based on multi-modal image fusion according to claim 1, wherein: The acousto-optic warning signal includes an acoustic signal and an optical signal. The acoustic signal is a beeping sound of a preset frequency, and the optical signal is a flashing light of a preset color. The acousto-optic warning signal is emitted by an acousto-optic alarm installed in the operating room.

10. The breast puncture surgical robot navigation device based on multi-modal image fusion, wherein: The navigation device includes: At least one processor; And a memory communicatively connected to the at least one processor; Wherein, the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the multi-modal image fusion-based breast puncture surgery robot navigation system according to any one of claims 1 to 9.

Images