Surgical Procedure Monitoring Method and System Based on Machine Vision

Abstract

The present invention provides a method and system for monitoring surgical procedures based on machine vision. First, a real-time image sequence of a surgical scene containing dynamic images of the operation area of surgical instruments and biological tissue areas is acquired. Then, the recognition results of surgical instruments (including instrument types and spatial postures) and the segmentation results of biological tissues (including tissue types and regional boundaries) are extracted from the images. Next, based on the above results, the operation trajectories of the instruments and the deformation parameters of the biological tissue areas are monitored to generate dynamic operation state parameters. Then, according to predefined safety thresholds for surgical stages, multi-modal anomaly detection is performed on the dynamic operation state parameters to determine the types of abnormal events and the risk levels. Finally, a visual surgical monitoring report is generated based on this, which covers the marking of abnormal areas, risk warning information, and correction suggestions, realizing comprehensive and intelligent monitoring of the surgical procedure and assisting in improving the safety and success rate of the surgery.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular, to a method and system for monitoring surgical procedures based on machine vision. Background Art

[0002] In the modern medical field, surgery, as an important means of treating many diseases, its safety and accuracy have always been the focus of attention. During the surgical procedure, doctors need to constantly monitor the operation of surgical instruments and the state of biological tissues to ensure the smooth progress of the surgery and reduce the occurrence of complications.

[0003] Currently, the monitoring of surgical procedures mainly relies on doctors' experience and on-site observation. However, this traditional monitoring method has many limitations. On the one hand, the surgical environment is complex. During the highly tense and concentrated operation process, doctors are inevitably prone to visual fatigue or negligence, resulting in the inability to detect some subtle but abnormal situations that may affect the surgical outcome in a timely manner, such as improper operation of surgical instruments or unexpected deformation of biological tissues. On the other hand, there are differences in the experience levels of different doctors, and their abilities to judge and identify potential risks in some complex surgical scenarios are also different, which may lead to untimely and inaccurate handling of abnormal events during the surgical procedure, thereby affecting the surgical effect and the prognosis of patients.

[0004] Some existing surgical monitoring technologies mainly focus on monitoring the operating status of surgical equipment, such as monitoring the electrical parameters of surgical instruments, the temperature of equipment, etc., while relatively less attention is paid to the complex interaction between instruments and biological tissues during the surgical procedure and the actual operating status. Even for some image monitoring technologies for surgical scenarios, most of them only stay at the level of simple image recording, failing to deeply analyze the key information in the images, unable to accurately extract key elements such as the type of surgical instrument, spatial posture, and the type and regional boundary of biological tissues, and it is even more difficult to effectively monitor the operating trajectory of surgical instruments and the deformation parameters of biological tissues.

[0005] In addition, most existing methods only set fixed thresholds for detecting single types of abnormal situations, unable to adapt to the diverse risks faced in different stages and different operations during the surgical procedure, and it is difficult to accurately judge the type and risk level of abnormal events. Moreover, after an abnormal situation occurs, existing technologies are also unable to provide an intuitive and targeted visualization report to help doctors quickly understand the abnormal situation and obtain effective corrective suggestions. Summary of the Invention

[0006] In view of the above-mentioned problems, in combination with the first aspect of the present invention, embodiments of the present invention provide a method for monitoring surgical procedures based on machine vision, and the method includes:

[0007] Obtain a real-time image sequence of the surgical scene, where the real-time image sequence includes dynamic images of the surgical instrument operation area and the biological tissue area;

[0008] Extract the surgical instrument recognition result and the biological tissue segmentation result from the real-time image sequence. The surgical instrument recognition result includes the instrument type and the spatial pose, and the biological tissue segmentation result includes the tissue type and the regional boundary;

[0009] Based on the surgical instrument recognition result and the biological tissue segmentation result, monitor the operation trajectory of the surgical instrument and the deformation parameters of the biological tissue area, and generate dynamic operation state parameters;

[0010] According to the predefined surgical stage safety threshold, perform multi-modal anomaly detection on the dynamic operation state parameters to determine the anomaly event type and the risk level;

[0011] Based on the anomaly event type and the risk level, generate a visual surgical monitoring report, where the visual surgical monitoring report includes anomaly area markings, risk prompt information, and correction suggestions.

[0012] In another aspect, an embodiment of the present invention further provides a surgical process monitoring system based on machine vision, including a processor and a machine-readable storage medium. The machine-readable storage medium is connected to the processor. The machine-readable storage medium is used to store programs, instructions, or codes, and the processor is used to execute the programs, instructions, or codes in the machine-readable storage medium to implement the above method.

[0013] Based on the above aspects, the embodiments of the present application obtain a real-time image sequence including dynamic images of the surgical instrument operation area and the biological tissue area, and extract the surgical instrument recognition result and the biological tissue segmentation result therefrom. Thus, the operation trajectory of the surgical instrument and the deformation parameters of the biological tissue area are monitored, and dynamic operation state parameters are generated, thereby quantifying and integrating the dynamic information in the surgical process. Then, a predefined surgical stage safety threshold is used to perform multi-modal anomaly detection on the dynamic operation state parameters, which can comprehensively and flexibly judge the abnormal conditions in the surgical process based on the characteristics of different surgical stages, accurately determine the anomaly event type and the risk level, not only considering the diversity and complexity of surgical operations, but also combining the safety standards of different stages, greatly improving the accuracy and reliability of anomaly detection and avoiding the limitations of a single detection method. Finally, a visual surgical monitoring report is generated based on the anomaly event type and the risk level, presenting the complex anomaly information to medical staff in an intuitive and easy-to-understand manner, including anomaly area markings, risk prompt information, and correction suggestions, which not only helps medical staff quickly locate and understand the problems in the surgery, but also provides them with targeted solutions, helps to take timely measures to correct surgical operations, reduce surgical risks, and improve the surgical success rate and patient safety. Description of the Drawings

[0014] Figure 1 It is a schematic flowchart of the execution process of the surgical procedure monitoring method based on machine vision provided by an embodiment of the present invention.

[0015] Figure 2 It is a schematic diagram of exemplary hardware and software components of the surgical procedure monitoring system based on machine vision provided by an embodiment of the present invention. Detailed Embodiments

[0016] The present invention will be specifically described below with reference to the accompanying drawings of the specification. Figure 1 It is a schematic flowchart of the surgical procedure monitoring method based on machine vision provided by an embodiment of the present invention. The surgical procedure monitoring method based on machine vision will be introduced in detail below.

[0017] Step S110: Obtain a real-time image sequence of the surgical scene, where the real-time image sequence includes dynamic images of the surgical instrument operation area and the biological tissue area.

[0018] In this embodiment, taking coronary artery bypass grafting as an example, during the coronary artery bypass grafting procedure, a real-time image sequence of the surgical scene can be obtained by a camera installed above the operating table. For example, the frame rate of the camera is set to 30 frames per second, and the resolution is 1920×1080 pixels to ensure that the dynamic images of the surgical instrument operation area and the biological tissue area can be clearly captured. The surgical instrument operation area may include the operation ranges of instruments such as vascular forceps and staplers used in coronary artery bypass grafting, and the biological tissue area covers the heart, coronary arteries, and surrounding vascular tissues, etc. As the surgery progresses, the camera continuously takes pictures, and 30 images containing the above areas are generated every second. These images arranged in chronological order can form a real-time image sequence. For example, when a surgeon uses vascular forceps to clamp the coronary artery, the camera can accurately record the dynamic images such as the moment when the vascular forceps contact the coronary artery, the movement process of the vascular forceps, and the morphological changes of the coronary artery.

[0019] Step S120: Extract the surgical instrument recognition result and the biological tissue segmentation result from the real-time image sequence. The surgical instrument recognition result includes the instrument type and the spatial posture, and the biological tissue segmentation result includes the tissue type and the regional boundary.

[0020] Specifically, first, key-frame sampling is performed on the real-time image sequence, and one frame is selected as a key frame every 5 seconds to obtain multiple groups of static images of the surgical scene. For each group of static images of the surgical scene, gray-level equalization processing is first performed to enhance the contrast difference between the instrument operation area and the biological tissue area. For example, in the image of coronary artery bypass grafting surgery, the gray values presented by the heart tissue and blood vessels are originally less different from the gray value of the surgical instrument. After gray-level equalization, the contrast between the metal part of the vascular clamp and the heart tissue is significantly increased. Then, the pixel area with a gradient amplitude exceeding the preset threshold in the image after gray-level equalization processing is extracted through an edge detection algorithm to generate an initial candidate box set. In this scenario, the preset threshold is set to 50. For instruments such as vascular clamps, the gradient amplitude of their edge parts is large and can be accurately identified and included in the initial candidate box set.

[0021] Next, the region growing algorithm is performed on the initial candidate box set to merge the candidate boxes with overlapping spatial positions and similar texture features, generating a merged candidate box set. For example, the two clamping arms of the vascular clamp may be separately recognized in the initial candidate boxes. Through the region growing algorithm, they are merged into a complete vascular clamp candidate box according to their similar texture features. Then, according to the preset instrument size range and tissue size range, the candidate boxes that meet the corresponding size constraints are screened from the merged candidate box set to obtain a preliminarily screened candidate box set. For example, the length range of the vascular clamp is preset to 10 - 20 cm, and the width is 1 - 3 cm. The candidate boxes that do not meet this size range will be excluded. Finally, morphological closing operation processing is performed on the preliminarily screened candidate box set to fill the broken areas inside the candidate boxes and smooth the edge serrations of the candidate boxes, generating the final candidate boxes for the instrument operation area and the biological tissue area.

[0022] Based on a three-dimensional convolutional network, multi-scale feature matching is performed on the candidate boxes of the instrument operation area to determine the instrument type and spatial pose, and generate the surgical instrument recognition result. For the vascular clamp in coronary artery bypass grafting surgery, the three-dimensional convolutional network accurately identifies its type as a vascular clamp by analyzing its features at different scales, such as the bending shape of the clamping arms and the structure of the clamp head, and at the same time determines the spatial pose of the vascular clamp, such as the opening direction of the vascular clamp and the angle with the heart surface. Through a semantic segmentation model, pixel-level classification is performed on the candidate boxes of the tissue area to obtain the tissue type and regional boundary, and generate the biological tissue segmentation result. In coronary artery bypass grafting surgery, the semantic segmentation model can accurately distinguish different tissue types such as heart tissue, coronary artery, and venous blood vessels, and accurately depict their regional boundaries, such as the diameter, direction, and connection area with the heart tissue of the coronary artery. Finally, the surgical instrument recognition result and the biological tissue segmentation result are aligned in space and time to form a unified spatial coordinate system mapping relationship, so that the relative position relationship between the surgical instrument and the biological tissue can be accurately described in the same coordinate system.

[0023] Step S130: Based on the surgical instrument recognition result and the biological tissue segmentation result, monitor the operation trajectory of the surgical instrument and the deformation parameters of the biological tissue region, and generate dynamic operation state parameters.

[0024] Specifically, according to the spatial posture in the surgical instrument recognition result, calculate the instrument movement direction and speed at continuous time stamps to generate an operation trajectory. For example, taking a hemostatic forceps as an example, if at time stamp t1, the hemostatic forceps is located at a certain point on the heart surface, its spatial posture is at an angle of 30 degrees with the heart surface, and the opening direction is towards the upper left. At time t2, the hemostatic forceps moves to another position, and the angle becomes 45 degrees, and the opening direction is towards the upper right. By calculating the position change and angle change of the hemostatic forceps at these two time stamps, the movement direction of the hemostatic forceps can be obtained as from the upper left to the upper right, and the speed is calculated based on the distance between the two points and the time interval (t2 - t1).

[0025] Based on the regional boundary in the biological tissue segmentation result, analyze the tissue deformation amplitude and deformation rate between adjacent time stamps to generate deformation parameters. For example, when a surgeon uses a stapler to anastomose the coronary artery, the regional boundary of the coronary artery will change. At time stamp t1, the diameter of a certain segment of the coronary artery is 2 mm. At time t2, due to the extrusion of the stapler, the diameter becomes 1.8 mm. By calculating the change amount of the diameter (2 - 1.8 = 0.2 mm), the deformation amplitude is obtained, and then the deformation rate is calculated according to the time interval (t2 - t1).

[0026] Fuse the operation trajectory and the deformation parameters, and calculate the stress distribution map of the contact area between the surgical instrument and the tissue. During coronary artery bypass grafting, when the hemostatic forceps clamps the coronary artery, according to the operation trajectory of the hemostatic forceps (such as the clamping force and direction) and the deformation parameters of the coronary artery (such as the compression degree of the clamped part), the stress distribution of the contact area between the hemostatic forceps and the coronary artery is calculated through a mechanical model. For example, the stress is relatively large around the contact point and gradually decreases towards the surrounding.

[0027] According to the stress distribution map, quantitative indicators of the operating force of the instrument, the elastic modulus of tissue response, and the proportion of contact time are generated. For the case of using a vascular clamp to grasp the coronary artery, the operating force of the vascular clamp is calculated to be 10 Newtons according to the stress distribution map. Through the analysis of the deformation of the coronary artery under stress, the elastic modulus of tissue response is obtained as 100 MPa. The proportion of contact time is calculated to be 5% based on the total time of using the vascular clamp to grasp the coronary artery and the total time of the entire surgical operation. These quantitative indicators are matched with the preset surgical stage model to output dynamic operation state parameters. In different stages of coronary artery bypass grafting, such as the vascular dissection stage and the anastomosis stage, the preset surgical stage model has different required ranges for these quantitative indicators. When these quantitative indicators are within the corresponding stage range, the dynamic operation state parameters indicate that the surgical operation is proceeding normally; otherwise, it indicates that there may be risks.

[0028] Step S140, perform multimodal anomaly detection on the dynamic operation state parameters according to the predefined surgical stage safety thresholds to determine the anomaly event type and risk level.

[0029] Specifically, the time series feature sequences of the operating force of the instrument, the tissue deformation rate, and the stress distribution map in the dynamic operation state parameters can be extracted. For example, assume that the surgical operation lasts for 10 minutes, and the data of the operating force of the instrument, the tissue deformation rate, and the stress distribution map are recorded every 30 seconds. In this way, a time series feature sequence is obtained.

[0030] According to the predefined surgical stage safety thresholds, the time series feature sequence is segmented into time window segments that match the current surgical stage. For example, in the vascular anastomosis stage of coronary artery bypass grafting, if the time window is set to 2 minutes, then the 10-minute time series feature sequence is segmented into segments of 2 minutes each.

[0031] Perform a difference operation between the average value of the operating force of the instrument within each time window segment and the upper limit value of the force in the surgical stage safety thresholds to generate a force deviation coefficient. In the vascular anastomosis stage, if the upper limit value of the operating force of the instrument in the safety thresholds is 15 Newtons, and the average value of the operating force of the instrument within a certain time window is 20 Newtons, then the force deviation coefficient is (20 - 15) / 15 = 0.33.

[0032] Synchronously compare the peak value of the tissue deformation rate within the time window segment with the deformation rate threshold in the surgical stage safety thresholds to generate a rate exceeding standard flag bit. Assume that the deformation rate threshold in the vascular anastomosis stage is 0.5 mm / s, and the peak value of the tissue deformation rate within a certain time window is 0.8 mm / s. Then the rate exceeding standard flag bit is 1, indicating exceeding the standard.

[0033] Calculate the overlap rate between the covered area of the stress distribution map and the contact area threshold in the surgical stage safety threshold to generate an area anomaly ratio value. For example, if the contact area covered by the stress distribution map is 10 square millimeters and the contact area threshold is 8 square millimeters, then the area anomaly ratio value is (10 - 8) / 8 = 0.25.

[0034] Input the force deviation coefficient, rate over - standard flag bit, and area anomaly ratio value into a pre - trained multi - level decision tree model. Among them, the first - level decision node judges whether the force deviation coefficient exceeds the first risk critical value. Suppose the first risk critical value is 0.3. Since the above - mentioned force deviation coefficient is 0.33, it exceeds this critical value. The second - level decision node generates a tissue cumulative damage index based on the trigger times of the rate over - standard flag bit. Since the rate over - standard flag bit is 1, assume that the tissue cumulative damage index increases by 0.2 for each over - standard time. If there was no over - standard situation before, the tissue cumulative damage index is 0.2 at this time. The third - level decision node generates a contact anomaly pattern based on the area anomaly ratio value and the deformation spatial distribution characteristics. According to the area anomaly ratio value and the spatial distribution of the stress distribution map, it is judged that the contact anomaly pattern is excessive local pressure.

[0035] Based on the intermediate judgment results output by the multi - level decision tree model, activate the corresponding abnormal event type labels. The abnormal event type labels at least include three categories: instrument over - pressure, tissue over - speed deformation, and abnormal contact area. Since the force deviation coefficient exceeds the critical value and the contact anomaly pattern is excessive local pressure, activate the abnormal event type label of instrument over - pressure.

[0036] Real - time obtain the change amount of the regional boundary pixels in the biological tissue segmentation result. When detecting the continuous expansion or contraction form of the boundary pixels, generate a tissue integrity damage signal. In a coronary artery bypass grafting operation, if the regional boundary pixels of the coronary artery continuously contract, it may indicate that the coronary artery is overly compressed, and a tissue integrity damage signal is generated.

[0037] Perform a logical AND operation on the tissue integrity damage signal and the activated abnormal event type label. If there is an overlapping time interval between the two, increase the risk level of the corresponding abnormal event type by one level. Suppose the initial risk level of the abnormal event type of instrument over - pressure is level 1. Since the tissue integrity damage signal and instrument over - pressure exist simultaneously in a certain time interval, increase the risk level of instrument over - pressure to level 2.

[0038] According to the level mapping rule set in the surgical stage safety threshold, bind the final abnormal event type to the increased risk level. In the safety threshold setting of a coronary artery bypass grafting operation, risk level 2 corresponds to the situation that requires timely adjustment of the operation. Bind the abnormal event type of instrument over - pressure to risk level 2.

[0039] Step S150: Generate a visual surgical monitoring report based on the abnormal event type and risk level. The visual surgical monitoring report includes abnormal area markings, risk warning information, and correction suggestions.

[0040] Specifically, based on the tissue area boundary corresponding to the abnormal event type and the coordinates of the instrument contact point, locate the abnormal area in the real-time image sequence, and generate a color-coded contour line and text label that match the risk level. For the abnormal event type of instrument overpressure, based on the boundary of the area where the coronary artery is overly compressed and the coordinates of the contact point between the vascular forceps and the coronary artery, find the corresponding position in the real-time image sequence. When the risk level is level 2, use a red color-coded contour line to mark the abnormal area and add a text label of "Instrument Overpressure - Risk Level 2" beside it.

[0041] Generate a multi-level voice alarm signal based on the alarm priority corresponding to the risk level. The alarm priority corresponding to risk level 2 is high, and the surgical monitoring system will emit a rapid voice alarm signal of "beep beep beep", and at the same time, voice prompt "Instrument overpressure, please pay attention to adjusting the operation".

[0042] Match the instrument path adjustment rules and force optimization parameters in the predefined surgical operation correction plan library according to the abnormal event type. Among them, the instrument overpressure type generates a reverse displacement vector arrow and a pressure decreasing curve. In coronary artery bypass grafting surgery, display a reverse displacement vector arrow in the monitoring report, indicating that the vascular forceps should move in the opposite direction, and at the same time give a pressure decreasing curve, showing how the pressure should gradually decrease to the safe range if the operation is performed correctly.

[0043] Synchronously transmit the color-coded contour line, text label, voice alarm signal, and correction guidance parameters to the display terminal of the surgical navigation system. Divide the first display area on the terminal interface to render the original image stream in real time, the second display area to overlay the abnormal contour and correction guidance layer, and the third display area to scroll and update the risk level and voice-to-text log. On the display terminal of the surgical navigation system, the first display area normally displays the real-time operation picture of coronary artery bypass grafting surgery, the second display area overlays a red abnormal area contour, a reverse displacement vector arrow, a pressure decreasing curve and other correction guidance layers on the original picture, and the third display area continuously scrolls and updates the log information of the risk level of level 2 and the voice-to-text "Instrument overpressure, please pay attention to adjusting the operation".

[0044] According to the dynamic changes in the progress of the surgical stage and the risk level, the color intensity of the contour line and the correction guidance parameters in the second display area are updated in real time. When the risk level of the same abnormal event type is detected to increase, the contour line of the corresponding area is switched to a higher-priority color and the display size of the correction guidance arrow is enlarged. If the risk level of instrument overpressure further increases to level 3, then in the second display area, the color intensity of the red contour line will increase, and the display size of the reverse displacement vector arrow will be enlarged, more prominently reminding the surgeon to pay attention to the abnormal situation and adjust the operation in a timely manner.

[0045] Based on the above steps, the embodiment of the present application obtains a real-time image sequence including dynamic images of the surgical instrument operation area and the biological tissue area, extracts the surgical instrument recognition result and the biological tissue segmentation result therefrom, thereby monitors the operation trajectory of the surgical instrument and the deformation parameters of the biological tissue area, and generates dynamic operation state parameters, so as to quantify and integrate the dynamic information during the surgical process. Then, a predefined surgical stage safety threshold is used to perform multi-modal anomaly detection on the dynamic operation state parameters, which can comprehensively and flexibly judge the abnormal situation during the surgical process based on the characteristics of different surgical stages, accurately determine the abnormal event type and risk level, not only considering the diversity and complexity of surgical operations, but also combining the safety standards of different stages, greatly improving the accuracy and reliability of anomaly detection and avoiding the limitations of a single detection method. Finally, a visual surgical monitoring report is generated based on the abnormal event type and risk level, presenting complex abnormal information to medical staff in an intuitive and easy-to-understand manner, including abnormal area markings, risk warning information, and correction suggestions, which not only helps medical staff quickly locate and understand the problems in the surgery, but also provides them with targeted solutions, contributing to taking timely measures to correct surgical operations, reducing surgical risks, and improving the surgical success rate and patient safety.

[0046] In a possible implementation manner, step S120 includes:

[0047] Step S121, performing key-frame sampling on the real-time image sequence to obtain multiple groups of static surgical scene images.

[0048] For example, considering the rhythm of the surgery and the key nodes of the operation, key-frame sampling can be set to be performed every 10 seconds. Since the entire surgery may last for several hours, multiple groups of static surgical scene images can thus be obtained at different time points. For example, a static surgical scene image is obtained at 10 seconds, 20 seconds, 30 seconds, etc. after the start of the surgery. These static surgical scene images include the operation conditions of surgical instrument operation areas such as vascular forceps and staplers, as well as the states of biological tissue areas such as the heart and coronary arteries.

[0049] Step S122: Extract the feature regions from each group of the static surgical scene images to obtain the candidate boxes for the instrument operation regions and the candidate boxes for the tissue regions.

[0050] Specifically, in a possible implementation manner, step S122 includes:

[0051] Step S1221: Perform gray-level equalization processing on the static surgical scene images to enhance the contrast difference between the instrument operation regions and the biological tissue regions.

[0052] For the images of coronary artery bypass grafting surgery, the gray levels of biological tissues such as heart tissues and coronary arteries may originally be relatively close to those of surgical instruments such as vascular forceps and staplers. After gray-level equalization processing, for example, the gray value of the metal part of the vascular forceps will form a sharp contrast with the surrounding heart tissues, making it easier to distinguish.

[0053] Step S1222: Extract the pixel regions in the images after gray-level equalization processing whose gradient magnitudes exceed a preset threshold through an edge detection algorithm to generate an initial candidate box set.

[0054] For example, set the preset threshold to 30. Since the edge parts of the vascular forceps and the edge parts of the coronary arteries have relatively large gradient magnitudes in the image, these regions will be accurately identified and included in the initial candidate box set.

[0055] Step S1223: Perform a region growing algorithm on the initial candidate box set to merge the candidate boxes with overlapping spatial positions and similar texture features, generating a merged candidate box set.

[0056] For example, the two arm parts of the vascular forceps may initially be recognized as two separate candidate boxes. However, due to their overlapping spatial positions and similar texture features, these two candidate boxes will be merged into a complete candidate box for the vascular forceps through the region growing algorithm.

[0057] Step S1224: According to the preset instrument size range and tissue size range, screen the candidate boxes that meet the corresponding size constraints from the merged candidate box set to obtain a preliminarily screened candidate box set.

[0058] For the vascular forceps, according to its actual size range, assume that its length range is set to 8 - 18 cm and the width is 0.8 - 2.5 cm. For the coronary artery, the diameter range is set to 1 - 3 mm. The candidate boxes that do not meet these size ranges will be excluded.

[0059] Step S1225: Perform morphological closing operation processing on the preliminarily screened candidate box set to fill the internal broken regions of the candidate boxes and smooth the edge serrations of the candidate boxes, generating the final candidate boxes for the instrument operation regions and the candidate boxes for the biological tissue regions.

[0060] For example, for the candidate bounding box of a hemostat, if there are internal fracture regions caused by image noise or other reasons, they will be filled after morphological closing operation, and the edge serrations will also be smoothed, making the candidate bounding box of the hemostat more accurate and complete. Similarly, such processing will also be performed on the candidate bounding box of the tissue region of the coronary artery.

[0061] Step S123: Based on a three-dimensional convolutional network, perform multi-scale feature matching on the candidate bounding box of the instrument operation region, determine the type and spatial pose of the instrument, and generate the surgical instrument recognition result.

[0062] For example, for the hemostat in the candidate bounding box of the instrument operation region, the three-dimensional convolutional network can analyze it from multiple scales. From a macroscopic scale, the overall shape structure of the hemostat can be recognized, such as the approximate length and bending degree of the forceps arms; from a microscopic scale, the fine structural features of the forceps head can be recognized. Through these multi-scale feature matches, it can be accurately recognized that the instrument is of the hemostat type, and its spatial pose can be determined. For example, the opening angle of the hemostat relative to the surface of the heart, assuming that at a certain moment the opening direction of the hemostat forms a 45-degree angle with the surface of the heart, and the forceps arms are parallel to the long axis direction of the heart. These are all specific descriptions of the spatial pose, thus generating the surgical instrument recognition result.

[0063] Step S124: Through a semantic segmentation model, perform pixel-level classification on the candidate bounding box of the tissue region, obtain the tissue type and region boundary, and generate the biological tissue segmentation result.

[0064] Specifically, the semantic segmentation model classifies the pixels in the candidate bounding box of the tissue region. For heart tissue, different tissue types such as myocardial tissue, endocardium, and epicardium can be distinguished. For the coronary artery, its region boundary can be accurately defined, including its starting point, direction, branching situation on the surface of the heart, and its connection relationship with the surrounding tissues. For example, the semantic segmentation model can accurately depict that the diameter of a certain segment of the coronary artery is 1.5 millimeters, it starts from the left anterior descending branch of the heart, extends along the surface of the heart to the right and branches out a small branch at a certain position. These information constitute the content of the tissue type and region boundary, thus generating the biological tissue segmentation result.

[0065] Step S125: Align the surgical instrument recognition result and the biological tissue segmentation result in space and time to form a unified spatial coordinate system mapping relationship.

[0066] In the context of coronary artery bypass grafting surgery, this means placing the surgical instrument recognition results of the vascular clamp, such as information about the position and orientation of the vascular clamp, in the same spatial coordinate system as the position, boundaries, etc. in the segmentation results of biological tissues such as the heart tissue and coronary arteries. For example, in this unified spatial coordinate system, it is possible to accurately describe the specific coordinate position on the heart surface of the contact point between the clamp head of the vascular clamp and the coronary artery, as well as the angular relationship between the operating direction of the vascular clamp and the direction of the coronary artery, thereby establishing an accurate spatial correspondence relationship to facilitate subsequent surgical monitoring and analysis operations.

[0067] In one possible implementation, step S130 includes:

[0068] Step S131, calculate the instrument movement direction and speed at consecutive time stamps according to the spatial orientation in the surgical instrument recognition result, and generate the operation trajectory.

[0069] Still taking the vascular clamp as an example of the surgical instrument, assume that at time stamp t1, the spatial orientation of the vascular clamp is determined through the surgical instrument recognition result as follows: its clamp head is located at the position with heart surface coordinates (x1, y1, z1), the angle between the clamp arm and the heart surface is α1, and the opening direction is the vector direction (v1). At the subsequent time stamp t2, the position of the clamp head of the vascular clamp becomes (x2, y2, z2), the angle between the clamp arm and the heart surface becomes α2, and the opening direction is the vector direction (v2). When calculating the instrument movement direction, first calculate the displacement vector from the coordinates (x1, y1, z1) to (x2, y2, z2), and this vector direction represents the movement direction. For example, calculate the displacement in the x direction as x2 - x1, in the y direction as y2 - y1, and in the z direction as z2 - z1 to obtain the displacement vector (dx, dy, dz). The speed is calculated according to the distance formula between two points. First, calculate the magnitude of the displacement, that is, the displacement distance s is equal to the square root of ((x2 - x1) squared + (y2 - y1) squared + (z2 - z1) squared), and then divide it by the time interval (t2 - t1) to obtain the speed value. By repeating such calculations at different consecutive time stamps, the operation trajectory of the vascular clamp can be completely determined.

[0070] Step S132, analyze the tissue deformation amplitude and deformation rate between adjacent time stamps based on the regional boundaries in the biological tissue segmentation result, and generate the deformation parameter.

[0071] For example, for the coronary artery, a biological tissue in coronary artery bypass surgery, obtain the regional boundary information at time stamp t1 from the biological tissue segmentation result. For example, the diameter of a certain cross-section of the coronary artery is d1. At the adjacent time stamp t2, obtain the diameter of the same cross-section of the coronary artery again, which is d2. The calculation of the tissue deformation amplitude is the absolute value of d1 - d2. The calculation of the deformation rate is to divide the deformation amplitude by the time interval (t2 - t1). If d1 is 3 mm, d2 is 2.8 mm, and the time interval is 10 seconds, then the deformation amplitude is 3 - 2.8 = 0.2 mm, and the deformation rate is 0.2 mm divided by 10 seconds, that is, 0.02 mm / s. Thus, the deformation parameters of the coronary artery are output.

[0072] Step S133: Integrate the operation trajectory and the deformation parameters to calculate the stress distribution map of the contact area between the surgical instrument and the tissue.

[0073] For example, when the vascular forceps contact the coronary artery, calculate the stress distribution map based on the previously obtained operation trajectory of the vascular forceps (such as the moving direction and speed, etc.) and the deformation parameters of the coronary artery (such as the change in diameter, etc.). Assume the moving speed of the vascular forceps is v, and the deformation of the coronary artery is Δd. According to the mechanical principle, the stress in the contact area is related to the moving speed of the vascular forceps and the deformation of the coronary artery. For example, if there is a calculation model obtained from a large number of experiments and theoretical derivations that the stress σ is equal to k multiplied by v multiplied by Δd (where k is a coefficient determined according to the tissue characteristics and instrument characteristics), by substituting the speed value of the vascular forceps, the deformation value of the coronary artery, and the known coefficient k, the stress values at different points in the contact area can be calculated, thereby constructing the stress distribution map.

[0074] Step S134: Generate quantitative indicators of the instrument operation force, tissue response elastic modulus, and contact time ratio according to the stress distribution map.

[0075] For example, for the instrument operation force, in the stress distribution map, the magnitude of the stress is directly related to the instrument operation force. For example, if the stress value at the center of the contact area in the stress distribution map is σ0, according to the relationship between force and stress (force equals stress multiplied by the contact area), assuming the contact area is A, then the instrument operation force F is equal to σ0 multiplied by A. For the tissue response elastic modulus, it is calculated according to the deformation of the coronary artery under the action of stress. Assume that under the action of stress σ, the strain of the coronary artery (strain equals deformation amplitude divided by the original size) is ε. According to the definition of elastic modulus (elastic modulus equals stress divided by strain), the tissue response elastic modulus E is equal to σ divided by ε. The contact time ratio is calculated according to the contact time between the vascular forceps and the coronary artery and the entire surgical operation time. Assume the contact time between the vascular forceps and the coronary artery is t0, and the entire surgical operation time is T, then the contact time ratio is equal to t0 divided by T.

[0076] Step S135: Match the quantization index with a preset surgical stage model and output the dynamic operation state parameter.

[0077] In different stages of coronary artery bypass grafting surgery, such as the vascular dissection stage, the vascular anastomosis stage, etc., there are preset surgical stage models. The surgical stage model stipulates the reasonable ranges of the instrument operation force, the tissue response elastic modulus, and the contact time ratio under normal operation in each stage. For example, in the vascular anastomosis stage, the preset instrument operation force range is F1 - F2, the tissue response elastic modulus range is E1 - E2, and the contact time ratio range is p1 - p2. Compare the quantization indexes of the instrument operation force, the tissue response elastic modulus, and the contact time ratio calculated previously with these ranges. If the instrument operation force is within the range of F1 - F2, the tissue response elastic modulus is within the range of E1 - E2, and the contact time ratio is within the range of p1 - p2, then the dynamic operation state parameter indicates that the current surgical operation is in a normal state; if one or some of the quantization indexes exceed the corresponding ranges, the dynamic operation state parameter indicates that there may be risks and further analysis or adjustment of the operation is required.

[0078] In a possible implementation manner, step S132 includes:

[0079] Step S1321: Extract the tissue region boundary coordinate sequences corresponding to adjacent timestamps in the biological tissue segmentation result, perform point-by-point matching of the boundary coordinates of the previous timestamp and the boundary coordinates of the subsequent timestamp, and generate the displacement vector of each coordinate point.

[0080] For example, in coronary artery bypass grafting surgery, for the coronary artery, a biological tissue, obtain the coronary artery region boundary coordinate sequences corresponding to adjacent timestamps t1 and t2 from the biological tissue segmentation result. For example, at time t1, the boundary coordinate points of a certain cross-section of the coronary artery are (x1, y1, z1), (x2, y2, z2), etc., and the corresponding coordinate points at time t2 are (x1', y1', z1'), (x2', y2', z2'), etc. Perform point-by-point matching of these coordinate points and calculate the displacement vector of each coordinate point. Taking the coordinate point (x1, y1, z1) as an example, its displacement vector is (x1' - x1, y1' - y1, z1' - z1). By performing such calculations for all coordinate points, the displacement vector of each coordinate point is obtained.

[0081] Step S1322: Calculate the average value of the overall displacement of the tissue region boundary between adjacent timestamps according to the length of the displacement vector, and generate the tissue deformation amplitude.

[0082] Specifically, for each displacement vector, calculate its length. For example, the length of the displacement vector (dx, dy, dz) is the square root of (dx squared + dy squared + dz squared). After calculating the lengths of all the displacement vectors of the coordinate points, find the average value of these lengths to obtain the average value of the overall displacement of the tissue region boundary. This average value of the overall displacement is the tissue deformation amplitude. Suppose there are n coordinate points, and the lengths of the displacement vectors of each coordinate point are L1, L2, …, Ln respectively. Then the tissue deformation amplitude is equal to (L1 + L2 + … + Ln) divided by n.

[0083] Step S1323: Based on the change rate of the displacement vector at consecutive time stamps, calculate the displacement acceleration of each coordinate point. Mark the coordinate points whose acceleration exceeds the preset acceleration threshold as deformation mutation points. Count the distribution density of the deformation mutation points within the tissue region boundary. Generate the local peak value of the tissue deformation rate according to the ratio of the distribution density to the acceleration threshold.

[0084] Specifically, for the displacement vector of each coordinate point, calculate its change rate at consecutive time stamps to obtain the displacement acceleration. For example, at time stamps t1, t2, t3, the displacement vectors of a certain coordinate point are (v1, v2, v3) respectively. First, calculate the change in the displacement vector from t1 to t2 as (v2 - v1), and the change in the displacement vector from t2 to t3 as (v3 - v2). Then divide them by the corresponding time intervals to obtain the displacement accelerations for these two time periods. Suppose the preset acceleration threshold is a0. If the displacement acceleration of a certain coordinate point exceeds a0, then mark it as a deformation mutation point. Count the number of all deformation mutation points within the tissue region boundary as m, and the total number of coordinate points within the tissue region boundary as N. Then the distribution density of the deformation mutation points is m divided by N. Generate the local peak value of the tissue deformation rate according to the ratio of the distribution density to the acceleration threshold, that is, (m divided by N) divided by a0.

[0085] Step S1324: Perform normalized weighted fusion on the tissue deformation amplitude, the distribution density of the deformation mutation points, and the local peak value of the tissue deformation rate to generate a set of deformation parameters.

[0086] For example, first perform normalization processing on the tissue deformation amplitude, the distribution density of the deformation mutation points, and the local peak value of the tissue deformation rate to make their values within the same order of magnitude range. Suppose the tissue deformation amplitude is A, the distribution density of the deformation mutation points is D, and the local peak value of the tissue deformation rate is P. Determine their weights as w1, w2, w3 respectively (these weights are determined in advance according to factors such as the importance of the impact on surgical safety, etc.). By calculating w1 multiplied by A + w2 multiplied by D + w3 multiplied by P, obtain a comprehensive value. Combine this value with other relevant information (such as coordinate information, etc.) to generate a set of deformation parameters.

[0087] Step S1325: According to the tissue type in the biological tissue segmentation result, match the predefined tissue elasticity attribute library, and extract the deformation tolerance coefficient corresponding to the tissue type.

[0088] In coronary artery bypass grafting, for the coronary artery tissue type, find its corresponding deformation tolerance coefficient from the predefined tissue elasticity attribute library. Suppose the deformation tolerance coefficient of the coronary artery is k, and this deformation tolerance coefficient reflects the degree of deformation that the coronary artery can withstand under normal conditions.

[0089] Step S1326: Perform a proportional operation on the tissue deformation amplitude in the deformation parameter set and the deformation tolerance coefficient to generate a deformation overrun risk index. According to the temporal change trend of the deformation overrun risk index, compare the index increment of adjacent timestamps with the predefined stage deformation increment threshold to generate a dynamic anomaly flag for deformation parameters.

[0090] Specifically, calculate the deformation overrun risk index. For example, if the tissue deformation amplitude in the deformation parameter set is A, then the deformation overrun risk index is equal to A divided by k. Analyze the temporal change trend of the deformation overrun risk index. At adjacent timestamps t1 and t2, the deformation overrun risk indices are I1 and I2 respectively, and calculate the index increment as I2 - I1. Suppose the predefined stage deformation increment threshold is ΔI. If I2 - I1 is greater than ΔI, then generate a dynamic anomaly flag for deformation parameters indicating an anomaly (for example, marked as 1), otherwise mark it as normal (for example, marked as 0).

[0091] Step S1327: Perform a spatial superposition of the deformation parameter set, the deformation overrun risk index, and the dynamic anomaly flag for deformation parameters to form a deformation parameter heat map with timestamp markings, and generate a predicted trajectory of tissue deformation propagation path based on the parameter change direction of consecutive timestamps in the deformation parameter heat map.

[0092] Specifically, the respective parameter values in the deformation parameter set, the deformation overrun risk index, and the dynamic anomaly flag for deformation parameters can be spatially superimposed according to their positions in the tissue region and marked with timestamps to form a deformation parameter heat map. In this deformation parameter heat map, observe the parameter change direction under consecutive timestamps. For example, the deformation parameters in a certain region gradually increase and move in a certain direction under consecutive timestamps. Based on this information, predict the tissue deformation propagation path to obtain a predicted trajectory of tissue deformation propagation path.

[0093] Step S1328: Adjust the dynamic weight coefficient of the deformation parameter according to the spatial distance between the predicted trajectory of tissue deformation propagation path and the operation trajectory of the surgical instrument to generate a final deformation parameter set.

[0094] For example, in the scenario of coronary artery bypass grafting surgery, calculate the spatial distance between the predicted trajectory of the deformation propagation path and the operation trajectory of the surgical instrument (such as the operation trajectory of a vascular clamp). Assume the spatial distance between the two is d. If d is small, it indicates that the operation of the surgical instrument has a greater impact on tissue deformation. At this time, appropriately increase the dynamic weight coefficient of the deformation parameters related to the operation of the surgical instrument (such as the amplitude of tissue deformation, etc.). If d is large, it indicates that the impact is small, and appropriately reduce the relevant weight coefficient. According to the adjusted weight coefficient, recalculate the previous weighted fusion and other operations to generate the final set of deformation parameters, which can more accurately reflect the deformation of the tissue and its relationship with the operation of the surgical instrument.

[0095] In one possible implementation, step S133 includes:

[0096] Step S1331, according to the instrument movement direction and speed at consecutive timestamps in the operation trajectory, extract the three-dimensional space coordinate sequence corresponding to the instrument tip at each timestamp.

[0097] For example, in coronary artery bypass grafting surgery, using a vascular clamp as the surgical instrument, assume that the timestamps are marked at fixed intervals (such as one timestamp per 1 second). At timestamp t1, the position coordinates of the vascular clamp tip are calculated as (x1, y1, z1) through the previously determined operation trajectory. At this time, the movement direction of the vascular clamp is the vector direction v1, and the speed is s1. As time goes by, at timestamp t2, the position of the vascular clamp tip becomes (x2, y2, z2), the movement direction is v2, and the speed is s2, etc. In this way, a series of three-dimensional space coordinate sequences of the vascular clamp tip at different timestamps can be obtained. This three-dimensional space coordinate sequence completely records the movement trajectory of the vascular clamp during the surgery.

[0098] Step S1332, based on the tissue deformation amplitude and deformation rate between adjacent timestamps in the deformation parameters, extract the deformation region boundary coordinate sequence of the tissue surface at the corresponding timestamps.

[0099] For the biological tissue of the coronary artery in coronary artery bypass grafting surgery, obtain information from the previously analyzed deformation parameters. Between adjacent timestamps t1 and t2, the tissue deformation amplitude is Δd1, and the deformation rate is r1. At time t1, the deformation region boundary coordinate points on the surface of the coronary artery are (a1, b1, c1), (a2, b2, c2), etc. At time t2, due to tissue deformation, these coordinate points may become (a1', b1', c1'), (a2', b2', c2'), etc. In this way, the deformation region boundary coordinate sequence of the coronary artery surface can be obtained at each timestamp.

[0100] Step S1333: Synchronize and align the three-dimensional spatial coordinate sequence of the instrument tip with the boundary coordinate sequence of the deformation region in time to generate a mapping relationship between the position of the instrument contact point after synchronization and the tissue deformation region.

[0101] Since the timestamps of the operation trajectory and tissue deformation are set at the same time interval, they can be made to correspond one by one. For example, when the coordinate of the vascular forceps tip at timestamp t1 is (x1, y1, z1), the boundary coordinates of the deformation region of the coronary artery at time t1 are (a1, b1, c1), etc. Through such time synchronization and alignment, the relative position relationship between the vascular forceps tip and the surface of the coronary artery at each time point can be determined, thus establishing a mapping relationship between the position of the instrument contact point and the tissue deformation region. This mapping relationship enables accurate knowledge of the specific position where the vascular forceps contact the coronary artery during the operation and the corresponding tissue deformation conditions.

[0102] Step S1334: Locate the spatial range of the contact area between the instrument tip and the tissue surface in the boundary coordinate sequence of the deformation region according to the position of the instrument contact point after synchronization, and calculate the deformation acceleration distribution of the contact area in the instrument movement direction based on the tissue deformation rate at each timestamp within the spatial range of the contact area.

[0103] Based on the established mapping relationship, when the contact point position between the vascular forceps tip and the surface of the coronary artery is determined, for example, within a certain time period, the contact area includes the spatial range of the coordinate points (a3, b3, c3) to (a5, b5, c5) on the surface of the coronary artery. For this contact area, the tissue deformation rate at each timestamp is used to calculate the deformation acceleration distribution. Assume that the deformation rates at timestamps t1, t2, and t3 are r1, r2, and r3 respectively. Calculate the change in deformation rate from t1 to t2 as r2 - r1, and the change in deformation rate from t2 to t3 as r3 - r2, and then divide by the corresponding time interval (such as 1 second) to obtain the change rate of the deformation rate in this time period for the contact area, which is the deformation acceleration. Such calculations are performed at each position point in the entire contact area to obtain the deformation acceleration distribution of the contact area in the instrument movement direction.

[0104] Step S1335: Determine the stress direction vector of each sub-region within the contact area according to the spatial angle between the deformation acceleration distribution and the instrument movement direction, and fuse the stress direction vector with the spatial gradient of the tissue deformation amplitude to generate a dynamic stress propagation path.

[0105] For each sub-region within the contact area, for example, dividing the contact area into multiple small sub-regions according to certain rules, the stress direction vector is determined based on the spatial angle between the deformation acceleration distribution calculated previously and the direction of the instrument movement. Suppose in a certain sub-region, the deformation acceleration distribution vector is A and the instrument movement direction vector is V. Through specific vector operations (here, instead of using formulas, the operation logic is described in words: first calculate the angle θ between A and V, and determine the magnitude and direction of the stress direction vector S based on the angle and the magnitude of A. For example, if θ is small, the direction of S is relatively close to the direction of A and its magnitude has a certain proportional relationship with the magnitude of A), the stress direction vector is obtained. At the same time, calculate the rate of change of the tissue deformation amplitude in space, that is, the spatial gradient of the tissue deformation amplitude. The stress direction vector and the spatial gradient of the tissue deformation amplitude are fused. For example, in each sub-region, the stress direction vector and the spatial gradient of the tissue deformation amplitude are added according to a certain weight (this weight is determined in advance according to the actual physical relationship and surgical situation), obtaining a comprehensive vector. According to the direction and magnitude of this comprehensive vector in different sub-regions, the propagation direction and trend of stress in the tissue can be determined, thereby generating a dynamic stress propagation path.

[0106] Step S1336, generate the stress distribution map according to the coverage range and direction consistency of the dynamic stress propagation path at consecutive time stamps, where different color regions in the stress distribution map represent the stress accumulation intensity levels.

[0107] At consecutive time stamps, observe the coverage range of the dynamic stress propagation path, that is, which regions the stress propagates to and the propagation degree within each region. At the same time, analyze the direction consistency of the stress propagation path. For example, in some regions, the stress propagation direction remains basically unchanged, while in other regions, there may be significant direction changes. Based on this information, the entire contact area is divided into different parts to quantify the stress accumulation intensity. For example, regions with higher stress accumulation intensity are represented by red, medium intensity regions by yellow, lower intensity regions by green, etc. In this way, a stress distribution map is generated, which can intuitively display the stress distribution in the contact area between the vascular clamp and the coronary artery during coronary artery bypass surgery, including the magnitude, direction of the stress, and its change trend over time, etc., providing an important basis for the safety assessment and operation adjustment of the surgery.

[0108] In a possible implementation manner, step S134 includes:

[0109] Step S1341: Extract the stress cumulative intensity level of the color-coded area in the stress distribution map. Mark the continuous area with a color depth exceeding the preset intensity level as the high-stress core area. Divide grid cells within the high-stress core area, and count the consistency ratio of stress direction vectors in each grid cell. Merge the grid cells with a consistency ratio exceeding the preset threshold into co-directional stress bands.

[0110] In coronary artery bypass grafting surgery, the stress distribution map uses different colors to represent the stress cumulative intensity level. For example, red represents high intensity, yellow represents medium intensity, and green represents low intensity. From this stress distribution map, set the preset intensity level. For instance, consider the red area as the area with a color depth exceeding the preset intensity level, and these continuous red areas are marked as the high-stress core area. Divide grid cells within the high-stress core area. Assume the grid cells are divided into square grid cells with a side length of 1 millimeter. For each grid cell, count the consistency ratio of stress direction vectors within it. The stress direction vector is determined based on factors such as stress acceleration distribution before, representing the direction of stress. For example, if there are 10 stress direction vector sample points in a grid cell and 8 of them have basically the same direction, then the consistency ratio of the stress direction vectors in this grid cell is 80%. Set the preset threshold to 70%, and merge the grid cells with a consistency ratio exceeding 70% into co-directional stress bands.

[0111] Step S1342: Generate an initial estimate of the instrument operation force at each time stamp based on the product of the coverage area of the co-directional stress band and the stress cumulative intensity level. According to the instrument type in the surgical instrument recognition result, match the predefined instrument rigidity coefficient, and perform weighted correction on the initial estimate of the instrument operation force with the rigidity coefficient to generate the instrument operation force.

[0112] In coronary artery bypass grafting surgery, for the co-directional stress band, calculate its coverage area. Assume a co-directional stress band contains 5 grid cells, and the area of each grid cell is 1 square millimeter, then the coverage area is 5 square millimeters. If the stress cumulative intensity level in the area where this co-directional stress band is located is high intensity (set the corresponding value for high intensity as 3), then the initial estimate of the instrument operation force is 5×3 = 15 (this is just a calculation example based on the set rules). According to the surgical instrument recognition result, if the surgical instrument is a vascular forceps, look up the rigidity coefficient of the vascular forceps in the predefined instrument rigidity coefficient library, which is 1.2. Perform weighted correction on the initial estimate of the instrument operation force 15 with the rigidity coefficient 1.2. For example, the weighting method is multiplication (the actual weighting method is determined based on physical principles and experience), and the obtained instrument operation force is 15×1.2 = 18. 18 is the quantitative index of the instrument operation force at this time stamp.

[0113] Step S1343, extracting the deformation recovery rate corresponding to the tissue deformation parameter in the biological tissue segmentation result, calculating the attenuation ratio of the deformation recovery rate and the stress cumulative intensity level in the same area in the stress distribution map, and determining the dynamic estimation of the tissue response elastic modulus based on the attenuation ratio and a predefined tissue elastic attenuation curve.

[0114] In heart bypass surgery, the deformation recovery rate corresponding to the tissue deformation parameters of the coronary artery is obtained from the biological tissue segmentation results. Assume that in a certain area, the deformation recovery rate is 0.2 mm / s. In the stress distribution map, the stress accumulation intensity level of the corresponding area is medium intensity (the corresponding value of medium intensity is set to 2), and the calculated attenuation ratio of the deformation recovery rate and the stress accumulation intensity level is 0.2÷2 = 0.1. According to the predefined tissue elastic attenuation curve, this curve is derived based on a large number of experiments and theoretical derivations. By finding the point corresponding to the attenuation ratio of 0.1 in the curve, the dynamic estimate of the tissue response elastic modulus is obtained. For example, the corresponding elastic modulus value found on the curve is 80 MPa, and this 80 MPa is the dynamic estimate of the tissue response elastic modulus in this area.

[0115] Step S1344, based on the duration of the high stress core area at each time stamp in the stress distribution map, the proportion of the high stress core area in the total surgical phase duration is counted to generate an initial value of the contact time proportion, and according to the deformation excess risk index of the tissue area boundary in the biological tissue segmentation result, the credibility weight of the initial value of the contact time proportion is adjusted to generate the contact time proportion.

[0116] During the coronary bypass surgery, the duration of the high stress core area at each time stamp in the stress distribution map is counted. Assuming that the total duration of the operation phase is 60 minutes, and the high stress core area lasts for 10 minutes in a certain period of time, the initial value of the contact time ratio is 10÷60≈0.167. The deformation over-limit risk index of the coronary artery tissue region boundary is obtained from the biological tissue segmentation result, and it is assumed that the deformation over-limit risk index is 0.3 (the deformation over-limit risk index is previously calculated based on factors such as tissue deformation amplitude). If the deformation over-limit risk index is high, it means that the tissue may be in an unstable state, which will reduce the credibility weight of the initial value of the contact time ratio. Assuming that the weight adjustment coefficient determined according to the deformation over-limit risk index is 0.8 (the weight adjustment coefficient is predetermined based on the relationship between the risk index and the contact time ratio), the adjusted contact time ratio is 0.167×0.8 = 0.134.

[0117] Step S1345: Align the instrument operation force, tissue response elastic modulus, and contact time ratio according to the timestamp to generate a quantitative index time series table. Then, based on the fluctuation frequencies of the various indices in the quantitative index time series table, match the predefined surgical operation mode feature library to generate an abnormal fluctuation range of the instrument operation force, a stable threshold of the tissue response elastic modulus, and a safe range of the contact time ratio.

[0118] In the context of a coronary artery bypass grafting (CABG) surgery scenario, arrange the instrument operation force, tissue response elastic modulus, and contact time ratio at each timestamp in chronological order to form a quantitative index time series table. For example, at timestamp t1, the instrument operation force is 18, the tissue response elastic modulus is 80 MPa, and the contact time ratio is 0.134; there are corresponding values at timestamp t2, etc. Analyze the fluctuation frequencies of the various indices in this time series table. For the instrument operation force, count the fluctuation of its values within a certain time range (such as within 10 timestamps). If the fluctuation amplitude is large and frequent, determine its fluctuation frequency. According to the predefined surgical operation mode feature library, which contains information such as the fluctuation ranges of the various indices under normal CABG surgery operations. By matching this surgical operation mode feature library, determine the abnormal fluctuation range of the instrument operation force. For example, if the normal fluctuation range of the instrument operation force is set to 15 - 20, then the abnormal fluctuation range may be less than 15 or greater than 20. For the tissue response elastic modulus, determine its stable threshold. For example, under normal circumstances, the tissue response elastic modulus should be between 70 - 90 MPa, and this 70 - 90 MPa is the stable threshold. For the contact time ratio, determine its safe range. For example, the range between 0.1 - 0.2 is the safe range.

[0119] Step S1346: Based on the abnormal fluctuation range, stable threshold, and safe range, perform segmented annotation on the quantitative index time series table to generate a set of quantitative indices with risk level markings.

[0120] During coronary artery bypass grafting surgery, the quantitative index time series table is segmented and labeled according to the previously determined abnormal fluctuation range of the instrument operation force, the stable threshold of the tissue response elastic modulus, and the safety range of the contact time ratio. If, within a certain time period, the instrument operation force exceeds the abnormal fluctuation range, the tissue response elastic modulus exceeds the stable threshold, or the contact time ratio exceeds the safety range, different risk levels are marked according to the degree of exceeding and the potential risk impact on the surgery. For example, if the instrument operation force exceeds the abnormal fluctuation range by a large margin, the tissue response elastic modulus also deviates significantly from the stable threshold, and the contact time ratio exceeds the safety range by a large margin, it is marked as a high-risk level; if only one of the indicators slightly exceeds the corresponding range, it is marked as a low-risk level. In this way, a set of quantitative indicators with risk level markings is generated, which can intuitively reflect the operation risk situation at each stage during coronary artery bypass grafting surgery and provide an important basis for subsequent surgical monitoring and decision-making.

[0121] In a possible implementation manner, step S140 includes:

[0122] Step S141, extracting the time series feature sequences of the instrument operation force, tissue deformation rate, and stress distribution map from the dynamic operation state parameters.

[0123] During coronary artery bypass grafting surgery, relevant data is obtained from the previously generated dynamic operation state parameters. For the instrument operation force, the operation force values at each time point are recorded in chronological order to form a time series feature sequence. For example, at different time points after the start of the surgery, such as the operation force is 15 units at 1 minute, 18 units at 2 minutes, 20 units at 3 minutes, etc., and arranging them in this order constitutes the time series feature sequence of the instrument operation force. Similarly, for the tissue deformation rate, the deformation rate of the coronary artery at each time point is recorded, such as the deformation rate is 0.1 mm / min at the beginning and then becomes 0.15 mm / min, etc., to form the time series feature sequence of the tissue deformation rate. For the stress distribution map, according to the relevant features of the stress distribution at each time point, such as the stress accumulation intensity level, the size of the high-stress core area, etc., the time series feature sequence of the stress distribution map is constructed.

[0124] Step S142, according to the predefined safety threshold of the current surgical stage, dividing the time series feature sequence into time window segments matching the current surgical stage.

[0125] During coronary artery bypass grafting, the entire surgical process is divided into different stages, such as the blood vessel dissection stage, the vascular anastomosis stage, etc. Each stage has corresponding predefined safety thresholds. Suppose the current stage is the vascular anastomosis stage, and the predefined time window for this vascular anastomosis stage is 3 minutes. Then, divide the above-mentioned time series feature sequence into segments of 3 minutes each. For example, from 0 - 3 minutes after the start of the operation is the first time window segment, 3 - 6 minutes is the second time window segment, and so on.

[0126] Step S143: Perform a difference operation between the average value of the instrument operation force within each time window segment and the upper limit value of the force in the surgical stage safety threshold to generate a force deviation coefficient.

[0127] For example, during the vascular anastomosis stage, assume that the upper limit value of the instrument operation force in the safety threshold is 20 units. For the first time window segment (0 - 3 minutes), calculate the average value of the instrument operation force within this time window segment. For example, if the operation forces within this time window segment are 18 units, 19 units, and 20 units respectively, then the average value is (18 + 19 + 20)÷3 = 19 units. Then perform the difference operation, and the force deviation coefficient is (19 - 20)÷20 = - 0.05 (here, calculating the difference and then dividing by the upper limit value is to obtain the relative deviation degree).

[0128] Step S144: Synchronously compare the peak value of the tissue deformation rate within the time window segment with the deformation rate threshold in the surgical stage safety threshold to generate a rate over - standard flag bit.

[0129] For example, during the vascular anastomosis stage, assume that the deformation rate threshold is 0.2 mm / min. Within the first time window segment (0 - 3 minutes), check the maximum value of the tissue deformation rate. If the maximum deformation rate is 0.18 mm / min, since 0.18 is less than 0.2, the rate over - standard flag bit is 0, indicating that it is not over - standard; if the maximum deformation rate is 0.25 mm / min, since 0.25 is greater than 0.2, the rate over - standard flag bit is 1, indicating that it is over - standard.

[0130] Step S145: Calculate the overlap rate between the covered area of the stress distribution map and the contact area threshold in the surgical stage safety threshold to generate an area anomaly proportion value.

[0131] For example, during the blood vessel anastomosis stage, assume the contact area threshold is 10 square millimeters. In the stress distribution map corresponding to the first time window segment, the covered area of the stress distribution map is 12 square millimeters. Calculate the overlap rate (here, actually calculate the proportion of the excess part), and the area anomaly proportion value is (12 - 10) ÷ 10 = 0.2, indicating that the covered area of the stress distribution map exceeds the contact area threshold by 0.2.

[0132] Step S146: Input the force deviation coefficient, rate over - standard flag bit, and area anomaly proportion value into a pre - trained multi - level decision tree model. Among them, the first - level decision node determines whether the force deviation coefficient exceeds the first risk critical value; the second - level decision node generates a tissue cumulative damage index according to the triggering times of the rate over - standard flag bit; the third - level decision node generates a contact anomaly pattern based on the area anomaly proportion value and the deformation spatial distribution characteristics.

[0133] Assume the first risk critical value is 0.1, and the force deviation coefficient calculated previously is - 0.05. Since - 0.05 is less than 0.1, the triggering condition is not met at the first - level decision node. The second - level decision node generates a tissue cumulative damage index according to the triggering times of the rate over - standard flag bit. If during the entire surgical process, the rate over - standard flag bit is triggered multiple times (for example, the rate over - standard occurs multiple times in different time window segments), assume the tissue cumulative damage index increases by 0.1 each time it exceeds the standard. If there have been 2 previous over - standards and it exceeds the standard again in the current time window segment, then the tissue cumulative damage index becomes 0.3. The third - level decision node generates a contact anomaly pattern based on the area anomaly proportion value and the deformation spatial distribution characteristics. For example, according to the area anomaly proportion value of 0.2 and the spatial distribution of the stress in the stress distribution map (such as the high - stress core area is concentrated on one side of the coronary artery and other spatial distribution characteristics), it is determined that the contact anomaly pattern is excessive local contact pressure.

[0134] Step S147: Based on the intermediate judgment result output by the multi - level decision tree model, activate the corresponding abnormal event type label. The abnormal event type label at least includes three categories: instrument over - pressure, tissue over - speed deformation, and abnormal contact area.

[0135] For example, since in the previous judgment, it is found that there is a situation where the contact anomaly pattern is excessive local contact pressure, the abnormal event type label of abnormal contact area is activated.

[0136] Step S148: Real - time obtain the change amount of the regional boundary pixels in the biological tissue segmentation result. When detecting the continuous expansion or contraction form of the boundary pixels, generate a tissue integrity damage signal.

[0137] For example, in coronary artery bypass grafting surgery, continuously monitor the change in the number of pixels at the regional boundary of the coronary artery from the biological tissue segmentation results. If, within a certain time period, it is found that the regional boundary pixels of the coronary artery continuously contract, for example, the number of lower boundary pixels continuously decreases at several consecutive time points, this indicates that the coronary artery may be overly compressed, thereby generating a tissue integrity damage signal.

[0138] Step S149: Perform a logical AND operation on the tissue integrity damage signal and the activated abnormal event type label. If there is an overlapping time interval between the two in terms of time stamp, increase the risk level of the corresponding abnormal event type by one level.

[0139] For example, assume that the abnormal event type label of abnormal contact area has been activated, and within a certain time interval, the time period corresponding to this abnormal event type label overlaps with the time period when the tissue integrity damage signal is generated (for example, both are within the time period of 5 - 8 minutes). Then, increase the risk level of the abnormal event type of abnormal contact area by one level.

[0140] Step S1410: Bind the final abnormal event type to the increased risk level according to the level mapping rule set in the surgical stage safety threshold.

[0141] For example, in the setting of the safety threshold for coronary artery bypass grafting surgery, for the vascular anastomosis stage, there are specific level mapping rules. For example, risk level 1 indicates that slight attention is required, risk level 2 indicates that the operation needs to be adjusted, risk level 3 indicates that emergency treatment is required, etc. If the risk level of the abnormal event type of abnormal contact area is increased to level 2 after the previous judgment and processing, then bind the abnormal contact area to risk level 2 according to the level mapping rule, so as to clearly take corresponding measures for this abnormal situation in subsequent surgical monitoring and operation adjustment.

[0142] In a possible implementation manner, step S150 includes:

[0143] Step S151: Locate the abnormal area in the real-time image sequence according to the coordinates of the contact point between the tissue area boundary corresponding to the abnormal event type and the instrument, and generate a color-coded contour line and a text label that match the risk level.

[0144] In this embodiment, during a coronary artery bypass grafting surgery, if the abnormal event type is an abnormal contact area, the corresponding tissue area boundary (such as the boundary coordinates of a specific part of the coronary artery) and the instrument contact point coordinates (such as the contact point coordinates between a vascular clamp and the coronary artery) are obtained from the previous biological tissue segmentation results and surgical instrument recognition results. In the real-time image sequence, the abnormal area is accurately located based on these coordinates. For different risk levels, different color-coded contour lines are set. For example, when the risk level is level 1, the abnormal area is marked with a yellow color-coded contour line, indicating that attention needs to be paid; when the risk level is level 2, an orange contour line is used, meaning the situation is relatively serious and requires close attention; when the risk level is level 3, a red contour line is used, indicating an emergency. At the same time, corresponding text labels are generated, such as "Abnormal Contact Area - Risk Level 2", clearly indicating the type and severity of the abnormality. The text labels are placed near the abnormal area for quick identification by the surgical staff.

[0145] Step S152, generate multi-level voice alarm signals based on the alarm priorities corresponding to the risk levels.

[0146] For example, in the monitoring of coronary artery bypass grafting surgery, the risk level is associated with the alarm priority. For a risk level of 1, the alarm priority is relatively low, and a relatively gentle voice alarm signal may be generated, such as a single "ding" prompt sound and a voice broadcast of "Abnormal contact area, please pay attention". When the risk level is 2, the alarm priority increases, and the voice alarm signal becomes a more rapid "beep-beep" sound, while the voice broadcast is "Abnormal contact area, the situation is relatively serious, please pay close attention". When the risk level is 3, the alarm priority is the highest, and a continuous and rapid "beep-beep-beep" alarm sound is emitted, with a voice broadcast of "Abnormal contact area, emergency situation, please handle immediately". This multi-level voice alarm signal can timely remind the surgical staff according to the severity of the risk.

[0147] Step S153, match the instrument path adjustment rules and force optimization parameters in the predefined surgical operation correction plan library according to the abnormal event type. Among them, for the instrument overpressure type, a reverse displacement vector arrow and a pressure decreasing curve are generated; for the tissue over-speed deformation type, a regional isolation identifier and an instrument hovering instruction are generated; for the abnormal contact area type, a contact point migration path and an angle calibration guide are generated.

[0148] For example, if the abnormal event type is instrument overpressure, the corresponding instrument path adjustment rule and force optimization parameter are found in the predefined surgical operation correction plan library. The instrument path adjustment rule may generate a reverse displacement vector arrow based on the contact point between the vascular forceps and the coronary artery and the overpressure direction. For example, if the vascular forceps apply excessive pressure to the coronary artery in a certain direction, the reverse displacement vector arrow indicates that the vascular forceps should move a certain distance in the opposite direction. At the same time, a pressure decreasing curve is generated, showing how the pressure should gradually decrease to a safe range over time or the moving distance if the operation is performed correctly. For the type of tissue over-speed deformation, a regional isolation identifier and an instrument hovering instruction are generated from the correction plan library. The regional isolation identifier is used to mark the tissue area where over-speed deformation occurs on the image, reminding the surgical staff to avoid continuing the operation in this area; the instrument hovering instruction tells the surgical staff to stop the current operation of the instrument to prevent further damage to the tissue. If the abnormal event type is the abnormal contact area type, a contact point migration path and an angle calibration guide are generated from the correction plan library. The contact point migration path indicates that the contact point of instruments such as vascular forceps should move to a more appropriate position, and the angle calibration guide gives the correct angle range of the instrument relative to the tissue to ensure normal surgical operation.

[0149] Step S154, synchronously transmit the color-coded contour line, text label, voice alarm signal, and correction guide parameter to the display terminal of the surgical navigation system, and divide the first display area on the terminal interface to render the original image stream in real time, the second display area to overlay the abnormal contour and correction guide layer, and the third display area to scroll and update the risk level and voice-to-text log.

[0150] On the display terminal of the coronary artery bypass grafting surgical navigation system, the first display area focuses on rendering the original image stream in real time, fully displaying the surgical scene, including the real-time operation images of instruments such as the heart, coronary artery, and vascular forceps, enabling the surgical staff to intuitively see the actual progress of the operation. The second display area overlays correction guide layers such as the abnormal area marked by the color-coded contour line, reverse displacement vector arrow, regional isolation identifier, and contact point migration path on the basis of the original image. The surgical staff can directly see the location of the abnormal situation and the corresponding correction suggestions in this area. The third display area scrolls and updates the risk level, such as timely displaying when it is updated from level 1 to level 2, and at the same time converts the voice alarm signal into a text form and records it to form a log, which is convenient for the surgical staff to review and check the risk changes during the operation.

[0151] Step S155, according to the dynamic changes of the surgical stage progress and risk level, real-time update the contour line color intensity and correction guide parameter in the second display area. When it is detected that the risk level of the same abnormal event type increases, switch the contour line of the corresponding area to a higher-priority color and enlarge the display size of the correction guide arrow.

[0152] During a coronary artery bypass grafting surgery, as the surgical stage progresses, for example, from the vascular dissection stage to the vascular anastomosis stage, the risk level may change. If, during the vascular anastomosis stage, a situation that was previously marked as an abnormal contact area with a risk level of 2 is elevated to a risk level of 3 due to certain operations. At this time, in the second display area, the orange contour line corresponding to the abnormal contact area will switch to a red contour line, and the color intensity will increase to more prominently display the emergency situation. At the same time, if it is of the instrument overpressure type and there is a reverse displacement vector arrow, the display size of the vector arrow will be enlarged to make it more prominent, enabling the surgical staff to more easily notice the change in the correction guidance, so as to timely adjust the surgical operation and ensure the safety and effectiveness of the surgery.

[0153] Figure 2 FIG. shows a schematic diagram of exemplary hardware and software components of a machine vision-based surgical process monitoring system 100 provided by some embodiments of the present application that can implement the idea of the present application. For example, the processor 120 can be used on the machine vision-based surgical process monitoring system 100 and is used to execute the functions in the present application.

[0154] The machine vision-based surgical process monitoring system 100 can be a general-purpose server or a special-purpose server, both of which can be used to implement the machine vision-based surgical process monitoring method of the present application. Although only one server is shown in the present application, for convenience, the functions described in the present application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0155] For example, the machine vision-based surgical process monitoring system 100 can include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and different forms of storage media 140, such as disks, ROM, or RAM, or any combination thereof. Exemplarily, the machine vision-based surgical process monitoring system 100 can also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application can be implemented according to these program instructions. The machine vision-based surgical process monitoring system 100 also includes an input / output (I / O) interface 150 between the computer and other input / output devices.

[0156] For ease of explanation, only one processor is described in the machine vision-based surgical procedure monitoring system 100. However, it should be noted that the machine vision-based surgical procedure monitoring system 100 in the present application may also include multiple processors. Therefore, the steps performed by one processor described in the present application may also be jointly performed or separately performed by multiple processors. For example, if the processor of the machine vision-based surgical procedure monitoring system 100 performs step A and step B, it should be understood that step A and step B may also be jointly performed by two different processors or separately performed in one processor. For example, the first processor performs step A, the second processor performs step B, or the first processor and the second processor jointly perform steps A and B.

[0157] In addition, an embodiment of the present invention further provides a readable storage medium, in which computer-executable instructions are preset. When the processor executes the computer-executable instructions, the above-mentioned machine vision-based surgical procedure monitoring method is implemented.

[0158] It should be noted that, in order to simplify the description of the present invention disclosure and thus help the understanding of one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, sometimes multiple features are merged into one embodiment, drawing, or description thereof.

Claims

1. A surgical process monitoring method based on machine vision, characterized in that: The method comprises: Acquire a real-time image sequence of a surgical scene, wherein the real-time image sequence includes dynamic images of a surgical instrument operation area and a biological tissue area; Extracting surgical instrument recognition results and biological tissue segmentation results from the real-time image sequence, wherein the surgical instrument recognition results include instrument type and spatial posture, and the biological tissue segmentation results include tissue type and region boundary; Based on the surgical instrument recognition result and the biological tissue segmentation result, monitoring the operation trajectory of the surgical instrument and the deformation parameters of the biological tissue area, and generating dynamic operation state parameters; According to the predefined safety threshold of the surgical stage, multi-modal abnormality detection is performed on the dynamic operation state parameters to determine the abnormal event type and risk level; Based on the abnormal event type and risk level, a visual surgery monitoring report is generated, wherein the visual surgery monitoring report includes abnormal area marking, risk warning information and correction suggestions; The step of monitoring the operation trajectory of the surgical instrument and the deformation parameters of the biological tissue region based on the surgical instrument recognition result and the biological tissue segmentation result to generate dynamic operation state parameters includes: According to the spatial posture in the surgical instrument recognition result, the moving direction and speed of the instrument under continuous time stamps are calculated to generate the operation trajectory; Based on the regional boundaries in the biological tissue segmentation result, analyzing the tissue deformation amplitude and deformation rate between adjacent timestamps to generate the deformation parameters; The operation trajectory and the deformation parameter are integrated to calculate a stress distribution map of the contact area between the surgical instrument and the tissue; Generating quantitative indicators of instrument operation force, tissue response elastic modulus and contact time ratio according to the stress distribution map; Matching the quantitative index with a preset surgical stage model and outputting the dynamic operation state parameter; The step of fusing the operation trajectory and the deformation parameter to calculate a stress distribution map of a contact area between the surgical instrument and the tissue includes: Extracting the three-dimensional spatial coordinate sequence corresponding to the tip of the instrument at each time stamp according to the moving direction and speed of the instrument at the continuous time stamps in the operation trajectory; Based on the tissue deformation amplitude and deformation rate between adjacent time stamps in the deformation parameters, extracting the deformation region boundary coordinate sequence of the tissue surface at the corresponding time stamps; Performing time synchronization to align the three-dimensional spatial coordinate sequence of the instrument tip with the coordinate sequence of the boundary of the deformation region, and generating a mapping relationship between the synchronized instrument contact point position and the tissue deformation region; According to the synchronized instrument contact point position, locate the contact area space range between the instrument tip and the tissue surface in the deformation area boundary coordinate sequence, and calculate the deformation acceleration distribution of the contact area in the instrument movement direction based on the tissue deformation rate at each time stamp within the contact area space range; Determine the stress direction vector of each sub-area within the contact area according to the spatial angle between the deformation acceleration distribution and the moving direction of the instrument, fuse the stress direction vector with the spatial gradient of the tissue deformation amplitude, and generate a dynamic stress propagation path; The stress distribution map is generated according to the coverage and direction consistency of the dynamic stress propagation path under continuous time stamps, wherein different color areas in the stress distribution map represent stress accumulation intensity levels.

2. The method for monitoring a surgical process based on machine vision according to claim 1, characterized in that: The step of extracting surgical instrument recognition results and biological tissue segmentation results from the real-time image sequence includes: Performing key frame sampling on the real-time image sequence to obtain multiple groups of surgical scene static images; Extracting feature regions from each group of static images of the surgical scene to obtain candidate frames of instrument operation regions and candidate frames of tissue regions; Performing multi-scale feature matching on the candidate frame of the instrument operation area based on a three-dimensional convolutional network, determining the instrument type and spatial posture, and generating the surgical instrument recognition result; Performing pixel-level classification on the tissue region candidate frame through a semantic segmentation model to obtain the tissue type and region boundary, and generating the biological tissue segmentation result; The surgical instrument recognition result and the biological tissue segmentation result are aligned in time and space to form a unified spatial coordinate system mapping relationship.

3. The method for monitoring a surgical procedure based on machine vision according to claim 2, characterized in that: The feature region extraction is performed on each group of the surgical scene static images to obtain the instrument operation region candidate frame and the tissue region candidate frame, including: Performing grayscale equalization processing on the static image of the surgical scene to enhance the contrast difference between the instrument operation area and the biological tissue area; The pixel area whose gradient amplitude exceeds a preset threshold in the image after grayscale equalization processing is extracted by an edge detection algorithm to generate an initial candidate frame set; Executing a region growing algorithm on the initial candidate frame set, merging candidate frames with overlapping spatial positions and similar texture features, and generating a merged candidate frame set; According to a preset instrument size range and tissue size range, candidate frames that meet corresponding size constraints are selected from the merged candidate frame set to obtain a preliminarily selected candidate frame set; The morphological closing operation is performed on the candidate frame set after the preliminary screening to fill the broken area inside the candidate frame and smooth the jagged edges of the candidate frame to generate the final instrument operation area candidate frame and biological tissue area candidate frame.

4. The method for monitoring a surgical process based on machine vision according to claim 1, characterized in that: The step of analyzing the deformation amplitude and deformation rate of the tissue between adjacent timestamps based on the regional boundaries in the biological tissue segmentation result to generate the deformation parameters includes: Extracting a tissue region boundary coordinate sequence corresponding to adjacent timestamps in the biological tissue segmentation result, matching the boundary coordinates of a previous timestamp with the boundary coordinates of a subsequent timestamp point by point, and generating a displacement vector for each coordinate point; Calculating the overall displacement mean of the tissue region boundary between adjacent time stamps according to the length of the displacement vector to generate the tissue deformation amplitude; Based on the rate of change of the displacement vector at continuous timestamps, the displacement acceleration of each coordinate point is calculated, the coordinate point whose acceleration exceeds a preset acceleration threshold is marked as a deformation mutation point, the distribution density of the deformation mutation point within the boundary of the tissue region is counted, and the local peak value of the tissue deformation rate is generated according to the ratio of the distribution density to the acceleration threshold; Performing normalized weighted fusion on the tissue deformation amplitude, deformation mutation point distribution density and local peak value of tissue deformation rate to generate a deformation parameter set; According to the tissue type in the biological tissue segmentation result, a predefined tissue elastic property library is matched to extract the deformation tolerance coefficient of the corresponding tissue type; The tissue deformation amplitude in the deformation parameter set is proportionally calculated with the deformation tolerance coefficient to generate a deformation over-limit risk index, and according to the temporal variation trend of the deformation over-limit risk index, the exponential increments of adjacent timestamps are compared with a predefined stage deformation increment threshold to generate a deformation parameter dynamic abnormality flag; The deformation parameter set, deformation over-limit risk index and deformation parameter dynamic anomaly flag are spatially superimposed to form a deformation parameter thermodynamic layer with a timestamp mark, and based on the parameter change direction of continuous timestamps in the deformation parameter thermodynamic layer, a tissue deformation propagation path prediction trajectory is generated; According to the spatial distance between the deformation propagation path prediction trajectory and the surgical instrument operation trajectory, the dynamic weight coefficient of the deformation parameter is adjusted to generate a final deformation parameter set.

5. The method for monitoring a surgical process based on machine vision according to claim 1, characterized in that: Generating quantitative indicators of instrument operation force, tissue response elastic modulus and contact time ratio according to the stress distribution map includes: Extracting the stress cumulative intensity level of the color-coded area in the stress distribution map, marking the continuous area whose color depth exceeds the preset intensity level as a high stress core area, dividing the grid units in the high stress core area, counting the consistency ratio of the stress direction vector in each grid unit, and merging the grid units whose consistency ratio exceeds the preset threshold value into isotropic stress belts; Based on the product of the coverage area of ​​the isotropic stress band and the stress cumulative intensity level, an initial estimate of the instrument operation force at each time stamp is generated, and according to the instrument type in the surgical instrument recognition result, a predefined instrument rigidity coefficient is matched, and the initial estimate of the instrument operation force and the rigidity coefficient are weightedly corrected to generate the instrument operation force; Extracting the deformation recovery rate corresponding to the tissue deformation parameter in the biological tissue segmentation result, calculating the attenuation ratio of the deformation recovery rate and the stress cumulative intensity level in the same area in the stress distribution map, and determining the dynamic estimation of the tissue response elastic modulus according to the attenuation ratio and a predefined tissue elastic attenuation curve; Based on the duration of the high stress core area at each time stamp in the stress distribution map, the proportion of the high stress core area in the total duration of the operation phase is counted to generate an initial value of the contact time proportion, and according to the deformation over-limit risk index of the tissue region boundary in the biological tissue segmentation result, the credibility weight of the initial value of the contact time proportion is adjusted to generate the contact time proportion; The instrument operation force, tissue response elastic modulus and contact time ratio are aligned according to timestamps to generate a quantitative indicator time series table, and according to the fluctuation frequency of each indicator in the quantitative indicator time series table, a predefined surgical operation mode feature library is matched to generate an abnormal fluctuation range of the instrument operation force, a stable threshold of the tissue response elastic modulus and a safe range of the contact time ratio; Based on the abnormal fluctuation interval, stability threshold and safety range, the quantitative indicator time series table is segmented and marked to generate a set of quantitative indicators with risk level marks.

6. The method for monitoring a surgical process based on machine vision according to claim 1, characterized in that: The method of performing multi-modal abnormality detection on the dynamic operation state parameters according to the predefined safety threshold of the surgical stage to determine the abnormal event type and risk level includes: Extracting the time series characteristic sequence of the instrument operation force, tissue deformation rate and stress distribution spectrum from the dynamic operation state parameters; According to the predefined surgical stage safety threshold, the temporal feature sequence is segmented into time window segments matching the current surgical stage; Performing a difference operation on the average value of the instrument operation force in each time window segment and the force upper limit value in the safety threshold of the surgical stage to generate a force deviation coefficient; Synchronously comparing the peak value of the tissue deformation rate within the time window segment with the deformation rate threshold in the surgical stage safety threshold to generate a rate exceeding mark; Calculate the overlap rate of the coverage area of ​​the stress distribution map and the contact area threshold in the surgical stage safety threshold to generate an area abnormality ratio value; The force deviation coefficient, the rate exceeding flag and the area abnormality ratio are input into a pre-trained multi-level decision tree model, wherein the first-level decision node determines whether the force deviation coefficient exceeds the first risk critical value, the second-level decision node generates a tissue cumulative damage index according to the number of triggering times of the rate exceeding flag, and the third-level decision node generates a contact abnormality pattern based on the area abnormality ratio and the deformation spatial distribution characteristics; Based on the intermediate judgment results output by the multi-level decision tree model, the corresponding abnormal event type labels are activated, and the abnormal event type labels include at least three types: instrument overpressure, tissue overspeed deformation and abnormal contact area; Acquiring in real time the amount of change in the region boundary pixels in the biological tissue segmentation result, and generating a tissue integrity destruction signal when a continuous expansion or contraction of the boundary pixels is detected; Performing a logical AND operation on the tissue integrity destruction signal and the activated abnormal event type tag, if there is an overlapping interval between the two in terms of timestamps, the risk level of the corresponding abnormal event type is increased by one level; According to the level mapping rules set in the safety threshold of the surgical stage, the final abnormal event type is bound to the enhanced risk level.

7. The method for monitoring a surgical process based on machine vision according to claim 1, characterized in that: The generating of a visual surgery monitoring report based on the abnormal event type and risk level includes: Locating the abnormal area in the real-time image sequence according to the tissue area boundary and the coordinates of the instrument contact point corresponding to the abnormal event type, and generating a color-coded contour line and text label matching the risk level; Generate a multi-level voice alarm signal based on the alarm priority corresponding to the risk level; Match the instrument path adjustment rules and force optimization parameters in the predefined surgical operation correction solution library according to the abnormal event type, wherein the instrument overpressure type generates a reverse displacement vector arrow and a pressure reduction curve, the tissue overspeed deformation type generates a regional isolation mark and an instrument hovering instruction, and the abnormal contact area type generates a contact point migration path and an angle calibration guide; The color-coded contour lines, text labels, voice alarm signals and correction guidance parameters are synchronously transmitted to a display terminal of a surgical navigation system, and the terminal interface is divided into a first display area for real-time rendering of the original image stream, a second display area for superimposing abnormal contours and correction guidance layers, and a third display area for rolling updates of risk levels and voice-to-text logs; According to the dynamic changes in the progress of the surgical stage and the risk level, the contour line color intensity and correction guidance parameters in the second display area are updated in real time. When the risk level of the same abnormal event type is detected to be increased, the contour line of the corresponding area is switched to a higher priority color and the display size of the correction guidance arrow is enlarged.

8. A surgical process monitoring system based on machine vision, characterized in that: The machine vision-based surgical process monitoring system includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the machine vision-based surgical process monitoring method described in any one of claims 1 to 7.

Images