Operation process monitoring method and system based on machine vision

Abstract

The invention provides a surgical process monitoring method and system based on machine vision, and the method comprises the steps: firstly obtaining a surgical scene real-time image sequence containing the dynamic images of a surgical instrument operation region and a biological tissue region; secondly, extracting a surgical instrument identification result (including an instrument type and a space posture) and a biological tissue segmentation result (including a tissue type and a region boundary), then monitoring an instrument operation track and a biological tissue region deformation parameter based on the results, generating a dynamic operation state parameter, and according to a predefined surgical stage safety threshold value, determining a biological tissue region deformation parameter; multi-modal anomaly detection is carried out on the dynamic operation state parameters, an abnormal event type and a risk level are determined, finally, a visual operation monitoring report is generated according to the abnormal event type and the risk level, the report covers abnormal area marks, risk prompt information and correction suggestions, comprehensive intelligent monitoring of the operation process is achieved, and the operation safety and the success rate are improved in an auxiliary mode.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a surgical process monitoring method and system based on machine vision. Background Art

[0002] In the modern medical field, surgery is an important means of treating many diseases, and its safety and accuracy have always been the focus of attention. During the operation, doctors need to pay attention to the operation of surgical instruments and the state of biological tissues to ensure the smooth progress of the operation and reduce the occurrence of complications.

[0003] At present, the monitoring of the surgical process mainly relies on the doctor's experience and on-site observation. However, this traditional monitoring method has many limitations. On the one hand, the surgical environment is complex, and doctors will inevitably experience visual fatigue or negligence during the highly tense and focused operation, resulting in the inability to detect some subtle but abnormal conditions that may affect the outcome of the operation, such as improper operation of surgical instruments or unexpected deformation of biological tissues. On the other hand, different doctors have different levels of experience, and their ability to judge and identify potential risks in certain complex surgical scenarios is also different, which may lead to the handling of abnormal events during the operation not being timely and accurate, thus affecting the surgical effect and the patient's prognosis.

[0004] Some existing surgical monitoring technologies mainly focus on monitoring the operating status of surgical equipment, such as monitoring the electrical parameters of surgical instruments, the temperature of equipment, etc., while monitoring the complex interaction between instruments and biological tissues during surgery and the actual operating status is relatively rare. Even if there are some image monitoring technologies for surgical scenes, most of them remain at the simple image recording level, failing to deeply analyze the key information in the image, and unable to accurately extract key elements such as surgical instrument type, spatial posture, biological tissue type and regional boundaries. It is even more difficult to effectively monitor the operation trajectory of surgical instruments and the deformation parameters of biological tissues.

[0005] In addition, most existing methods only set fixed thresholds for a single type of abnormal situation for detection, which cannot adapt to the diverse risks faced by different stages and operations during surgery, and it is difficult to accurately determine the type and risk level of abnormal events. Moreover, after an abnormal situation occurs, the existing technology cannot provide an intuitive and targeted visual report to help doctors quickly understand the abnormal situation and obtain effective correction suggestions. Summary of the invention

[0006] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides a surgical process monitoring method based on machine vision, the method comprising:

[0007] Acquire a real-time image sequence of a surgical scene, wherein the real-time image sequence includes dynamic images of a surgical instrument operation area and a biological tissue area;

[0008] Extracting surgical instrument recognition results and biological tissue segmentation results from the real-time image sequence, wherein the surgical instrument recognition results include instrument type and spatial posture, and the biological tissue segmentation results include tissue type and region boundary;

[0009] Based on the surgical instrument recognition result and the biological tissue segmentation result, monitoring the operation trajectory of the surgical instrument and the deformation parameters of the biological tissue area, and generating dynamic operation state parameters;

[0010] According to the predefined safety threshold of the surgical stage, multi-modal abnormality detection is performed on the dynamic operation state parameters to determine the abnormal event type and risk level;

[0011] Based on the abnormal event type and risk level, a visual surgery monitoring report is generated, and the visual surgery monitoring report includes abnormal area marking, risk warning information and correction suggestions.

[0012] On the other hand, an embodiment of the present invention also provides a surgical process monitoring system based on machine vision, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0013] Based on the above aspects, the embodiment of the present application obtains a real-time image sequence containing dynamic images of the surgical instrument operation area and the biological tissue area, and extracts the surgical instrument recognition results and the biological tissue segmentation results therefrom, thereby monitoring the surgical instrument operation trajectory and the biological tissue area deformation parameters and generating dynamic operation state parameters, thereby quantifying and integrating the dynamic information during the operation. Then, a multi-modal abnormality detection is performed on the dynamic operation state parameters using a predefined surgical stage safety threshold, which can comprehensively and flexibly judge the abnormal situation during the operation based on the characteristics of different surgical stages, accurately determine the abnormal event type and risk level, and not only consider the diversity and complexity of surgical operations, but also combine the safety standards of different stages, greatly improve the accuracy and reliability of abnormality detection, and avoid the limitations of a single detection method. Finally, a visual surgical monitoring report is generated based on the abnormal event type and risk level, and complex abnormal information is presented to medical staff in an intuitive and easy-to-understand manner, including abnormal area markings, risk warning information and correction suggestions, which not only helps medical staff quickly locate and understand problems during surgery, but also provides them with targeted solutions, which helps to take timely measures to correct surgical operations, reduce surgical risks, and improve surgical success rates and patient safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the execution flow of the surgical process monitoring method based on machine vision provided by an embodiment of the present invention.

[0015] Figure 2 Schematic diagram of exemplary hardware and software components of a machine vision-based surgical process monitoring system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1 The figure is a flow chart of a method for monitoring a surgical process based on machine vision provided by an embodiment of the present invention. The method for monitoring a surgical process based on machine vision is introduced in detail below.

[0017] Step S110, acquiring a real-time image sequence of the surgical scene, wherein the real-time image sequence includes dynamic images of the surgical instrument operation area and the biological tissue area.

[0018] In this embodiment, taking heart bypass surgery as an example, during the heart bypass surgery, a real-time image sequence of the surgical scene can be obtained by a camera installed above the operating table. For example, the frame rate of the camera is set to 30 frames per second, and the resolution is 1920×1080 pixels to ensure that the dynamic images of the surgical instrument operation area and the biological tissue area can be clearly captured. The surgical instrument operation area may include the operating range of instruments such as vascular clamps and staplers used in heart bypass surgery, and the biological tissue area covers the heart, coronary arteries, and surrounding vascular tissues. As the operation proceeds, the camera continues to shoot, and 30 images containing the above areas are generated every second. These images are arranged in chronological order to form a real-time image sequence. For example, when a surgeon uses a vascular clamp to clamp a coronary artery, the camera can accurately record dynamic images such as the moment when the vascular clamp contacts the coronary artery, the movement process of the vascular clamp, and the morphological changes of the coronary artery.

[0019] Step S120, extracting surgical instrument recognition results and biological tissue segmentation results from the real-time image sequence, the surgical instrument recognition results include instrument type and spatial posture, and the biological tissue segmentation results include tissue type and region boundary.

[0020] In detail, firstly, the real-time image sequence is sampled by key frames, and one frame is selected as the key frame every 5 seconds to obtain multiple groups of static images of surgical scenes. For each group of static images of surgical scenes, grayscale equalization is first performed to enhance the contrast difference between the instrument operation area and the biological tissue area. For example, in the image of heart bypass surgery, the grayscale value of the heart tissue and blood vessels is originally slightly different from the grayscale value of the surgical instrument. After grayscale equalization, the contrast between the metal part of the vascular clamp and the heart tissue is significantly increased. Then, the pixel area whose gradient amplitude exceeds the preset threshold in the image after grayscale equalization is extracted by the edge detection algorithm to generate an initial candidate frame set. In this scenario, the preset threshold is set to 50. For instruments such as vascular clamps, the gradient amplitude of the edge part is large, which can be accurately identified and included in the initial candidate frame set.

[0021] Next, the region growing algorithm is performed on the initial candidate frame set to merge candidate frames with overlapping spatial positions and similar texture features to generate a merged candidate frame set. For example, the two clamp arms of the vascular clamp may be identified separately in the initial candidate frame. Through the region growing algorithm, they are merged into a complete vascular clamp candidate frame based on their similar texture features. Then, according to the preset instrument size range and tissue size range, the candidate frames that meet the corresponding size constraints are selected from the merged candidate frame set to obtain a preliminary screened candidate frame set. For example, the length range of the vascular clamp is preset to 10-20 cm and the width is 1-3 cm. Candidate frames that do not meet this size range will be excluded. Finally, the morphological closing operation is performed on the candidate frame set after the preliminary screening to fill the broken area inside the candidate frame and smooth the jagged edges of the candidate frame to generate the final instrument operation area candidate frame and biological tissue area candidate frame.

[0022] Based on the three-dimensional convolutional network, the candidate frames of the instrument operation area are matched with multi-scale features to determine the instrument type and spatial posture, and generate surgical instrument recognition results. For the vascular clamps in heart bypass surgery, the three-dimensional convolutional network accurately identifies them as vascular clamps by analyzing their features at different scales, such as the bending shape of the clamp arm and the structure of the clamp head, and at the same time determines the spatial posture of the vascular clamp, such as the opening direction of the vascular clamp and the angle with the heart surface. The candidate frames of the tissue area are classified at the pixel level through the semantic segmentation model to obtain the tissue type and regional boundaries, and generate biological tissue segmentation results. In heart bypass surgery, the semantic segmentation model can accurately distinguish different tissue types such as heart tissue, coronary artery, and venous blood vessels, and accurately depict their regional boundaries, such as the diameter and direction of the coronary artery and the connection area with the heart tissue. Finally, the surgical instrument recognition results and the biological tissue segmentation results are aligned in time and space to form a unified spatial coordinate system mapping relationship, so that the relative position relationship between surgical instruments and biological tissues can be accurately described in the same coordinate system.

[0023] Step S130 , based on the surgical instrument recognition result and the biological tissue segmentation result, the operation trajectory of the surgical instrument and the deformation parameters of the biological tissue region are monitored to generate dynamic operation state parameters.

[0024] In detail, the movement direction and speed of the instrument under continuous time stamps can be calculated according to the spatial posture in the surgical instrument recognition result to generate the operation trajectory. For example, taking the vascular clamp as an example, if at time stamp t1, the vascular clamp is located at a certain point on the heart surface, its spatial posture is 30 degrees with the heart surface, and the opening direction is toward the upper left. At time t2, the vascular clamp moves to another position, the angle becomes 45 degrees, and the opening direction is toward the upper right. By calculating the position change and angle change of the vascular clamp under these two time stamps, it can be obtained that the movement direction of the vascular clamp is from the upper left to the upper right, and the speed is calculated according to the distance between the two points and the time interval (t2 - t1).

[0025] Based on the regional boundaries in the biological tissue segmentation results, the deformation amplitude and deformation rate of the tissue between adjacent timestamps are analyzed to generate deformation parameters. For example, when a surgeon uses a stapler to perform an anastomosis operation on the coronary artery, the regional boundary of the coronary artery will change. At time t1, the diameter of a section of the coronary artery is 2 mm. At time t2, due to the squeezing of the stapler, the diameter becomes 1.8 mm. The deformation amplitude is obtained by calculating the change in diameter (2 - 1.8 = 0.2 mm), and then the deformation rate is calculated according to the time interval (t2 - t1).

[0026] The operation trajectory and deformation parameters are integrated to calculate the stress distribution map of the contact area between the surgical instrument and the tissue. In heart bypass surgery, when the vascular clamp clamps the coronary artery, the stress distribution in the contact area between the vascular clamp and the coronary artery is calculated through the mechanical model based on the operation trajectory of the vascular clamp (such as the clamping force and direction) and the deformation parameters of the coronary artery (such as the degree of compression of the clamped part). For example, the stress is greater around the contact point, and the stress gradually decreases as it diffuses to the surrounding area.

[0027] According to the stress distribution map, the quantitative indicators of instrument operation force, tissue response elastic modulus and contact time ratio are generated. For the case of clamping the coronary artery with a vascular clamp, the operation force of the vascular clamp is calculated to be 10 Newtons according to the stress distribution map. The elastic modulus of tissue response is 100 MPa through the deformation analysis of the coronary artery under stress. The contact time ratio is calculated to be 5% based on the total time of clamping the coronary artery with the vascular clamp and the entire operation time. These quantitative indicators are matched with the preset operation stage model to output dynamic operation state parameters. In different stages of heart bypass surgery, such as the vascular free stage and the anastomosis stage, the preset operation stage model has different requirements for these quantitative indicators. When these quantitative indicators are within the range of the corresponding stage, the dynamic operation state parameters indicate that the operation is proceeding normally, otherwise it indicates that there may be risks.

[0028] Step S140, performing multimodal abnormality detection on the dynamic operation state parameters according to the predefined safety threshold of the surgical stage, and determining the abnormal event type and risk level.

[0029] In detail, the time series feature sequence of the instrument operation force, tissue deformation rate and stress distribution map in the dynamic operation state parameters can be extracted. For example, assuming that the operation lasts for 10 minutes, the data of the instrument operation force, tissue deformation rate and stress distribution map are recorded every 30 seconds, thus obtaining a time series feature sequence.

[0030] According to the predefined safety threshold of the surgical stage, the time series feature sequence is segmented into time window segments that match the current surgical stage. For example, in the vascular anastomosis stage of heart bypass surgery, the time window is set to 2 minutes, so the 10-minute time series feature sequence is segmented into 2-minute segments.

[0031] The force deviation coefficient is generated by performing a difference operation on the mean force of the instrument operation in each time window segment and the force upper limit value in the safety threshold of the surgical stage. In the vascular anastomosis stage, if the upper limit value of the instrument operation force in the safety threshold is 15 Newtons, and the mean force of the instrument operation in a certain time window is 20 Newtons, then the force deviation coefficient is (20 -15) / 15 = 0.33.

[0032] The peak value of the tissue deformation rate in the time window segment is compared with the deformation rate threshold in the safety threshold of the surgical stage to generate a rate exceeding flag. Assuming that the deformation rate threshold in the vascular anastomosis stage is 0.5 mm / s, and the peak value of the tissue deformation rate in a certain time window is 0.8 mm / s, then the rate exceeding flag is 1, indicating that it exceeds the standard.

[0033] The overlap ratio of the stress distribution map coverage area and the contact area threshold in the surgical safety threshold is calculated to generate an area abnormality ratio value. For example, if the contact area covered by the stress distribution map is 10 square millimeters and the contact area threshold is 8 square millimeters, then the area abnormality ratio value is (10 - 8) / 8 = 0.25.

[0034] The force deviation coefficient, rate exceeding flag and area abnormality ratio are input into the pre-trained multi-level decision tree model, where the first-level decision node determines whether the force deviation coefficient exceeds the first risk critical value. Assuming that the first risk critical value is 0.3, since the force deviation coefficient is 0.33, it exceeds the critical value. The second-level decision node generates a tissue cumulative damage index based on the number of times the rate exceeding flag is triggered. Since the rate exceeding flag is 1, it is assumed that the tissue cumulative damage index increases by 0.2 each time it exceeds the standard. If there is no exceeding before, the tissue cumulative damage index is 0.2 at this time. The third-level decision node generates a contact abnormality pattern based on the area abnormality ratio value and the deformation spatial distribution characteristics. According to the spatial distribution of the area abnormality ratio value and the stress distribution map, the contact abnormality pattern is judged to be excessive local pressure.

[0035] Based on the intermediate judgment results output by the multi-level decision tree model, the corresponding abnormal event type label is activated. The abnormal event type label includes at least three categories: instrument overpressure, tissue overspeed deformation, and abnormal contact area. Because the force deviation coefficient exceeds the critical value and the contact abnormality mode is excessive local pressure, the abnormal event type label of instrument overpressure is activated.

[0036] The change in the regional boundary pixels in the biological tissue segmentation results is obtained in real time. When the continuous expansion or contraction of the boundary pixels is detected, a tissue integrity destruction signal is generated. In heart bypass surgery, if the regional boundary pixels of the coronary artery shrink continuously, it may indicate that the coronary artery is over-compressed, generating a tissue integrity destruction signal.

[0037] Perform a logical AND operation on the tissue integrity destruction signal and the activated abnormal event type tag. If there is an overlapping interval between the two in the timestamp, the risk level of the corresponding abnormal event type is increased by one level. Assuming that the initial risk level of the abnormal event type of device overpressure is level 1, since the tissue integrity destruction signal and device overpressure exist at the same time in a certain time interval, the risk level of device overpressure is increased to level 2.

[0038] According to the level mapping rules set in the safety threshold of the surgical stage, the final abnormal event type is bound to the increased risk level. In the safety threshold setting of coronary bypass surgery, risk level 2 corresponds to the situation that requires timely adjustment of the operation, and the abnormal event type of device overpressure is bound to risk level 2.

[0039] Step S150, generating a visual surgery monitoring report based on the abnormal event type and risk level, wherein the visual surgery monitoring report includes abnormal area marking, risk warning information and correction suggestions.

[0040] In detail, the abnormal area can be located in the real-time image sequence according to the tissue area boundary and the device contact point coordinates corresponding to the abnormal event type, and a color-coded contour line and text label matching the risk level can be generated. For the abnormal event type of device overpressure, the corresponding position is found in the real-time image sequence according to the area boundary where the coronary artery is over-compressed and the coordinates of the contact point between the vascular clamp and the coronary artery. When the risk level is 2, the abnormal area is marked with a red color-coded contour line, and a text label "Device Overpressure-Risk Level 2" is added next to it.

[0041] Based on the alarm priority corresponding to the risk level, a multi-level voice alarm signal is generated. The alarm priority corresponding to risk level 2 is high, and the surgical monitoring system will issue a "beep, beep" urgent voice alarm signal, and at the same time, the voice prompt "The instrument is over-pressurized, please pay attention to adjust the operation."

[0042] The instrument path adjustment rules and force optimization parameters in the predefined surgical operation correction solution library are matched according to the abnormal event type. Among them, the instrument overpressure type generates a reverse displacement vector arrow and a pressure reduction curve. In heart bypass surgery, a reverse displacement vector arrow is displayed in the monitoring report to indicate that the vascular clamp should move in the opposite direction, and a pressure reduction curve is given to show how the pressure should gradually decrease to a safe range if the correct operation is followed.

[0043] The color-coded contours, text labels, voice alarm signals, and correction guidance parameters are synchronously transmitted to the display terminal of the surgical navigation system. The terminal interface is divided into the first display area for real-time rendering of the original image stream, the second display area for superimposing abnormal contours and correction guidance layers, and the third display area for rolling updates of risk levels and voice-to-text logs. On the display terminal of the surgical navigation system, the first display area normally displays the real-time operation screen of the heart bypass surgery, the second display area superimposes the red abnormal area contours and correction guidance layers such as reverse displacement vector arrows and pressure reduction curves on the original screen, and the third display area continuously rolls out the risk level of 2 and the log information of "Device overpressure, please pay attention to adjust the operation" after voice-to-text conversion.

[0044] According to the dynamic changes in the progress of the surgical stage and the risk level, the contour line color intensity and correction guidance parameters in the second display area are updated in real time. When the risk level of the same abnormal event type is detected to increase, the contour line of the corresponding area is switched to a higher priority color and the display size of the correction guidance arrow is enlarged. If the risk level of the instrument overpressure is further increased to level 3, then in the second display area, the color intensity of the red contour line will be enhanced, and the display size of the reverse displacement vector arrow will be enlarged, which will more prominently remind the surgeon to pay attention to the abnormal situation and adjust the operation in time.

[0045] Based on the above steps, the embodiment of the present application obtains a real-time image sequence containing dynamic images of the surgical instrument operation area and the biological tissue area, and extracts the surgical instrument recognition results and biological tissue segmentation results from it, thereby monitoring the surgical instrument operation trajectory and the biological tissue area deformation parameters and generating dynamic operation state parameters, thereby quantifying and integrating the dynamic information in the surgical process. Then, a multi-modal abnormality detection is performed on the dynamic operation state parameters using a predefined surgical stage safety threshold, which can comprehensively and flexibly judge the abnormal situation in the surgical process based on the characteristics of different surgical stages, accurately determine the abnormal event type and risk level, and not only consider the diversity and complexity of surgical operations, but also combine the safety standards of different stages, greatly improve the accuracy and reliability of abnormality detection, and avoid the limitations of a single detection method. Finally, a visual surgical monitoring report is generated based on the abnormal event type and risk level, and complex abnormal information is presented to medical staff in an intuitive and easy-to-understand manner, including abnormal area markings, risk warning information and correction suggestions, which not only helps medical staff quickly locate and understand problems in surgery, but also provides them with targeted solutions, which helps to take timely measures to correct surgical operations, reduce surgical risks, and improve surgical success rates and patient safety.

[0046] In a possible implementation, step S120 includes:

[0047] Step S121, performing key frame sampling on the real-time image sequence to obtain multiple groups of surgical scene static images.

[0048] For example, considering the rhythm of the operation and the key nodes of the operation, key frame sampling can be set every 10 seconds. Since the entire operation may last for several hours, multiple sets of static images of the surgical scene can be obtained at different time points. For example, a static image of the surgical scene is obtained at 10 seconds, 20 seconds, 30 seconds, etc. after the start of the operation. These static images of the surgical scene include the operation of surgical instrument operating areas such as vascular clamps and staplers, as well as the status of biological tissue areas such as the heart and coronary arteries.

[0049] Step S122, extracting feature regions from each group of surgical scene static images to obtain instrument operation region candidate frames and tissue region candidate frames.

[0050] In detail, in a possible implementation, step S122 includes:

[0051] Step S1221, performing grayscale equalization processing on the static image of the surgical scene to enhance the contrast difference between the instrument operation area and the biological tissue area.

[0052] For images of heart bypass surgery, biological tissues such as heart tissue and coronary arteries and surgical instruments such as vascular clamps and staplers may originally have similar grayscale values. After grayscale equalization processing, for example, the grayscale value of the metal part of the vascular clamp will form a sharp contrast with the surrounding heart tissue, making it easier to distinguish.

[0053] Step S1222, extracting pixel areas whose gradient amplitude exceeds a preset threshold in the image after grayscale equalization processing by an edge detection algorithm, and generating an initial candidate frame set.

[0054] For example, the preset threshold is set to 30. Since the edge of the vascular clamp and the edge of the coronary artery have large gradient amplitudes in the image, these areas will be accurately identified and included in the initial candidate frame set.

[0055] Step S1223 , executing a region growing algorithm on the initial candidate frame set, merging candidate frames with overlapping spatial positions and similar texture features, and generating a merged candidate frame set.

[0056] For example, the two arms of a vascular clamp may be initially identified as two separate candidate boxes, but because their spatial positions overlap and their texture features are similar, the two candidate boxes are merged into a complete vascular clamp candidate box through the region growing algorithm.

[0057] Step S1224 , based on the preset instrument size range and tissue size range, select candidate frames that meet the corresponding size constraints from the merged candidate frame set to obtain a preliminarily selected candidate frame set.

[0058] For the vascular clamp, its actual size range can be set as 8 - 18 cm in length and 0.8 - 2.5 cm in width. For the coronary artery, the diameter range is set as 1 - 3 mm. Candidate frames that do not meet these size ranges will be excluded.

[0059] Step S1225, performing morphological closing operation processing on the candidate frame set after the preliminary screening, filling the broken area inside the candidate frame and smoothing the jagged edges of the candidate frame, and generating the final instrument operation area candidate frame and biological tissue area candidate frame.

[0060] For example, for the candidate box of the vascular clamp, if there is an internal broken area caused by image noise or other reasons, it will be filled after morphological closing operation, and the edge jaggedness will be smoothed, making the candidate box of the vascular clamp more accurate and complete. The same processing will be performed on the candidate box of the tissue area of ​​the coronary artery.

[0061] Step S123, performing multi-scale feature matching on the candidate frame of the instrument operation area based on a three-dimensional convolutional network, determining the instrument type and spatial posture, and generating the surgical instrument recognition result.

[0062] For example, for the vascular clamp in the candidate box of the instrument operation area, the three-dimensional convolutional network can analyze it from multiple scales. From the macroscopic scale, the overall shape structure of the vascular clamp can be identified, such as the approximate length and curvature of the clamp arm; from the microscopic scale, the fine structural features of the clamp head can be identified. Through these multi-scale feature matching, the instrument is accurately identified as a vascular clamp type, and its spatial posture is determined, such as the opening angle of the vascular clamp relative to the surface of the heart. Assuming that at a certain moment the opening direction of the vascular clamp is at a 45-degree angle to the surface of the heart, and the clamp arm is parallel to the long axis of the heart, these are specific descriptions of the spatial posture, thereby generating surgical instrument recognition results.

[0063] Step S124, performing pixel-level classification on the tissue region candidate frame through a semantic segmentation model to obtain the tissue type and region boundary, and generating the biological tissue segmentation result.

[0064] In detail, the semantic segmentation model classifies the pixels in the tissue region candidate frame. For cardiac tissue, it can distinguish different tissue types such as myocardial tissue, endocardium, epicardium, etc. For coronary arteries, it can accurately define their regional boundaries, including their starting point, direction, branching, and connection relationship with surrounding tissues on the surface of the heart. For example, the semantic segmentation model can accurately depict that the diameter of a certain section of coronary artery is 1.5 mm, which starts from the left anterior descending branch of the heart, extends to the right along the surface of the heart and divides into a small branch at a certain position. This information constitutes the content of tissue type and regional boundary, thereby generating biological tissue segmentation results.

[0065] Step S125 , aligning the surgical instrument recognition result with the biological tissue segmentation result in time and space to form a unified spatial coordinate system mapping relationship.

[0066] In the scenario of heart bypass surgery, this means placing the surgical instrument recognition results of the vascular clamp, such as the position and posture of the vascular clamp, and the position and boundary information of the segmentation results of biological tissues such as heart tissue and coronary arteries in the same spatial coordinate system. For example, in this unified spatial coordinate system, the specific coordinate position of the contact point between the clamp head and the coronary artery on the surface of the heart, as well as the angular relationship between the operating direction of the vascular clamp and the direction of the coronary artery, can be accurately described, thereby establishing an accurate spatial correspondence, which is convenient for subsequent surgical monitoring and analysis operations.

[0067] In a possible implementation, step S130 includes:

[0068] Step S131, according to the spatial posture in the surgical instrument recognition result, the moving direction and speed of the instrument under continuous time stamps are calculated to generate the operation trajectory.

[0069] Still taking the surgical instrument of vascular clamp as an example, it is assumed that at timestamp t1, the spatial posture of the vascular clamp is determined by the surgical instrument recognition result: its clamp head is located at the position of the heart surface coordinate (x1, y1, z1), the angle between the clamp arm and the heart surface is α1, and the opening direction is the vector direction (v1). At the subsequent timestamp t2, the position of the clamp head of the vascular clamp becomes (x2, y2, z2), the angle between the clamp arm and the heart surface becomes α2, and the opening direction is the vector direction (v2). When calculating the direction of movement of the instrument, first calculate the displacement vector from the coordinates (x1, y1, z1) to (x2, y2, z2), and the direction of the vector represents the direction of movement. For example, the displacement in the x direction is calculated as x2 - x1, the y direction is y2 - y1, and the z direction is z2 - z1, and the displacement vector (dx, dy, dz) is obtained. The speed is calculated based on the distance formula between two points. First, the displacement is calculated, that is, the displacement distance s is equal to the square root of ((x2 -x1) square + (y2 - y1) square + (z2 - z1) square), and then divided by the time interval (t2 - t1) to get the speed value. By repeating such calculations at different consecutive time stamps, the operation trajectory of the vascular clamp can be completely determined.

[0070] Step S132, based on the region boundary in the biological tissue segmentation result, analyzing the tissue deformation amplitude and deformation rate between adjacent time stamps to generate the deformation parameter.

[0071] For example, for the biological tissue of the coronary artery in heart bypass surgery, the regional boundary information at timestamp t1 is obtained from the biological tissue segmentation result, such as the diameter of a certain section of the coronary artery is d1. At the adjacent timestamp t2, the diameter of the same section of the coronary artery is obtained again as d2. The deformation amplitude of the tissue is calculated as the absolute value of d1 - d2. The deformation rate is calculated by dividing the deformation amplitude by the time interval (t2 - t1). If d1 is 3 mm, d2 is 2.8 mm, and the time interval is 10 seconds, then the deformation amplitude is 3 - 2.8, which is equal to 0.2 mm, and the deformation rate is 0.2 mm divided by 10 seconds, that is, 0.02 mm / second, and the deformation parameters of the coronary artery are output.

[0072] Step S133, integrating the operation trajectory and the deformation parameter to calculate a stress distribution map of the contact area between the surgical instrument and the tissue.

[0073] For example, when the vascular clamp contacts the coronary artery, the stress distribution map is calculated based on the previously obtained operation trajectory of the vascular clamp (such as the moving direction and speed, etc.) and the deformation parameters of the coronary artery (such as the change in diameter, etc.). Assuming that the moving speed of the vascular clamp is v and the deformation of the coronary artery is Δd, according to the principles of mechanics, the stress in the contact area is related to the moving speed of the vascular clamp and the deformation of the coronary artery. For example, if there is a calculation model based on a large number of experiments and theoretical derivations that stress σ is equal to k times v times Δd (here k is a coefficient determined according to tissue characteristics and device characteristics), by substituting the speed value of the vascular clamp and the deformation value of the coronary artery, as well as the known coefficient k, the stress values ​​at different points in the contact area can be calculated, thereby constructing a stress distribution map.

[0074] Step S134, generating quantitative indicators of instrument operation force, tissue response elastic modulus and contact time ratio according to the stress distribution map.

[0075] For example, for the instrument operation force, in the stress distribution map, the magnitude of stress is directly related to the instrument operation force. For example, if the stress distribution map shows that the stress value at the center of the contact area is σ0, based on the relationship between force and stress (force equals stress multiplied by the contact area), assuming that the contact area is A, then the instrument operation force F is equal to σ0 multiplied by A. For the tissue response elastic modulus, it is calculated based on the deformation of the coronary artery under stress. Assuming that under the action of stress σ, the strain of the coronary artery (strain equals deformation amplitude divided by original size) is ε, according to the definition of elastic modulus (elastic modulus equals stress divided by strain), the tissue response elastic modulus E is equal to σ divided by ε. The contact time ratio is calculated based on the contact time between the vascular clamp and the coronary artery and the entire surgical operation time. Assuming that the contact time between the vascular clamp and the coronary artery is t0 and the entire surgical operation time is T, then the contact time ratio is equal to t0 divided by T.

[0076] Step S135, matching the quantitative index with a preset surgical stage model, and outputting the dynamic operation state parameter.

[0077] There are preset operation stage models in different stages of heart bypass surgery, such as the vascular free stage and the vascular anastomosis stage. The operation stage model specifies the reasonable range of instrument operation force, tissue response elastic modulus and contact time ratio under normal operation in each stage. For example, in the vascular anastomosis stage, the preset instrument operation force range is F1-F2, the tissue response elastic modulus range is E1-E2, and the contact time ratio range is p1-p2. The quantitative indicators of instrument operation force, tissue response elastic modulus and contact time ratio calculated previously are compared with these ranges. If the instrument operation force is within the range of F1-F2, the tissue response elastic modulus is within the range of E1-E2, and the contact time ratio is within the range of p1-p2, the dynamic operation state parameter indicates that the current surgical operation is in a normal state; if one or some quantitative indicators exceed the corresponding range, the dynamic operation state parameter indicates that there may be risks and further analysis or adjustment of the operation is required.

[0078] In a possible implementation, step S132 includes:

[0079] Step S1321, extracting the tissue region boundary coordinate sequence corresponding to adjacent timestamps in the biological tissue segmentation result, matching the boundary coordinates of the previous timestamp with the boundary coordinates of the next timestamp point by point, and generating a displacement vector for each coordinate point.

[0080] For example, in heart bypass surgery, for the biological tissue of the coronary artery, the boundary coordinate sequence of the coronary artery region corresponding to adjacent timestamps t1 and t2 is obtained from the biological tissue segmentation results. For example, at time t1, the boundary coordinate points of a section of the coronary artery are (x1, y1, z1), (x2, y2, z2), etc., and the corresponding coordinate points at time t2 are (x1', y1', z1'), (x2', y2', z2'), etc. These coordinate points are matched point by point, and the displacement vector of each coordinate point is calculated. Taking the coordinate point (x1, y1, z1) as an example, its displacement vector is (x1' - x1, y1' - y1, z1' - z1). By performing such calculations on all coordinate points, the displacement vector of each coordinate point is obtained.

[0081] Step S1322: Calculate the overall displacement mean of the tissue region boundary between adjacent time stamps according to the length of the displacement vector to generate the tissue deformation amplitude.

[0082] In detail, for each displacement vector, its length is calculated. For example, the length of the displacement vector (dx, dy, dz) is the square root of (dx squared + dy squared + dz squared). After calculating the lengths of the displacement vectors of all coordinate points, the average of these lengths is calculated to obtain the overall displacement mean of the tissue region boundary, which is the tissue deformation amplitude. Assuming that there are n coordinate points in total, and the lengths of the displacement vectors of each coordinate point are L1, L2, ..., Ln, respectively, then the tissue deformation amplitude is equal to (L1 + L2 + ... + Ln) divided by n.

[0083] Step S1323, based on the rate of change of the displacement vector under continuous timestamps, calculate the displacement acceleration of each coordinate point, mark the coordinate point whose acceleration exceeds the preset acceleration threshold as a deformation mutation point, count the distribution density of the deformation mutation point within the boundary of the tissue area, and generate the local peak value of the tissue deformation rate according to the ratio of the distribution density to the acceleration threshold.

[0084] In detail, for the displacement vector of each coordinate point, its rate of change is calculated under continuous timestamps to obtain the displacement acceleration. For example, at timestamps t1, t2, and t3, the displacement vectors of a certain coordinate point are (v1, v2, v3) respectively. The displacement vector change from t1 to t2 is first calculated as (v2 - v1), and the displacement vector change from t2 to t3 is (v3 - v2), and then divided by the corresponding time intervals to obtain the displacement acceleration of these two time periods. Assuming that the preset acceleration threshold is a0, if the displacement acceleration of a certain coordinate point exceeds a0, it is marked as a deformation mutation point. The number of all deformation mutation points within the boundary of the tissue region is counted as m, and the total number of coordinate points within the boundary of the tissue region is N, then the distribution density of the deformation mutation points is m divided by N. According to the ratio of the distribution density to the acceleration threshold, that is, (m divided by N) divided by a0, the local peak value of the tissue deformation rate is generated.

[0085] Step S1324, performing normalized weighted fusion on the tissue deformation amplitude, deformation mutation point distribution density and local peak value of tissue deformation rate to generate a deformation parameter set.

[0086] For example, the tissue deformation amplitude, deformation mutation point distribution density and local peak value of tissue deformation rate are first normalized so that their values ​​are in the same order of magnitude. Assuming that the tissue deformation amplitude is A, the deformation mutation point distribution density is D, and the local peak value of tissue deformation rate is P, their weights are determined as w1, w2, and w3 respectively (these weights are predetermined based on factors such as the importance of the impact on surgical safety). By calculating w1 multiplied by A + w2 multiplied by D + w3 multiplied by P, a comprehensive value is obtained, and this value is combined with other relevant information (such as coordinate information) to generate a set of deformation parameters.

[0087] Step S1325 , according to the tissue type in the biological tissue segmentation result, a predefined tissue elastic property library is matched to extract a deformation tolerance coefficient corresponding to the tissue type.

[0088] In heart bypass surgery, for the coronary artery tissue type, the corresponding deformation tolerance coefficient is found from the predefined tissue elastic property library. Assuming that the deformation tolerance coefficient of the coronary artery is k, the deformation tolerance coefficient reflects the degree of deformation that the coronary artery can withstand under normal circumstances.

[0089] Step S1326, perform a proportional operation on the tissue deformation amplitude in the deformation parameter set and the deformation tolerance coefficient to generate a deformation over-limit risk index, and compare the exponential increments of adjacent timestamps with a predefined stage deformation increment threshold according to the temporal variation trend of the deformation over-limit risk index to generate a deformation parameter dynamic abnormality flag.

[0090] In detail, the deformation over-limit risk index is calculated. For example, if the tissue deformation amplitude in the deformation parameter set is A, then the deformation over-limit risk index is equal to A divided by k. The temporal change trend of the deformation over-limit risk index is analyzed. At adjacent timestamps t1 and t2, the deformation over-limit risk index is I1 and I2 respectively, and the calculated index increment is I2 - I1. Assuming that the predefined stage deformation increment threshold is ΔI, if I2 - I1 is greater than ΔI, a dynamic abnormal flag of the deformation parameter indicating the abnormality is generated (for example, marked as 1), otherwise it is marked as normal (for example, marked as 0).

[0091] Step S1327, spatially superimpose the deformation parameter set, deformation over-limit risk index and deformation parameter dynamic anomaly flag to form a deformation parameter thermal layer with a timestamp mark, and generate a tissue deformation propagation path prediction trajectory based on the parameter change direction of continuous timestamps in the deformation parameter thermal layer.

[0092] In detail, each parameter value in the deformation parameter set, the deformation over-limit risk index, and the deformation parameter dynamic abnormal flag can be spatially superimposed according to their positions in the tissue area, and marked with a timestamp to form a deformation parameter thermodynamic layer. In the deformation parameter thermodynamic layer, the change direction of the parameters under continuous timestamps is observed. For example, the deformation parameters of a certain area gradually increase and move in a certain direction under continuous timestamps. Based on this information, the tissue deformation propagation path is predicted to obtain the tissue deformation propagation path prediction trajectory.

[0093] Step S1328, adjusting the dynamic weight coefficient of the deformation parameter according to the spatial distance between the deformation propagation path prediction trajectory and the surgical instrument operation trajectory, and generating a final deformation parameter set.

[0094] For example, in the scenario of heart bypass surgery, the spatial distance between the predicted trajectory of the deformation propagation path and the trajectory of the surgical instrument operation (such as the operation trajectory of the vascular clamp) is calculated. Assuming that the spatial distance between the two is d, if d is small, it means that the surgical instrument operation has a greater impact on tissue deformation. At this time, the dynamic weight coefficient of the deformation parameters related to the surgical instrument operation (such as tissue deformation amplitude, etc.) is appropriately increased; if d is large, it means that the impact is small, and the relevant weight coefficient is appropriately reduced. According to the adjusted weight coefficient, the previous weighted fusion and other operations are recalculated to generate the final deformation parameter set, which can more accurately reflect the deformation of the tissue and its relationship with the surgical instrument operation.

[0095] In a possible implementation, step S133 includes:

[0096] Step S1331, extracting the three-dimensional space coordinate sequence corresponding to each time stamp of the instrument tip according to the instrument movement direction and speed at the continuous time stamps in the operation trajectory.

[0097] For example, in a heart bypass surgery, a vascular clamp is used as a surgical instrument, and it is assumed that the timestamps are marked at fixed intervals (such as one timestamp every 1 second). At timestamp t1, the position coordinates of the tip of the vascular clamp are calculated to be (x1, y1, z1) through the previously determined operation trajectory. At this time, the movement direction of the vascular clamp is the vector direction v1, and the speed is s1. As time goes by, at timestamp t2, the position of the tip of the vascular clamp becomes (x2, y2, z2), the movement direction is v2, the speed is s2, and so on. In this way, a series of three-dimensional spatial coordinate sequences of the tip of the vascular clamp at different timestamps can be obtained. This three-dimensional spatial coordinate sequence completely records the movement trajectory of the vascular clamp during the operation.

[0098] Step S1332: extracting a deformation region boundary coordinate sequence of the tissue surface at the corresponding time stamp based on the tissue deformation amplitude and deformation rate between adjacent time stamps in the deformation parameters.

[0099] For the biological tissue of the coronary artery in heart bypass surgery, information is obtained from the deformation parameters obtained in the previous analysis. Between the adjacent timestamps t1 and t2, the deformation amplitude of the tissue is Δd1, and the deformation rate is r1. At time t1, the boundary coordinate points of the deformation area on the surface of the coronary artery are (a1, b1, c1), (a2, b2, c2), etc. At time t2, due to the deformation of the tissue, these coordinate points may become (a1', b1', c1'), (a2', b2', c2'), etc. In this way, the boundary coordinate sequence of the deformation area on the surface of the coronary artery can be obtained at each timestamp.

[0100] Step S1333: Time-synchronize and align the three-dimensional spatial coordinate sequence of the instrument tip with the coordinate sequence of the boundary of the deformation region to generate a mapping relationship between the synchronized instrument contact point position and the tissue deformation region.

[0101] Since the timestamps of the operation trajectory and tissue deformation are set at the same time interval, they can be matched one to one. For example, when the coordinates of the tip of the vascular clamp at timestamp t1 are (x1, y1, z1), the corresponding coordinates of the boundary of the deformation area of ​​the coronary artery at time t1 are (a1, b1, c1), etc. Through such time synchronization alignment, the relative position relationship between the tip of the vascular clamp and the surface of the coronary artery at each time point can be determined, thereby establishing a mapping relationship between the position of the instrument contact point and the tissue deformation area. This mapping relationship makes it possible to accurately know the specific position where the vascular clamp contacts the coronary artery during the operation and the corresponding tissue deformation.

[0102] Step S1334, according to the synchronized instrument contact point position, locate the spatial range of the contact area between the instrument tip and the tissue surface in the deformation area boundary coordinate sequence, and calculate the deformation acceleration distribution of the contact area in the instrument movement direction based on the tissue deformation rate at each time stamp within the contact area spatial range.

[0103] On the basis of the established mapping relationship, after the contact point position between the tip of the vascular clamp and the surface of the coronary artery is determined, for example, within a certain time period, the contact area includes the spatial range of the coordinate points (a3, b3, c3) to (a5, b5, c5) on the surface of the coronary artery. For this contact area, the deformation acceleration distribution is calculated using the tissue deformation rate at each time stamp. Assuming that the deformation rates at time stamps t1, t2, and t3 are r1, r2, and r3, respectively, the deformation rate change from t1 to t2 is calculated as r2-r1, and the deformation rate change from t2 to t3 is calculated as r3-r2, and then divided by the corresponding time interval (such as 1 second), the deformation rate change rate of the contact area during this period of time, that is, the deformation acceleration, is obtained. Such calculations are performed at each position point in the entire contact area, and the deformation acceleration distribution of the contact area in the direction of device movement can be obtained.

[0104] Step S1335, determining the stress direction vector of each sub-area in the contact area according to the spatial angle between the deformation acceleration distribution and the moving direction of the instrument, fusing the stress direction vector with the spatial gradient of the tissue deformation amplitude, and generating a dynamic stress propagation path.

[0105] For each sub-region in the contact area, for example, the contact area is divided into multiple small sub-regions according to certain rules, and the stress direction vector is determined according to the spatial angle between the deformation acceleration distribution and the instrument movement direction obtained by the previous calculation. Assuming that in a certain sub-region, the deformation acceleration distribution vector is A, and the instrument movement direction vector is V, the stress direction vector is obtained through a specific vector operation (here, the formula is not used, but the operation logic is described in words: first calculate the angle θ between A and V, and determine the size and direction of the stress direction vector S according to the size of the angle and A. For example, if θ is small, the direction of S is closer to the direction of A and the size is in a certain proportional relationship with the size of A). At the same time, the spatial rate of change of the tissue deformation amplitude, that is, the spatial gradient of the tissue deformation amplitude, is calculated. The stress direction vector is fused with the spatial gradient of the tissue deformation amplitude. For example, in each sub-region, the stress direction vector and the spatial gradient of the tissue deformation amplitude are added according to a certain weight (the weight is predetermined according to the actual physical relationship and surgical situation) to obtain a comprehensive vector. According to the direction and size of the comprehensive vector in different sub-regions, the propagation direction and trend of stress in the tissue can be determined, thereby generating a dynamic stress propagation path.

[0106] Step S1336, generating the stress distribution map according to the coverage and direction consistency of the dynamic stress propagation path under continuous time stamps, wherein different color areas in the stress distribution map represent stress accumulation intensity levels.

[0107] Under continuous time stamps, observe the coverage of the dynamic stress propagation path, that is, to which areas the stress propagates and the degree of propagation in each area. At the same time, analyze the directional consistency of the stress propagation path. For example, the stress propagation direction remains basically unchanged in some areas, while there may be large directional changes in other areas. Based on this information, the entire contact area is divided into different parts, and the stress accumulation intensity is quantified. For example, areas with higher stress accumulation intensity are represented by red, areas with medium intensity are represented by yellow, and areas with lower intensity are represented by green, etc. In this way, a stress distribution map is generated, which can intuitively show the distribution of stress in the contact area between the vascular clamp and the coronary artery during heart bypass surgery, including the magnitude, direction, and change trend of stress over time, etc., providing an important basis for safety assessment and operation adjustment of the surgery.

[0108] In a possible implementation, step S134 includes:

[0109] Step S1341, extracting the stress cumulative intensity level of the color-coded area in the stress distribution map, marking the continuous area whose color depth exceeds the preset intensity level as a high stress core area, dividing the high stress core area into grid units, counting the consistency ratio of the stress direction vector in each grid unit, and merging the grid units whose consistency ratio exceeds the preset threshold into isotropic stress belts.

[0110] In heart bypass surgery, the stress distribution map uses different colors to represent the cumulative intensity level of stress, such as red for high intensity, yellow for medium intensity, and green for low intensity. From the stress distribution map, a preset intensity level is set, such as treating the red area as an area where the color depth exceeds the preset intensity level, and these continuous red areas are marked as high stress core areas. Divide the grid cells in the high stress core area, assuming that they are divided into square grid cells with a side length of 1 mm. For each grid cell, the consistency ratio of the stress direction vector is counted. The stress direction vector is previously determined based on factors such as stress acceleration distribution, and indicates the direction of stress. For example, if there are 10 stress direction vector sample points in a grid cell, 8 of which are basically the same in direction, then the consistency ratio of the stress direction vector of the grid cell is 80%. Set the preset threshold to 70%, and merge the grid cells with a consistency ratio exceeding 70% into the same direction stress belt.

[0111] Step S1342, based on the product of the coverage area of ​​the isotropic stress band and the stress cumulative intensity level, generate an initial estimate of the instrument operation force at each timestamp, and match the predefined instrument rigidity coefficient according to the instrument type in the surgical instrument identification result, perform weighted correction on the initial estimate of the instrument operation force and the rigidity coefficient to generate the instrument operation force.

[0112] In heart bypass surgery, for isotropic stress bands, calculate their coverage area. Assuming that a isotropic stress band contains 5 grid units, each with an area of ​​1 square millimeter, the coverage area is 5 square millimeters. If the stress accumulation intensity level in the area where the isotropic stress band is located is high intensity (the corresponding value of high intensity is set to 3), then the initial estimate of the instrument operation force is 5×3 = 15 (this is just a calculation example based on the set rules). According to the surgical instrument identification result, if the surgical instrument is a vascular clamp, the rigidity coefficient of the vascular clamp is found to be 1.2 from the predefined instrument rigidity coefficient library. The initial estimate of the instrument operation force 15 is weighted and corrected with the rigidity coefficient 1.2, for example, the weighting method is multiplication (the actual weighting method is determined according to physical principles and experience), and the instrument operation force is 15×1.2 = 18, which is the quantitative indicator of the instrument operation force at this timestamp.

[0113] Step S1343, extracting the deformation recovery rate corresponding to the tissue deformation parameter in the biological tissue segmentation result, calculating the attenuation ratio of the deformation recovery rate and the stress cumulative intensity level in the same area in the stress distribution map, and determining the dynamic estimation of the tissue response elastic modulus based on the attenuation ratio and a predefined tissue elastic attenuation curve.

[0114] In heart bypass surgery, the deformation recovery rate corresponding to the tissue deformation parameters of the coronary artery is obtained from the biological tissue segmentation results. Assume that in a certain area, the deformation recovery rate is 0.2 mm / s. In the stress distribution map, the stress accumulation intensity level of the corresponding area is medium intensity (the corresponding value of medium intensity is set to 2), and the calculated attenuation ratio of the deformation recovery rate and the stress accumulation intensity level is 0.2÷2 = 0.1. According to the predefined tissue elastic attenuation curve, this curve is derived based on a large number of experiments and theoretical derivations. By finding the point corresponding to the attenuation ratio of 0.1 in the curve, the dynamic estimate of the tissue response elastic modulus is obtained. For example, the corresponding elastic modulus value found on the curve is 80 MPa, and this 80 MPa is the dynamic estimate of the tissue response elastic modulus in this area.

[0115] Step S1344, based on the duration of the high stress core area at each time stamp in the stress distribution map, the proportion of the high stress core area in the total surgical phase duration is counted to generate an initial value of the contact time proportion, and according to the deformation excess risk index of the tissue area boundary in the biological tissue segmentation result, the credibility weight of the initial value of the contact time proportion is adjusted to generate the contact time proportion.

[0116] During the coronary bypass surgery, the duration of the high stress core area at each time stamp in the stress distribution map is counted. Assuming that the total duration of the operation phase is 60 minutes, and the high stress core area lasts for 10 minutes in a certain period of time, the initial value of the contact time ratio is 10÷60≈0.167. The deformation over-limit risk index of the coronary artery tissue region boundary is obtained from the biological tissue segmentation result, and it is assumed that the deformation over-limit risk index is 0.3 (the deformation over-limit risk index is previously calculated based on factors such as tissue deformation amplitude). If the deformation over-limit risk index is high, it means that the tissue may be in an unstable state, which will reduce the credibility weight of the initial value of the contact time ratio. Assuming that the weight adjustment coefficient determined according to the deformation over-limit risk index is 0.8 (the weight adjustment coefficient is predetermined based on the relationship between the risk index and the contact time ratio), the adjusted contact time ratio is 0.167×0.8 = 0.134.

[0117] Step S1345, align the instrument operation force, tissue response elastic modulus and contact time ratio according to the timestamp, generate a quantitative indicator time series table, and match the predefined surgical operation mode feature library according to the fluctuation frequency of each indicator in the quantitative indicator time series table to generate the abnormal fluctuation range of the instrument operation force, the stable threshold of the tissue response elastic modulus and the contact time ratio safety range.

[0118] In the scenario of heart bypass surgery, the instrument operation force, tissue response elastic modulus and contact time ratio at each timestamp are arranged in chronological order to form a quantitative indicator time series table. For example, at timestamp t1, the instrument operation force is 18, the tissue response elastic modulus is 80 MPa, and the contact time ratio is 0.134; there are corresponding values ​​at timestamp t2, etc. Analyze the fluctuation frequency of each indicator in the time series table. For the instrument operation force, the fluctuation of its value within a certain time range (such as 10 timestamps) is counted. If the fluctuation amplitude is large and frequent, determine its fluctuation frequency. According to the predefined surgical operation mode feature library, the surgical operation mode feature library contains information such as the fluctuation range of each indicator under normal heart bypass surgery. By matching the surgical operation mode feature library, the abnormal fluctuation range of the instrument operation force is determined. For example, if the normal fluctuation range of the instrument operation force is set to 15-20, then the abnormal fluctuation range may be less than 15 or greater than 20. For the tissue response elastic modulus, determine its stable threshold, for example, under normal circumstances, the tissue response elastic modulus should be between 70 and 90 MPa, and this 70 to 90 MPa is the stable threshold. For the contact time ratio, determine its safe range, for example, between 0.1 and 0.2 is the safe range.

[0119] Step S1346, based on the abnormal fluctuation range, stability threshold and safety range, the quantitative indicator time series table is segmented and marked to generate a set of quantitative indicators with risk level marks.

[0120] In heart bypass surgery, the quantitative indicator time series table is segmented and marked according to the abnormal fluctuation range of the instrument operation force, the stable threshold of the tissue response elastic modulus, and the safe range of the contact time ratio determined previously. If the instrument operation force exceeds the abnormal fluctuation range, the tissue response elastic modulus exceeds the stable threshold, or the contact time ratio exceeds the safe range within a certain period of time, different risk levels are marked according to the degree of excess and the potential risk impact on the operation. For example, if the instrument operation force exceeds the abnormal fluctuation range by a large amount, the tissue response elastic modulus also deviates from the stable threshold by a large amount, and the contact time ratio exceeds the safe range by a large amount, it is marked as a high risk level; if only one of the indicators slightly exceeds the corresponding range, it is marked as a low risk level. In this way, a set of quantitative indicators with risk level marks is generated, which can intuitively reflect the operational risk situation at each stage of heart bypass surgery, and provide an important basis for subsequent surgical monitoring and decision-making.

[0121] In a possible implementation, step S140 includes:

[0122] Step S141, extracting the time series characteristic sequence of the instrument operation force, tissue deformation rate and stress distribution spectrum in the dynamic operation state parameters.

[0123] During the heart bypass surgery, relevant data is obtained from the previously generated dynamic operation state parameters. For the instrument operation force, the operation force value at each time point is recorded in chronological order to form a time series feature sequence. For example, at different time points after the start of the operation, such as 15 units at 1 minute, 18 units at 2 minutes, 20 units at 3 minutes, etc., this arrangement in sequence constitutes the time series feature sequence of the instrument operation force. Similarly, for the tissue deformation rate, the deformation rate of the coronary artery at each time point is recorded, such as the deformation rate at the beginning is 0.1 mm / min, and then becomes 0.15 mm / min, etc., to form a time series feature sequence of the tissue deformation rate. For the stress distribution map, according to the relevant characteristics of the stress distribution at each time point, such as the stress accumulation intensity level, the size of the high stress core area, etc., the time series feature sequence of the stress distribution map is constructed.

[0124] Step S142, according to the predefined surgical stage safety threshold, the temporal feature sequence is segmented into time window segments matching the current surgical stage.

[0125] In heart bypass surgery, the entire surgical process is divided into different stages, such as the vascular freeing stage, the vascular anastomosis stage, etc. Each stage has a corresponding predefined safety threshold. Assuming that the current stage is the vascular anastomosis stage, the predefined time window for the vascular anastomosis stage is 3 minutes. Then from the above time series feature sequence, it is divided into segments of 3 minutes. For example, 0-3 minutes from the start of the operation is the first time window segment, 3-6 minutes is the second time window segment, and so on.

[0126] Step S143, performing a difference operation between the average value of the instrument operation force in each time window segment and the upper limit value of the force in the safety threshold of the surgical stage to generate a force deviation coefficient.

[0127] For example, in the vascular anastomosis stage, assume that the upper limit of the instrument operation force in the safety threshold is 20 units. For the first time window segment (0 - 3 minutes), calculate the average of the instrument operation force in the time window segment. For example, if the operation forces in the time window segment are 18 units, 19 units, and 20 units, respectively, then the average is (18 + 19 + 20) ÷ 3 = 19 units. Then perform the difference operation, and the force deviation coefficient is (19 - 20) ÷ 20 = - 0.05 (here the difference is calculated and divided by the upper limit to obtain the relative deviation degree).

[0128] Step S144, synchronously comparing the peak value of the tissue deformation rate within the time window segment with the deformation rate threshold in the surgical stage safety threshold to generate a rate exceeding mark.

[0129] For example, in the vascular anastomosis stage, assuming that the deformation rate threshold is 0.2 mm / min. In the first time window segment (0 - 3 minutes), check the maximum value of the tissue deformation rate. If the maximum deformation rate is 0.18 mm / min, because 0.18 is less than 0.2, the rate exceeding flag is 0, indicating that it is within the limit; if the maximum deformation rate is 0.25 mm / min, because 0.25 is greater than 0.2, the rate exceeding flag is 1, indicating that it is exceeded.

[0130] Step S145 , calculating the overlap rate between the coverage area of ​​the stress distribution map and the contact area threshold in the surgical stage safety threshold, and generating an area abnormality ratio value.

[0131] For example, in the vascular anastomosis stage, assuming that the contact area threshold is 10 square millimeters. In the stress distribution map corresponding to the first time window segment, the coverage area of ​​the stress distribution map is 12 square millimeters. The overlap rate is calculated (here, it actually calculates the proportion of the excess part), and the area abnormality ratio value is (12 - 10) ÷ 10 = 0.2, indicating that the coverage area of ​​the stress distribution map exceeds the contact area threshold by 0.2.

[0132] Step S146, input the force deviation coefficient, rate excess flag and area abnormality ratio value into a pre-trained multi-level decision tree model, wherein the first-level decision node determines whether the force deviation coefficient exceeds the first risk critical value, the second-level decision node generates a tissue cumulative damage index according to the number of triggering times of the rate excess flag, and the third-level decision node generates a contact abnormality pattern based on the area abnormality ratio value and the deformation spatial distribution characteristics.

[0133] Assume that the first risk threshold is 0.1, and the force deviation coefficient calculated above is -0.05. Since -0.05 is less than 0.1, the trigger condition is not met at the first-level decision node. The second-level decision node generates a tissue cumulative damage index based on the number of times the rate exceeding flag is triggered. If the rate exceeding flag is triggered multiple times during the entire operation (for example, the rate exceeding occurs multiple times in different time window segments), the tissue cumulative damage index is assumed to increase by 0.1 each time it exceeds the standard. If there have been two times of exceeding the standard before, and the standard is exceeded again in the current time window segment, the tissue cumulative damage index becomes 0.3. The third-level decision node generates a contact anomaly mode based on the area anomaly ratio value and the deformation spatial distribution characteristics. For example, based on the area anomaly ratio value of 0.2 and the spatial distribution of stress in the stress distribution map (such as the high stress core area is concentrated on one side of the coronary artery and other spatial distribution characteristics), the contact anomaly mode is judged to be excessive local contact pressure.

[0134] Step S147, based on the intermediate judgment result output by the multi-level decision tree model, activate the corresponding abnormal event type label, and the abnormal event type label includes at least three types: instrument overpressure, tissue overspeed deformation and abnormal contact area.

[0135] For example, since it is found in the previous judgment that there is a contact abnormality mode in which the local contact pressure is too large, the abnormal event type label of abnormal contact area is activated.

[0136] Step S148, obtaining in real time the amount of change in the regional boundary pixels in the biological tissue segmentation result, and generating a tissue integrity destruction signal when a continuous expansion or contraction of the boundary pixels is detected.

[0137] For example, in heart bypass surgery, the change in the regional boundary pixels of the coronary artery is continuously concerned from the biological tissue segmentation results. If the regional boundary pixels of the coronary artery are found to shrink continuously within a certain period of time, for example, the number of boundary pixels decreases continuously at several consecutive time points, this indicates that the coronary artery may be over-compressed, thereby generating a signal of tissue integrity damage.

[0138] Step S149, performing a logical AND operation on the tissue integrity destruction signal and the activated abnormal event type tag. If there is an overlapping interval between the two in terms of timestamps, the risk level of the corresponding abnormal event type is increased by one level.

[0139] For example, assuming that the abnormal contact area abnormal event type label has been activated, and within a certain time interval, the time period corresponding to the abnormal event type label overlaps with the time period when the tissue integrity destruction signal is generated (for example, both are within the time period of 5-8 minutes), then the risk level of the abnormal contact area abnormal event type will be increased by one level.

[0140] Step S1410, binding the final abnormal event type with the enhanced risk level according to the level mapping rule set in the safety threshold of the operation stage.

[0141] For example, in the safety threshold setting of heart bypass surgery, there are specific level mapping rules for the vascular anastomosis stage. For example, risk level 1 means that slight attention is needed, risk level 2 means that the operation needs to be adjusted, and risk level 3 means that emergency treatment is needed. If the risk level of the abnormal contact area abnormal event type is raised to level 2 after the previous judgment and processing, then the abnormal contact area is bound to risk level 2 according to the level mapping rules, so that corresponding measures can be taken for this abnormal situation in subsequent surgical monitoring and operation adjustments.

[0142] In a possible implementation, step S150 includes:

[0143] Step S151, locating the abnormal area in the real-time image sequence according to the tissue area boundary and the coordinates of the instrument contact point corresponding to the abnormal event type, and generating a color-coded contour line and text label matching the risk level.

[0144] In this embodiment, in a heart bypass surgery, if the abnormal event type is an abnormal contact area, the corresponding tissue area boundary (such as the boundary coordinates of a specific part of the coronary artery) and the instrument contact point coordinates (such as the contact point coordinates of the vascular clamp and the coronary artery) are obtained from the previous biological tissue segmentation results and surgical instrument recognition results. In the real-time image sequence, the abnormal area is accurately located based on these coordinates. Different color-coded contours are set for different risk levels. For example, when the risk level is level 1, the abnormal area is marked with a yellow color-coded contour line, indicating that attention is needed; when the risk level is level 2, an orange contour line is used, which means that the situation is more serious and needs close attention; when the risk level is level 3, a red contour line is used to indicate an emergency. At the same time, a corresponding text label is generated, such as "Abnormal Contact Area-Risk Level 2", which clearly indicates the type and severity of the abnormality. The text label is placed close to the abnormal area to facilitate quick identification by surgical personnel.

[0145] Step S152: generating a multi-level voice alarm signal based on the alarm priority corresponding to the risk level.

[0146] For example, in the monitoring of heart bypass surgery, the risk level is associated with the alarm priority. For risk level 1, the alarm priority is low, and a relatively mild voice alarm signal may be generated, such as a "ding" prompt sound and a voice broadcast of "abnormal contact area, please pay attention". When the risk level is level 2, the alarm priority increases, and the voice alarm signal becomes a more rapid "di di" sound, and the voice broadcast "abnormal contact area, the situation is more serious, please pay close attention". When the risk level is level 3, the alarm priority is the highest, and a continuous and rapid "di di di" alarm sound will be issued, and the voice broadcast "abnormal contact area, emergency, please deal with it immediately" will be issued. This multi-level voice alarm signal can remind surgical personnel in a timely manner according to the severity of the risk.

[0147] Step S153, matching the instrument path adjustment rules and force optimization parameters in the predefined surgical operation correction solution library according to the abnormal event type, wherein the instrument overpressure type generates a reverse displacement vector arrow and a pressure decrease curve, the tissue overspeed deformation type generates an area isolation mark and an instrument hovering instruction, and the abnormal contact area type generates a contact point migration path and an angle calibration guide.

[0148] For example, if the abnormal event type is device overpressure, the corresponding device path adjustment rules and force optimization parameters are found in the predefined surgical operation correction solution library. The device path adjustment rules may generate a reverse displacement vector arrow based on the contact point between the vascular clamp and the coronary artery and the direction of overpressure. For example, if the vascular clamp applies excessive pressure to the coronary artery in a certain direction, the reverse displacement vector arrow indicates that the vascular clamp should move a certain distance in the opposite direction. At the same time, a pressure reduction curve is generated to show how the pressure should gradually decrease to a safe range over time or moving distance if the correct operation is performed. For tissue overspeed deformation type, regional isolation marks and device hovering instructions are generated from the correction solution library. The regional isolation mark is used to mark the tissue area where overspeed deformation occurs on the image, reminding the surgical staff to avoid continuing to operate in this area; the device hovering instruction informs the surgical staff to stop the current device operation to prevent further damage to the tissue. If the abnormal event type is an abnormal contact area type, a contact point migration path and angle calibration guide are generated from the correction solution library. The contact point migration path indicates that the contact point of instruments such as vascular clamps should be moved to a more appropriate position, and the angle calibration guide gives the correct angle range of the instrument relative to the tissue to ensure normal surgical operation.

[0149] Step S154, synchronously transmit the color-coded contour lines, text labels, voice alarm signals and correction guidance parameters to the display terminal of the surgical navigation system, divide the terminal interface into a first display area for real-time rendering of the original image stream, a second display area for superimposing abnormal contours and correction guidance layers, and a third display area for scrolling updates of risk levels and voice-to-text logs.

[0150] On the display terminal of the heart bypass surgery navigation system, the first display area focuses on real-time rendering of the original image stream, fully displaying the surgical scene, including real-time operation images of instruments such as the heart, coronary arteries, and vascular clamps, so that the surgeon can intuitively see the actual progress of the operation. The second display area superimposes correction guidance layers such as abnormal areas marked by color-coded contour lines, reverse displacement vector arrows, area isolation markers, and contact point migration paths on the basis of the original image. The surgeon can directly see the location of the abnormal situation and the corresponding correction suggestions in this area. The third display area scrolls and updates the risk level. For example, it is displayed in time when it is updated from level 1 to level 2. At the same time, the voice alarm signal is converted into text form and recorded to form a log, which is convenient for the surgeon to review the risk changes during the operation.

[0151] Step S155, based on the dynamic changes in the progress of the surgical stage and the risk level, the contour line color intensity and correction guidance parameters in the second display area are updated in real time. When the risk level of the same abnormal event type is detected to be increased, the contour line of the corresponding area is switched to a higher priority color and the display size of the correction guidance arrow is enlarged.

[0152] During coronary bypass surgery, the risk level may change as the surgical stage progresses, such as from the vascular free stage to the vascular anastomosis stage. If during the vascular anastomosis stage, the risk level of a previously marked abnormal contact area with a risk level of 2 is raised to level 3 due to certain operations. At this time, in the second display area, the orange outline corresponding to the abnormal contact area will switch to a red outline with enhanced color intensity to more prominently display the emergency situation. At the same time, if it is an instrument overpressure type and there is a reverse displacement vector arrow, the display size of the vector arrow will be enlarged to make it more eye-catching, allowing the surgeon to more easily notice the changes in the correction guidance, thereby adjusting the surgical operation in a timely manner to ensure the safety and effectiveness of the operation.

[0153] Figure 2 The schematic diagram shows exemplary hardware and software components of a machine vision-based surgical process monitoring system 100 that can implement the concept of the present application provided by some embodiments of the present application. For example, the processor 120 can be used in the machine vision-based surgical process monitoring system 100 and used to perform the functions in the present application.

[0154] The machine vision-based surgical process monitoring system 100 can be a general-purpose server or a special-purpose server, both of which can be used to implement the machine vision-based surgical process monitoring method of the present application. Although only one server is shown in the present application, for convenience, the functions described in the present application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0155] For example, the machine vision-based surgical process monitoring system 100 may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and storage media 140 in different forms, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the machine vision-based surgical process monitoring system 100 may also include program instructions stored in ROM, RAM, or other types of non-temporary storage media, or any combination thereof. The method of the present application can be implemented according to these program instructions. The machine vision-based surgical process monitoring system 100 also includes an input / output (I / O) interface 150 between the computer and other input / output devices.

[0156] For ease of explanation, only one processor is described in the machine vision-based surgical process monitoring system 100. However, it should be noted that the machine vision-based surgical process monitoring system 100 in the present application may also include multiple processors, so the steps performed by one processor described in the present application may also be performed jointly or individually by multiple processors. For example, if the processor of the machine vision-based surgical process monitoring system 100 executes step A and step B, it should be understood that step A and step B may also be performed jointly by two different processors or individually in one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor execute steps A and B together.

[0157] In addition, an embodiment of the present invention further provides a readable storage medium, in which computer executable instructions are preset. When a processor executes the computer executable instructions, the above-mentioned surgical process monitoring method based on machine vision is implemented.

[0158] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, various features are sometimes combined into one embodiment, drawing or description thereof.

Claims

1. A surgical process monitoring method based on machine vision, characterized in that: The method comprises: Acquire a real-time image sequence of a surgical scene, wherein the real-time image sequence includes dynamic images of a surgical instrument operation area and a biological tissue area; Extracting surgical instrument recognition results and biological tissue segmentation results from the real-time image sequence, wherein the surgical instrument recognition results include instrument type and spatial posture, and the biological tissue segmentation results include tissue type and region boundary; Based on the surgical instrument recognition result and the biological tissue segmentation result, monitoring the operation trajectory of the surgical instrument and the deformation parameters of the biological tissue area, and generating dynamic operation state parameters; According to the predefined safety threshold of the surgical stage, multi-modal abnormality detection is performed on the dynamic operation state parameters to determine the abnormal event type and risk level; Based on the abnormal event type and risk level, a visual surgery monitoring report is generated, and the visual surgery monitoring report includes abnormal area marking, risk warning information and correction suggestions.

2. The method for monitoring a surgical process based on machine vision according to claim 1, characterized in that: The step of extracting surgical instrument recognition results and biological tissue segmentation results from the real-time image sequence includes: Performing key frame sampling on the real-time image sequence to obtain multiple groups of surgical scene static images; Extracting feature regions from each group of static images of the surgical scene to obtain candidate frames of instrument operation regions and candidate frames of tissue regions; Performing multi-scale feature matching on the candidate frame of the instrument operation area based on a three-dimensional convolutional network, determining the instrument type and spatial posture, and generating the surgical instrument recognition result; Performing pixel-level classification on the tissue region candidate frame through a semantic segmentation model to obtain the tissue type and region boundary, and generating the biological tissue segmentation result; The surgical instrument recognition result and the biological tissue segmentation result are aligned in time and space to form a unified spatial coordinate system mapping relationship.

3. The method for monitoring a surgical procedure based on machine vision according to claim 2, characterized in that: The feature region extraction is performed on each group of the surgical scene static images to obtain the instrument operation region candidate frame and the tissue region candidate frame, including: Performing grayscale equalization processing on the static image of the surgical scene to enhance the contrast difference between the instrument operation area and the biological tissue area; The pixel area whose gradient amplitude exceeds a preset threshold in the image after grayscale equalization processing is extracted by an edge detection algorithm to generate an initial candidate frame set; Executing a region growing algorithm on the initial candidate frame set, merging candidate frames with overlapping spatial positions and similar texture features, and generating a merged candidate frame set; According to a preset instrument size range and tissue size range, candidate frames that meet corresponding size constraints are selected from the merged candidate frame set to obtain a preliminarily selected candidate frame set; The morphological closing operation is performed on the candidate frame set after the preliminary screening to fill the broken area inside the candidate frame and smooth the jagged edges of the candidate frame to generate the final instrument operation area candidate frame and biological tissue area candidate frame.

4. The method for monitoring a surgical process based on machine vision according to claim 1, characterized in that: The step of monitoring the operation trajectory of the surgical instrument and the deformation parameters of the biological tissue region based on the surgical instrument recognition result and the biological tissue segmentation result to generate dynamic operation state parameters includes: According to the spatial posture in the surgical instrument recognition result, the moving direction and speed of the instrument under continuous time stamps are calculated to generate the operation trajectory; Based on the regional boundaries in the biological tissue segmentation result, analyzing the tissue deformation amplitude and deformation rate between adjacent timestamps to generate the deformation parameters; The operation trajectory and the deformation parameter are integrated to calculate a stress distribution map of the contact area between the surgical instrument and the tissue; Generating quantitative indicators of instrument operation force, tissue response elastic modulus and contact time ratio according to the stress distribution map; The quantitative index is matched with a preset surgical stage model, and the dynamic operation state parameter is output.

5. The method for monitoring a surgical process based on machine vision according to claim 4, characterized in that: The step of analyzing the deformation amplitude and deformation rate of the tissue between adjacent timestamps based on the regional boundaries in the biological tissue segmentation result to generate the deformation parameters includes: Extracting a tissue region boundary coordinate sequence corresponding to adjacent timestamps in the biological tissue segmentation result, matching the boundary coordinates of a previous timestamp with the boundary coordinates of a subsequent timestamp point by point, and generating a displacement vector for each coordinate point; Calculating the overall displacement mean of the tissue region boundary between adjacent time stamps according to the length of the displacement vector to generate the tissue deformation amplitude; Based on the rate of change of the displacement vector at continuous timestamps, the displacement acceleration of each coordinate point is calculated, the coordinate point whose acceleration exceeds a preset acceleration threshold is marked as a deformation mutation point, the distribution density of the deformation mutation point within the boundary of the tissue region is counted, and the local peak value of the tissue deformation rate is generated according to the ratio of the distribution density to the acceleration threshold; Performing normalized weighted fusion on the tissue deformation amplitude, deformation mutation point distribution density and local peak value of tissue deformation rate to generate a deformation parameter set; According to the tissue type in the biological tissue segmentation result, a predefined tissue elastic property library is matched to extract the deformation tolerance coefficient of the corresponding tissue type; The tissue deformation amplitude in the deformation parameter set is proportionally calculated with the deformation tolerance coefficient to generate a deformation over-limit risk index, and according to the temporal variation trend of the deformation over-limit risk index, the exponential increments of adjacent timestamps are compared with a predefined stage deformation increment threshold to generate a deformation parameter dynamic abnormality flag; The deformation parameter set, deformation over-limit risk index and deformation parameter dynamic anomaly flag are spatially superimposed to form a deformation parameter thermodynamic layer with a timestamp mark, and based on the parameter change direction of continuous timestamps in the deformation parameter thermodynamic layer, a tissue deformation propagation path prediction trajectory is generated; According to the spatial distance between the deformation propagation path prediction trajectory and the surgical instrument operation trajectory, the dynamic weight coefficient of the deformation parameter is adjusted to generate a final deformation parameter set.

6. The method for monitoring a surgical process based on machine vision according to claim 4, characterized in that: The step of fusing the operation trajectory and the deformation parameter to calculate a stress distribution map of a contact area between the surgical instrument and the tissue includes: Extracting the three-dimensional spatial coordinate sequence corresponding to the tip of the instrument at each time stamp according to the moving direction and speed of the instrument at the continuous time stamps in the operation trajectory; Based on the tissue deformation amplitude and deformation rate between adjacent time stamps in the deformation parameters, extracting the deformation region boundary coordinate sequence of the tissue surface at the corresponding time stamps; Performing time synchronization to align the three-dimensional spatial coordinate sequence of the instrument tip with the coordinate sequence of the boundary of the deformation region, and generating a mapping relationship between the synchronized instrument contact point position and the tissue deformation region; According to the synchronized instrument contact point position, locate the contact area space range between the instrument tip and the tissue surface in the deformation area boundary coordinate sequence, and calculate the deformation acceleration distribution of the contact area in the instrument movement direction based on the tissue deformation rate at each time stamp within the contact area space range; Determine the stress direction vector of each sub-area within the contact area according to the spatial angle between the deformation acceleration distribution and the moving direction of the instrument, fuse the stress direction vector with the spatial gradient of the tissue deformation amplitude, and generate a dynamic stress propagation path; The stress distribution map is generated according to the coverage and direction consistency of the dynamic stress propagation path under continuous time stamps, wherein different color areas in the stress distribution map represent stress accumulation intensity levels.

7. The method for monitoring a surgical procedure based on machine vision according to claim 4, characterized in that: Generating quantitative indicators of instrument operation force, tissue response elastic modulus and contact time ratio according to the stress distribution map includes: Extracting the stress cumulative intensity level of the color-coded area in the stress distribution map, marking the continuous area whose color depth exceeds the preset intensity level as a high stress core area, dividing the grid units in the high stress core area, counting the consistency ratio of the stress direction vector in each grid unit, and merging the grid units whose consistency ratio exceeds the preset threshold value into isotropic stress belts; Based on the product of the coverage area of ​​the isotropic stress band and the stress cumulative intensity level, an initial estimate of the instrument operation force at each time stamp is generated, and according to the instrument type in the surgical instrument recognition result, a predefined instrument rigidity coefficient is matched, and the initial estimate of the instrument operation force and the rigidity coefficient are weightedly corrected to generate the instrument operation force; Extracting the deformation recovery rate corresponding to the tissue deformation parameter in the biological tissue segmentation result, calculating the attenuation ratio of the deformation recovery rate and the stress cumulative intensity level in the same area in the stress distribution map, and determining the dynamic estimation of the tissue response elastic modulus according to the attenuation ratio and a predefined tissue elastic attenuation curve; Based on the duration of the high stress core area at each time stamp in the stress distribution map, the proportion of the high stress core area in the total duration of the operation phase is counted to generate an initial value of the contact time proportion, and according to the deformation over-limit risk index of the tissue region boundary in the biological tissue segmentation result, the credibility weight of the initial value of the contact time proportion is adjusted to generate the contact time proportion; The instrument operation force, tissue response elastic modulus and contact time ratio are aligned according to timestamps to generate a quantitative indicator time series table, and according to the fluctuation frequency of each indicator in the quantitative indicator time series table, a predefined surgical operation mode feature library is matched to generate an abnormal fluctuation range of the instrument operation force, a stable threshold of the tissue response elastic modulus and a safe range of the contact time ratio; Based on the abnormal fluctuation interval, stability threshold and safety range, the quantitative indicator time series table is segmented and marked to generate a set of quantitative indicators with risk level marks.

8. The method for monitoring a surgical procedure based on machine vision according to claim 1, characterized in that: The method of performing multi-modal abnormality detection on the dynamic operation state parameters according to the predefined safety threshold of the surgical stage to determine the abnormal event type and risk level includes: Extracting the time series characteristic sequence of the instrument operation force, tissue deformation rate and stress distribution spectrum from the dynamic operation state parameters; According to the predefined surgical stage safety threshold, the temporal feature sequence is segmented into time window segments matching the current surgical stage; Performing a difference operation on the average value of the instrument operation force in each time window segment and the force upper limit value in the safety threshold of the surgical stage to generate a force deviation coefficient; Synchronously comparing the peak value of the tissue deformation rate within the time window segment with the deformation rate threshold in the surgical stage safety threshold to generate a rate exceeding mark; Calculate the overlap rate of the coverage area of ​​the stress distribution map and the contact area threshold in the surgical stage safety threshold to generate an area abnormality ratio value; The force deviation coefficient, the rate exceeding flag and the area abnormality ratio are input into a pre-trained multi-level decision tree model, wherein the first-level decision node determines whether the force deviation coefficient exceeds the first risk critical value, the second-level decision node generates a tissue cumulative damage index according to the number of triggering times of the rate exceeding flag, and the third-level decision node generates a contact abnormality pattern based on the area abnormality ratio and the deformation spatial distribution characteristics; Based on the intermediate judgment results output by the multi-level decision tree model, the corresponding abnormal event type labels are activated, and the abnormal event type labels include at least three types: instrument overpressure, tissue overspeed deformation and abnormal contact area; Acquiring in real time the amount of change in the region boundary pixels in the biological tissue segmentation result, and generating a tissue integrity destruction signal when a continuous expansion or contraction of the boundary pixels is detected; Performing a logical AND operation on the tissue integrity destruction signal and the activated abnormal event type tag, if there is an overlapping interval between the two in terms of timestamps, the risk level of the corresponding abnormal event type is increased by one level; According to the level mapping rules set in the safety threshold of the surgical stage, the final abnormal event type is bound to the enhanced risk level.

9. The method for monitoring a surgical process based on machine vision according to claim 1, characterized in that: The generating of a visual surgery monitoring report based on the abnormal event type and risk level includes: Locating the abnormal area in the real-time image sequence according to the tissue area boundary and the coordinates of the instrument contact point corresponding to the abnormal event type, and generating a color-coded contour line and text label matching the risk level; Generate a multi-level voice alarm signal based on the alarm priority corresponding to the risk level; Match the instrument path adjustment rules and force optimization parameters in the predefined surgical operation correction solution library according to the abnormal event type, wherein the instrument overpressure type generates a reverse displacement vector arrow and a pressure reduction curve, the tissue overspeed deformation type generates a regional isolation mark and an instrument hovering instruction, and the abnormal contact area type generates a contact point migration path and an angle calibration guide; The color-coded contour lines, text labels, voice alarm signals and correction guidance parameters are synchronously transmitted to a display terminal of a surgical navigation system, and the terminal interface is divided into a first display area for real-time rendering of the original image stream, a second display area for superimposing abnormal contours and correction guidance layers, and a third display area for rolling updates of risk levels and voice-to-text logs; According to the dynamic changes in the progress of the surgical stage and the risk level, the contour line color intensity and correction guidance parameters in the second display area are updated in real time. When the risk level of the same abnormal event type is detected to be increased, the contour line of the corresponding area is switched to a higher priority color and the display size of the correction guidance arrow is enlarged.

10. A surgical process monitoring system based on machine vision, characterized in that: The machine vision-based surgical process monitoring system includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the machine vision-based surgical process monitoring method described in any one of claims 1 to 9.

Images