Microsurgery auxiliary system based on visual identification

Abstract

The invention relates to the technical field of computer vision, in particular to a microsurgery auxiliary system based on visual identification, which comprises an instrument target positioning module, a behavior characteristic analysis module, a fuzzy dynamic tracking module, an action trend analysis module and a path response correction module. According to the method, the closed boundary region is extracted and the image center offset is combined for judgment, so that the center of the end part of the instrument can be accurately positioned, the spatial perception precision of the key points of the surgical field is improved, the behavior characteristic indexes are constructed in combination with the angle change rate and the included angle amplitude, and the recognition accuracy of complex actions is enhanced; by analyzing the difference between the image gray gradient and the edge direction, extracting a fuzzy region and tracking the motion trend, the dynamic capturing capability of an instrument in a low-definition environment is improved, the direction change frequency is counted to extract a repeated trend sequence, the path deviation is corrected, and the action coherence and controllability are enhanced. And the reliability of intelligent response and path planning of instrument control in a micromanipulation scene is integrally improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and particularly to a microsurgical assistance system based on visual recognition. Background Art

[0002] The field of computer vision technology includes research and applications in aspects such as image acquisition, image processing, image recognition and understanding. It mainly perceives and analyzes image or video information in the real world by simulating human vision. The core content of this technical field includes the use of image acquisition devices, the extraction of image features, object detection and segmentation, 3D reconstruction, scene understanding, etc. Computer vision is widely applied in scenarios such as autonomous driving, medical image analysis, face recognition, industrial inspection, and intelligent monitoring. It relies on the collaborative work of image sensors, neural network models, and image processing algorithms to extract structured information from raw visual data and achieve automatic perception and judgment of the environment.

[0003] Among them, a microsurgical assistance system refers to a system that assists in microsurgical operations based on computer vision technology, mainly used to improve the operation accuracy and visualization degree during the operation. This system covers key technical matters such as high-resolution acquisition of intraoperative images, image registration processing, automatic recognition and tracking of the microsurgical operation area, and three-dimensional structure reconstruction of the surgical area. It accurately extracts the boundary information of the surgical target area by using image segmentation methods, obtains the relative position changes between the surgical instruments and tissues through image sequence analysis, and uses a deep learning model to automatically classify and locate key organizational structures during the operation, thereby supporting microsurgical doctors to perform operation guidance and auxiliary decision-making in a limited visual field environment.

[0004] When the prior art is applied to the microsurgical assistance scenario, although it has the ability of image acquisition and recognition, when dealing with the evolution trend of instrument behavior in dynamic continuous image frames, it lacks targeted time series analysis means and cannot effectively identify the continuous and trend features in instrument actions, resulting in problems such as fuzzy judgment or response delay in behavior judgment. In scenarios where there are blurred interferences in the surgical field image and the tissue boundaries are not clear, the prior methods have insufficient ability to identify and dynamically track local abnormal areas in the image, causing instrument position recognition drift and path tracking interruption, affecting the stability and safety of the intraoperative assistance system. The existing systems rely on static image features for operation guidance and lack a long-term behavior modeling mechanism based on the change trend of the instrument movement direction, making the judgment accuracy of repetitive operation paths not high and easily ignoring the cumulative error of subtle path deviations. For example, during a surgical operation with a complex microsurgical field and variable tissue morphology, the prior methods are difficult to achieve stable tracking of continuous operations of the instrument in the blurred area, restricting the effective response of the system to the actual operation intention and affecting the auxiliary ability for doctors' fine operations. Summary of the Invention

[0005] The object of the present invention is to solve the disadvantages existing in the prior art, and a microsurgical assistance system based on visual recognition is proposed.

[0006] To achieve the above object, the present invention adopts the following technical solutions: The microsurgical assistance system based on visual recognition includes: The instrument target positioning module extracts the closed boundary region based on the continuous frame images of the surgical field under the microscope, judges the boundary closure degree and the image center offset value, locates the center position of the instrument end in the image, and generates the instrument end spatial coordinate set; The behavior feature analysis module analyzes the angular change rate and the included angle amplitude of the connection line of the end points in the continuous frames according to the instrument end spatial coordinate set, judges the corresponding behavior type of the line segment fragment, and generates the instrument action behavior state group; The fuzzy dynamic tracking module calls the instrument action behavior state group, extracts the image of the microsurgical tissue boundary region, identifies the distance between the surgical instrument and the tissue boundary, records the change amplitude of adjacent frames, analyzes the gray scale gradient and the difference in the instrument edge direction in the jump region, screens the regions with high fuzzy degree, and tracks the movement direction of the instrument end in the region to generate the fuzzy region motion alignment direction set; The action trend analysis module extracts the motion direction line segments based on the fuzzy region motion alignment direction set, identifies the included angle values of adjacent direction line segments and counts the occurrence frequencies, screens the continuous direction sequences, and obtains the repeated trend direction intervals.

[0007] As a further solution of the present invention, the instrument end spatial coordinate set includes spatial position information, image reference offset, and center indication parameters. The instrument action behavior state group includes behavior recognition labels, angular change indicators, and motion amplitude parameters. The fuzzy region motion alignment direction set includes direction change features, fuzzy edge attributes, and region dynamic indicators. The repeated trend direction interval includes continuous direction patterns, included angle change distributions, and trend stability parameters.

[0008] As a further solution of the present invention, the instrument target positioning module includes: The image sequence extraction sub-module identifies the image data of each frame based on the continuous frame images of the surgical field under the microscope, detects the brightness gradient regions in each frame image, integrates the gray scale difference and the texture distribution change, and obtains the image dynamic sequence frame set; The boundary judgment analysis sub-module calls the image dynamic sequence frame set, extracts the closed boundary according to the pixel edge continuity, identifies the boundary contour point set, analyzes the closure degree and the connection integrity, measures the distance between the contour centroid and the image center and classifies it, analyzes the boundary closure degree and the center offset, and uses the formula: ; Calculate the offset correlation value, sort the offset correlation values in each frame of image, and screen the front regions to obtain the effective boundary positioning interval value; Among them, represents the offset correlation value, represents the pixel coordinates of the centroid of the contour in the th frame, represents the pixel coordinates of the center point of the image, is the boundary closure value of the th frame, represents the offset pixel value of the candidate region , is the full-frame offset reference pixel value, is the number of valid frames, is the number of candidate regions; The spatial coordinate recognition sub-module selects the internal pixel density region of the boundary according to the effective boundary positioning interval value, extracts the geometric center coordinates, combines the imaging parameters and the scale factor to recognize the three-dimensional spatial position, and generates the spatial coordinate set of the instrument end.

[0009] As a further solution of the present invention, the behavior feature analysis module includes: The trajectory speed calculation sub-module selects the end coordinates of two adjacent frames according to the spatial coordinate set of the instrument end, recognizes the spatial distance, and extracts the speed per unit time according to the time interval to obtain the line segment angle change speed value; The included angle amplitude extraction sub-module calls the line segment angle change speed value, identifies the included angle based on the vectors of adjacent line segments among three frames and screens out the segments exceeding the rotation amplitude threshold to obtain the line segment included angle amplitude interval value; The behavior type determination sub-module calculates the line segment action behavior feature metric value according to the line segment angle change speed value, and judges it with the behavior classification boundary value, using the formula: ; Generate the instrument action behavior state group; Among them, represents the line segment action behavior feature metric value, represents the angle change speed, represents the angle change reference value, and respectively represent the upper and lower limits of the included angle amplitude interval, is the stability factor of the behavior , is the number of behaviors.

[0010] As a further solution of the present invention, the fuzzy dynamic tracking module includes: The boundary image extraction sub-module calls the instrument action behavior state group, identifies the current position and posture of the instrument in the microscopic tissue image, extracts the microscopic tissue boundary region image of the instrument in the image, and obtains the instrument boundary registration image set; The jump region screening sub-module analyzes the gray-scale change amplitude between the instrument edge and the tissue boundary in adjacent frames based on the instrument boundary registration image set, screens the image regions with large gray-scale changes and large differences in edge directions, and generates a high-gradient difference region map; The end movement direction extraction sub-module tracks the position of the instrument end in the region frame by frame according to the high-gradient difference region map, extracts the coordinate changes and region direction vectors of the end in consecutive frames, identifies the matching degree and angle increment of the end movement direction, and uses the formula: ; Calculate the fitting value of the region direction corresponding to the end, and generate a fuzzy region motion alignment direction set according to the change trend in the fitting value sequence; Among them, represents the fitting value of the region direction corresponding to the end, represents the frame velocity vector of the instrument end, represents the frame and frame end coordinate difference vector, represents the frame and frame end direction angle change value, represents the frame region direction vector group modulus value sequence, is the number of consecutive tracking frames.

[0011] As a further solution of the present invention, the action trend analysis module includes: The direction line segment extraction sub-module identifies the continuous coordinate sequence of the instrument end movement path based on the fuzzy region motion alignment direction set, connects adjacent coordinates to form a direction line segment, eliminates the line segments with lengths lower than the specified threshold in the trajectory, and obtains a direction line segment vector set; The included angle value statistics sub-module analyzes the included angle value according to the direction vector dot product based on the direction line segment vector set, divides it into multiple intervals, and statistically analyzes the frequency and direction fluctuation of the intervals, using the formula: ; Calculate the included angle interval change value, extract the high-change region according to the index size, and obtain the main included angle trend interval; Among them, represents the change value of the included angle interval, represents the appearance frequency of the included angle interval, represents the The average length of the direction line segments within the included angle range, indicating the frequency change value of adjacent intervals of the included angle range, indicating the fluctuation range of the direction angle within the included angle range, indicating the total number of included angle ranges; The repeated interval extraction sub-module filters the fragment sequences of the direction line segments whose included angle values continuously fall within the interval according to the main included angle trend interval, integrates the distribution segments of the sequences, and obtains the repeated trend direction interval.

[0012] As a further aspect of the present invention, the system further includes a path response correction module: The path response correction module compares the angle of the current instrument direction line segment with the angle of the trend interval according to the repeated trend direction interval, filters the positions of the line segments deviating from the direction and adjusts the movement direction in the corresponding frame to generate an optimized surgical assistance instrument movement path; The optimized surgical assistance instrument movement path includes a corrected movement trajectory, direction optimization parameters, and deviation correction positions.

[0013] As a further aspect of the present invention, the path response correction module includes: The trend interval identification sub-module identifies the continuous direction line segments in the instrument movement trajectory according to the repeated trend direction interval, calculates the angle difference between adjacent line segments, analyzes the continuous interval of the angle difference change, determines the start and end boundaries of the trend direction, and obtains the trend direction interval; The deviation direction detection sub-module extracts the current frame instrument direction line segment based on the trend direction interval, compares it with the angle difference within the trend direction interval, filters the direction line segments that deviate from the threshold, and obtains the deviation direction line segments; The direction correction generation sub-module identifies the adjustment angle through the angle difference according to the deviation direction line segments, corrects the instrument movement direction to be close to the trend direction, and obtains the optimized surgical assistance instrument movement path.

[0014] Compared with the prior art, the advantages and positive effects of the present invention are as follows: In the present invention, by extracting regions with closed boundary features from a microscopic image sequence and combining with the offset of the image center for determination, the center position of the instrument tip can be accurately identified, realizing the spatial precise positioning of key operation points, enhancing the perception accuracy of operation targets in the surgical field, constructing a behavioral feature index based on the angular change rate and included angle amplitude of the tip connection lines in consecutive image frames, strengthening the recognition accuracy of complex action features, effectively distinguishing the subtle changes between instrument operation states, and improving the accuracy and response speed of intraoperative behavior judgment. By combining the image gray gradient and edge direction difference to extract regions with high ambiguity degree, and tracking the movement trend of the instrument in the ambiguous tissue region, the stable capture ability of the instrument dynamics in a low-definition environment is improved, and the decoding ability of the instrument operation intention in a visually interfering region is enhanced. By statistically analyzing the direction change frequency and extracting the repeated trend direction sequence, a regular movement trend pattern is constructed, providing a directional reference for subsequent path judgment and reducing the interference of invalid movement paths. By comparing the deviation between the current direction line segment and the angle of the trend interval and correcting the movement direction, the stability and path consistency of the instrument movement are ensured, the coherence and controllability of the overall operation path are improved, and through the technical linkage of multi-dimensional parameter analysis, enhanced recognition of visually ambiguous regions, extraction of dynamic path rules, and deviation direction adjustment, the intelligent response ability of instrument control, the accuracy of action discrimination, and the reliability of path planning in the microscopic operation scenario are effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is the system flowchart of the present invention; Figure 2 is the flowchart of the instrument target positioning module in the present invention; Figure 3 is the flowchart of the behavioral feature analysis module in the present invention; Figure 4 is the flowchart of the fuzzy dynamic tracking module in the present invention; Figure 5 is the flowchart of the action trend analysis module in the present invention; Figure 6 is the flowchart of the path response correction module in the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0017] In the description of the present invention, it should be understood that the orientation or positional relationships indicated by the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. are based on the orientation or positional relationships shown in the drawings. These are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation on the present invention. In addition, in the description of the present invention, the meaning of "a plurality of" is two or more, unless otherwise specifically defined.

[0018] Please refer to Figure 1 , a microsurgical assistance system based on visual recognition includes: The instrument target positioning module extracts a closed boundary region based on consecutive frame images of the surgical field under the microscope, judges the boundary closure degree and the image center offset value, locates the center position of the instrument end in the image, and generates a set of spatial coordinates of the instrument end; The behavior feature analysis module analyzes the angular change rate and the included angle amplitude of the connecting lines of the end points in consecutive frames according to the set of spatial coordinates of the instrument end, judges the corresponding behavior type of the line segment fragment, and generates a set of instrument action behavior states; The fuzzy dynamic tracking module calls the set of instrument action behavior states, extracts the image of the microsurgical tissue boundary region, identifies the distance between the surgical instrument and the tissue boundary, records the change amplitude of adjacent frames, analyzes the gray gradient and the difference in the instrument edge direction of the jumping region, screens the regions with high fuzzy degree, and tracks the moving direction of the instrument end in the region to generate a set of alignment directions for the movement in the fuzzy region; The action trend analysis module extracts the movement direction line segments based on the set of alignment directions for the movement in the fuzzy region, identifies the included angle values between adjacent direction line segments and counts the occurrence frequencies, screens the continuous direction sequences, and obtains the repeated trend direction intervals; The path response correction module compares the angle of the current instrument direction line segment with the angle of the trend interval according to the repeated trend direction intervals, screens the positions of the line segments deviating from the direction and adjusts the movement direction in the corresponding frames to generate an optimized movement path for the surgical assistance instrument.

[0019] The set of spatial coordinates of the instrument end includes spatial position information, image reference offset, and center indication parameters. The set of instrument action behavior states includes behavior recognition labels, angular change indicators, and movement amplitude parameters. The set of alignment directions for the movement in the fuzzy region includes direction change features, fuzzy edge attributes, and regional dynamic indicators. The repeated trend direction intervals include continuous direction patterns, included angle change distributions, and trend stability parameters. The optimized movement path for the surgical assistance instrument includes corrected movement trajectories, direction optimization parameters, and deviation correction positions.

[0020] Please refer to Figure 2, the instrument target positioning module includes: The image sequence extraction sub-module is based on the continuous frame images of the surgical field under the microscope, identifies the image data of each frame, detects the brightness gradient regions within each frame image, integrates the gray-scale differences and texture distribution changes, and obtains the dynamic sequence frame set of images; Detect the brightness gradient regions within each frame image, analyze according to the changes in the brightness gradient and the continuity of the texture distribution of the image. The key to the process lies in identifying representative image features. For example, during neurosurgery, identifying the edges of blood vessels and nerve tissues can help doctors accurately judge the position of surgical instruments. By calculating the gray-scale differences and texture distribution changes in consecutive frames, the dynamic changes during the operation can be effectively tracked. This analysis is achieved by calculating the differences in the gray-scale histograms between frames. The data is integrated to construct a detailed relationship between the time frame index and the image frame data. The data set is used for further analysis and operations. For example, in practical applications, the data can be used to evaluate the stability and accuracy during the operation. Through high-precision image processing techniques, the dynamic sequence frame set of images is obtained, providing strong visual support for the operation.

[0021] The boundary judgment and analysis sub-module calls the dynamic sequence frame set of images, extracts the closed boundary according to the pixel edge continuity, identifies the set of boundary contour points, analyzes the degree of closure and connection integrity, measures the distance between the center of gravity of the contour and the center of the image and classifies it, analyzes the boundary closure degree and the center offset amount, and uses the formula: ; Calculate the offset correlation degree value, sort the offset correlation degree values in each frame image, and screen the front regions to obtain the effective boundary positioning interval value; Among them, represents the offset correlation degree value, represents the pixel coordinates of the center of gravity of the contour in the th frame, represents the pixel coordinates of the center point of the image, is the boundary closure degree value of the th frame, represents the offset pixel value of the candidate region , is the full-frame offset reference pixel value, is the number of valid frames, is the number of candidate regions; First, perform an edge continuity extraction operation on each frame of the image to obtain the initial contour of the boundary. During the edge extraction process, the Canny operator is selected for pixel gradient scanning. After calculating the gradient magnitude of the image grayscale through convolution, the pixel points above the edge response threshold are recorded as the preliminary boundary point set. For example, select the 12th frame of the intraoperative microscopic image, set the gradient change amplitude threshold to 15, and filter a total of 416 edge pixels to form the preliminary boundary. Subsequently, connect the boundary point sets in order of position, and judge the connection integrity by whether the distance difference between consecutive point pairs is greater than 2 pixels. For each boundary line, use the cubic B-spline fitting method to calculate the curvature, and take the derivative of the tangent vector of the fitted curve, calculate the absolute value of its second derivative and then integrate it as the closed curvature value, which is uniformly normalized to the 0-1 interval range in each frame of the image. For example, the curvature integral value of a certain frame is 238.4, and the closure degree after normalization is 0.87; Using the center coordinate point G of the image plane as the reference point, detect the geometric center coordinates of the contour area of each frame This point is obtained by the arithmetic mean of all pixel coordinates within the contour. For example, if there are 1372 pixel points within the contour, the mean value of the x coordinates of all points is 103.7, and the mean value of the y coordinates is 119.6, then the centroid of this frame of the contour is (103.7, 119.6), the image size is 256×256, and the geometric center G is (128, 128). Calculate the Euclidean distance between the two points to obtain the offset value of this frame Execute this process to obtain the offset distance of each frame; Furthermore, extract the offset pixel values of the candidate regions Take the distance between the centroid of each region and the center of the image as the offset distance, which is also calculated through the Euclidean distance formula, and statistically analyze the average offset within all frames as the reference value Suppose a total of 8 frames of data are analyzed, and their offset distances are 22.1, 25.7, 19.3, 26.9, 24.2, 28.5, 21.8, 27.4, then the offset reference value ; Call the above parameters, substitute them into the formula for calculation, set the current number of processed frames n to 5, and the number of candidate regions m to 3. The corresponding data is as follows: , ; , ; , ; , ; , ; , , , ; Substitute it into the formula: ; Calculate the numerator part of each item respectively: , and we get ; , and we get ; , and we get ; , and we get ; , and we get ; Denominator part calculation: Sum of offset differences ; Denominator ; Substitute all the results, the numerator ; Final calculation: ; This result shows that the comprehensive offset correlation value is 14.47. The higher the value, the more significant the offset of the boundary from the image center. Combining with the sorting and screening, it can be used for further boundary region determination. By fusing and calculating the boundary closure degree and the contour offset position in a weighted manner, and introducing the candidate region offset benchmark to normalize the overall offset influence, it can more reasonably reflect the comprehensive performance of the current frame boundary in two dimensions of spatial offset and structural closure.

[0022] The spatial coordinate recognition sub-module selects the pixel density region inside the boundary according to the effective boundary positioning interval value, extracts the geometric center coordinates, and combines the imaging parameters and scale factor to recognize the three-dimensional spatial position, generating the spatial coordinate set of the instrument end; Obtain the set of pixel points inside the boundary in the corresponding frame image. The process involves the region selection of the image and the precise calculation of pixel points. By selecting the section with the largest density in the boundary point group, its geometric center coordinates can be extracted. The extraction process is realized by calculating the spatial distribution density of each pixel point. Call the index of the corresponding frame of the image and the imaging parameters, combine the optical axis calibration parameters and the image scale factor, and the parameters are all obtained through pre-experiment settings and calibrations. Calculate its position relationship relative to the image plane, which includes a series of geometric transformations and proportional calculations, and obtain its position representation in the three-dimensional space. The position representation is extremely important for accurately navigating the surgical instrument to the target operation area. Establish the spatial coordinate set of the instrument end, and this coordinate set provides the accurate position information of the surgical instrument during the operation, thus ensuring the accuracy and safety of the operation.

[0023] Please refer to Figure 3 , the behavior feature analysis module includes: The trajectory rate calculation sub-module selects the end coordinates of two adjacent frames according to the set of spatial coordinates of the instrument end, identifies the spatial distance, extracts the rate per unit time according to the time interval, and obtains the line segment angle change rate value; First, obtain the end coordinate points in two adjacent frames, use the spatial geometry calculation method to obtain the straight-line distance between the two points, and further calculate the time interval between the two frames. Assume that in a certain instance, the frame interval is 0.02 seconds, and the coordinate points of two adjacent frames are (1, 2, 3) and (4, 6, 8) respectively. Then, the distance between the two points is calculated as 5 units through the spatial distance formula. After dividing by the time interval of 0.02 seconds, the obtained rate is 250 units per second. This rate reflects the linear movement rate of the instrument end, similar to the movement speed of a robotic arm. The calculation of the rate value needs to ensure that the calculated time interval is accurate to reflect the true speed of the instrument movement and ensure that the rate of the line segment fragment will not mislead the behavior analysis due to inaccurate measurement. According to the value of the rate, the coherence and stability of the instrument movement can be further analyzed, and finally the line segment angle change rate value is generated for subsequent included angle analysis.

[0024] The included angle amplitude extraction sub-module calls the line segment angle change rate value, identifies the included angle based on the vectors of adjacent line segments among three frames, and filters out the segments exceeding the rotation amplitude threshold to obtain the line segment included angle amplitude interval value; By calculating the included angle between the vectors of the instrument end, further based on the end vectors of adjacent two line segments within three frames, first calculate the direction vectors of the ends of two adjacent frames. Assume that in a certain instance, the vectors of adjacent line segments are (3, 2, 1) and (5, 6, 2) respectively. Then calculate the inner product of the two vectors, and then calculate the included angle value through the inner product formula. Assume the result is 45 degrees. In order to filter out outliers, a threshold for the included angle amplitude needs to be set. If the calculated included angle exceeds this threshold, the segment will be excluded. In this way, only the segments that meet the conditions are retained. For example, assume that the set included angle threshold is 60 degrees. When the calculated included angle is 45 degrees, this segment meets the conditions, is retained and continues to be analyzed. Finally, the line segment included angle amplitude interval value is generated. This step helps to reduce the influence of noise and ensure that the subsequent determination of the behavior type is more accurate and reliable.

[0025] The behavior type determination sub-module calculates the line segment action behavior feature metric value according to the line segment angle change rate value, and judges it with the behavior classification boundary value, using the formula: ; Generate the instrument action behavior state group; Among them, represents the line segment action behavior feature metric value, represents the angular change rate, represents the angular change reference value, and respectively represent the upper and lower limits of the included angle amplitude interval, is the stability factor of the behavior ; is the number of behaviors; By obtaining the angular change rate and included angle amplitude of each line segment fragment, the behavior characteristic measurement value of the fragment is calculated. Assuming that in a certain embodiment, the angular change rate of the line segment of the instrument is 250 units per second and the included angle amplitude is 45 degrees, and the preset behavior classification formula is used for calculation. The rate value is 250 units per second, and the included angle amplitude is 45 degrees, and respectively represent the upper and lower limits of the included angle amplitude interval of the line segment. Assuming = 60 degrees and = 30 degrees; Substitute the specific values into the formula: ; First, calculate the absolute difference and the square root part: ; ; Then, calculate the numerator part of the characteristic measurement value F: ; Next, assume that the stability factor of this behavior is 0.5 and the number of behavior types is 3. Then the denominator part in the behavior classification formula is calculated as follows: ; Finally, the calculation result of the behavior characteristic measurement value F is: ; According to this value, if the boundary of behavior classification is set to 1.0, then since F = 108.83 is greater than 1.0, this fragment is determined to be a high-frequency motion type. In this way, by combining the behavior characteristic measurement values of each fragment, the actions of the instrument can be classified, and finally an instrument action behavior state group is generated. The unification of units is very important to ensure the dimensional consistency of the rate, angle, and behavior stability factor. The rate is expressed in units / second, the angle is in degrees, and the stability factor is a dimensionless value to ensure the rationality and correctness of the formula calculation. In actual operation, the movement rate of the instrument end should be adjusted according to the real-time collected data, and the selection of the included angle amplitude interval should be set according to the movement characteristics and requirements of the instrument to ensure accurate classification of behavior types.

[0026] Please refer to Figure 4 , the fuzzy dynamic tracking module includes: The boundary image extraction sub-module calls the instrument action behavior state group to identify the current position and posture of the instrument in the microscopic tissue image, and extracts the microscopic tissue boundary region image of the instrument in the image to obtain the instrument boundary registration image set; It is the recognition of the position and posture of the instrument in the microscopic tissue image. Suppose during ophthalmic surgery, through real-time image recognition technology, the position of the surgical instrument is accurately captured, which helps the doctor perform the surgery more precisely, avoid accidentally touching the sensitive tissues of the patient, and the doctor can adjust the surgical strategy by observing the immediate feedback of the instrument in the field of view. The image extraction includes the interaction between the instrument and the tissue boundary. For example, if the instrument is close to the edge tissue of the eyeball, the image edge sharpness of that part is automatically enhanced so that the doctor can see the operation area more clearly. By analyzing the relationship between the edge of the instrument and the surrounding tissues, the instrument boundary registration image set is generated, and the image set can be used for subsequent precise operation guidance.

[0027] The jump region screening sub-module analyzes the gray-scale change amplitude between the instrument edge and the tissue boundary in adjacent frames based on the instrument boundary registration image set, screens the image regions with large gray-scale changes and large edge direction differences, and generates a high-gradient difference region map; First, it involves calculating the gray-scale change amplitude between the instrument edge and the tissue boundary, which can be used to judge whether the instrument is too close to the key tissue, causing potential surgical risks. For example, during neurosurgery, the gray-scale change amplitude between the edge and the tissue boundary can indicate whether the instrument touches the sensitive area. By setting the threshold of the gray-scale change amplitude, such as the threshold is 15, only when the gray-scale change amplitude exceeds this value, the high-risk area is marked. The marking helps the doctor stay alert to the dangerous area during the real-time surgery, generates a high-gradient difference region map, provides a visual aid tool, and adjusts the surgical path according to the marked high-risk area in the map to avoid unnecessary harm to the patient.

[0028] The end movement direction extraction sub-module tracks the position of the instrument end in the region frame by frame according to the high-gradient difference region map, extracts the corresponding coordinate changes and regional direction vectors of the end in consecutive frames, and identifies the matching degree and angle increment of the end movement direction, using the formula: ; Calculate the fitting value of the corresponding regional direction of the end, and generate a fuzzy region motion alignment direction set according to the change trend in the fitting value sequence; Among them, represents the fitting value of the corresponding regional direction of the end, represents the frame velocity vector of the instrument end, represents the frame and frame end coordinate difference vector, Representing the frame and the change value of the end direction angle of the frame, representing the modulus sequence of the regional direction vector group of the frame, is the number of consecutive tracking frames; Perform position tracking operations on the instrument end frame by frame in the regional map. The tracking data is sourced from the microscopic surgery image acquisition sequence, where the interval between each frame of the image is 0.05 seconds to ensure real-time and continuity. Extract the relative displacement of the instrument end in the two-dimensional image coordinate system through the change of the pixel coordinates of the instrument end in consecutive frames to form , for example, the coordinates of the instrument end in the t-th frame are (120, 84), and the coordinates in the t - 1-th frame are (117, 81), then , and the modulus length is pixels. Considering that the image resolution is 1 pixel corresponding to 2 microns, the displacement length is converted to ; Velocity vector is calculated by dividing the end displacement by the time interval between frames, pixels per second, and the modulus length is pixels per second; Direction change amount is calculated by the included angle between the unit vectors of the instrument directions in two consecutive frames. Assuming that the unit vector of the instrument direction in the current frame is (0.707, 0.707), and the previous frame is (0.894, 0.447), and the cosine included angle is then , and the modulus of the regional direction vector is the modulus length of the main direction vector of the gray gradient in the corresponding region within the atlas. Assuming its value is 46.29 pixels per second; Substitute the above items into the formula: ; The calculation results show that in the current consecutive 3 frames, the fitting degree of the instrument end relative to the regional direction is 273.3. The larger the value, the more concentrated the direction change is within the same regional direction range. This result is a representative direction quantity index for the concentration of the blurred area movement alignment direction. Subsequently, such direction quantities of all frame segments will form a vector set to provide an auxiliary reference for the instrument operation path; By jointly participating in non-linear calculations with the velocity vector, position difference, direction change amplitude, and regional direction modulus in multiple dimensions, a quantitative fitting index is formed, strengthening the expression of the consistency between the instrument movement trajectory and the image region structure direction. This result indicates that the instrument operation direction is highly concentrated and is applicable to controlling the judgment of its movement trend and path update.

[0029] Please refer to Figure 5 , the action trend analysis module includes: The direction segment extraction sub-module identifies the continuous coordinate sequence of the movement path of the instrument tip based on the fuzzy region motion alignment direction set, connects adjacent coordinates to form direction segments, eliminates the segments in the trajectory with lengths lower than the specified threshold, and obtains the direction segment vector set; Obtain the continuous coordinate sequence of the movement of the instrument tip from the fuzzy region motion alignment direction set. The coordinate sequence represents the specific position of the instrument at different time points. By connecting adjacent coordinates, a set of line segments representing the movement direction is formed. This process is similar to capturing the dynamic path of the instrument through an endoscope during laparoscopic surgery. Assume that during the surgery, by real-time tracking the movement of the instrument in the patient's body, each generated line segment is the straight-line movement path of the instrument from one position to another. Any line segment shorter than three pixels is considered a minor movement or a stationary state, so it is eliminated from the dataset to ensure that only the clearly meaningful movement trajectories are analyzed. Such processing improves the accuracy of the data and the reliability of the analysis. Through screening and processing, a direction segment vector set is finally generated, which will be used for further trend analysis and surgical path optimization to improve surgical precision and safety.

[0030] The included angle value statistics sub-module analyzes the included angle value based on the direction segment vector set according to the dot product of the direction vectors, divides it into multiple intervals, and statistically analyzes the frequency and direction fluctuation of the intervals. The formula is used: ; Calculate the change value of the included angle interval, extract the high-change region according to the index size, and obtain the main included angle trend interval; Among them, represents the change value of the included angle interval, represents the appearance frequency of the included angle interval, represents the average length of the direction segments within the included angle interval, represents the frequency change value of the adjacent intervals of the included angle interval, represents the fluctuation range of the direction angle within the included angle interval, represents the total number of included angle intervals; First, extract the direction vectors of all adjacent line segment pairs from the direction segment vector set. The direction vector is the unit vector from the starting point to the ending point of the line segment in the two-dimensional coordinate system. For example, if the starting point of line segment A is (12, 20) and the ending point is (16, 24), then its direction vector is (4, 4), which is normalized to the unit vector (0.707, 0.707). If the starting point of line segment B is (16, 24) and the ending point is (20, 22), the direction vector is (4, -2), which is normalized to the unit vector (0.894, -0.447). The dot product of the two is , and the calculated included angle is , by performing this operation on all line segment pairs, obtaining the entire sequence of included angle values, dividing the included angle values into N categories of intervals according to a range, with the upper and lower limits of each category of interval separated by a fixed angle, set as 15 degrees, and the occurrence frequency of each included angle interval is , for example, the occurrence frequency of the first category of included angle interval 0° - 15° is 4 times, the second category is 7 times, the third category is 5 times, and so on. The average line segment length is the average value of the line segment lengths in this interval, in pixels. For example, in the second category, the line segment lengths are 6, 7, 6, 5, 7, 6, 8 pixels respectively, and the average value is pixels. The direction fluctuation value is the standard deviation of the included angle direction of this category within this interval. If the calculation result is , the adjacent frequency change value is the absolute value of the difference between the frequencies of adjacent intervals. The difference between the second category and the first category is ; Substitute the parameters into the formula: ; After performing similar calculations for the remaining k intervals and then summarizing and summing, the final value is obtained. For example, the sum is 12.63. This value is used as the screening basis for the main included angle trend interval, representing the comprehensive quantitative output of the current included angle frequency concentration and direction stability. The unified standard angle unit is adopted, all angle values are expressed in degrees, the length is in pixels, the frequency is an integer number of times, and all calculation results maintain consistent dimensions. Finally, the value has no unit and is used as a comprehensive trend evaluation indicator. This result indicates that the current direction change trend is mainly concentrated within a certain included angle range, which can be used as the basis for extracting subsequent repeated direction intervals. By weighted integration of multi-dimensional data such as frequency, direction change, and line segment length, the main intervals with concentrated direction changes can be effectively distinguished, which helps to clarify the distribution range of the main direction trend.

[0031] The repeated interval extraction sub-module screens the segment sequence in which the included angle values of the direction line segments continuously fall within the interval according to the main included angle trend interval, and integrates the distribution segments of the sequence to obtain the repeated trend direction interval; Screen continuous direction segments according to the main angle trend interval. The process is similar to analyzing the periodic pattern of heart movement during a heart operation. By identifying and tracking the continuous direction of heart movement, doctors can better understand the movement law of the heart. By screening those segments whose directions continuously fall within the main angle trend interval, the dominant trend in the instrument movement can be effectively captured. The trend reflects the consistency and regularity of instrument operation during the operation, which helps the surgical team predict and plan the optimal movement path of the instrument, thereby reducing the operation time and potential risks. Through this method, useful movement patterns can be extracted from a large amount of surgical dynamic data. The importance of the steps lies in being able to transform complex dynamic information into practical applicable surgical strategies and providing scientific decision-making support for the surgical team.

[0032] Please refer to Figure 6 , the path response correction module includes: The trend interval identification sub-module identifies the continuous direction segments in the instrument movement trajectory according to the repeated trend direction interval, calculates the angle difference between adjacent segments, analyzes the continuous interval of the angle difference change, determines the start and end boundaries of the trend direction, and obtains the trend direction interval; The instrument movement trajectory data consists of continuous positioning points and direction vectors. The data points can come from sensors, cameras or visual recognition, and are used to determine the movement path of the instrument in three-dimensional space. For example, assume that in consecutive frames, the instrument moves from point A(0, 0) to point B(1, 1), then the direction vector is (1, 1), and calculate the angle difference of the direction segments between every two frames. If the angle difference between two direction segments is greater than a certain set threshold, it is considered that there is a significant change in these two directions. From point B to point C(2, 0), calculate the angle difference (from 45 degrees to 90 degrees). If the angle difference is greater than the preset 10 degrees, it is considered that this segment has deviated. Extract the trend direction in the instrument movement trajectory, identify the interval of the angle change between its adjacent direction segments. The interval defines the "trend direction interval" of the instrument movement. Through multiple calculations and adjustments, finally generate the start and end boundaries of the trend direction and accurately correct them so that the subsequent steps can achieve direction adjustment and optimization, and finally obtain the trend direction interval, which can be used for subsequent deviation direction detection and movement path correction.

[0033] The deviation direction detection sub-module extracts the current frame instrument direction segment based on the trend direction interval, compares it with the angle difference within the trend direction interval, and screens the direction segments that deviate from the threshold to obtain the deviation direction segments; Obtain the direction line segment of the instrument from the current frame and compare it with the recognized trend direction interval. For example, in the current frame, the direction vector of the instrument is (0.8, 1.0), and calculate the angle difference with the trend direction interval. Assume that the trend direction interval is from 30° to 45°. If the angle of the current direction line segment is 60°, then calculate its angle difference with the trend interval as 15°. If this difference exceeds the preset deviation threshold (such as 10°), it is considered that the direction in the current frame has deviated significantly, and then it is screened. This operation ensures that further adjustment is only made when the direction deviates significantly, avoiding unnecessary error adjustment, and obtaining the deviated direction line segment. The deviated direction line segment will be used for direction correction in subsequent steps to ensure that the movement path of the instrument can conform to the predetermined trend.

[0034] The direction correction generation sub-module identifies the adjustment angle based on the deviated direction line segment through angle difference, and corrects the movement direction of the instrument to be close to the trend direction, obtaining the optimized movement path of the surgical assistance instrument; First, calculate the angle difference between it and the trend direction interval. Assume that the angle of the deviated direction line segment is 60° and the trend direction interval is from 30° to 45°, so the angle difference is 15°. Based on this difference, the minimum angle adjustment method is used for correction. For example, if the angle difference is greater than 5°, then calculate the minimum adjustment angle to correct the instrument direction, and the adjusted angle should be between the minimum angle and the maximum angle of the trend direction interval. In this way, the direction of the original path will be adjusted to be closer to the trend direction, thus forming a corrected movement trajectory of the instrument. When performing this operation, further refinement adjustments will also be made according to factors such as the movement speed and path density of the instrument. For example, if the speed of the instrument is fast and the distance is far, the adjustment step size can be appropriately enlarged to obtain the optimized movement path of the surgical assistance instrument. This path ensures that the instrument can always maintain a movement trajectory that is more in line with the predetermined trend when performing tasks, thereby effectively improving the path accuracy.

[0035] The above is only the preferred embodiment of the present invention, and does not limit the present invention in other forms. Any person skilled in the art may use the disclosed technical content to make changes or modifications into equivalent embodiments with equivalent changes and apply them to other fields. However, as long as it does not depart from the technical solution content of the present invention, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A microsurgery auxiliary system based on visual recognition, characterized in that: The system comprises: The instrument target positioning module extracts the closed boundary area based on the continuous frames of the surgical field image under the microscope, judges the boundary closure and the image center offset value, locates the center position of the instrument end in the image, and generates the spatial coordinate set of the instrument end; The behavior feature analysis module analyzes the angle change rate and angle amplitude of the line connecting the end points in the continuous frames according to the spatial coordinate set of the end of the instrument, determines the behavior type corresponding to the line segment, and generates an instrument action behavior state group; The fuzzy dynamic tracking module calls the instrument action behavior state group, extracts the microscopic tissue boundary area image, identifies the distance between the surgical instrument and the tissue boundary, records the change amplitude of adjacent frames, performs grayscale gradient and instrument edge direction difference analysis on the jump area, selects the high fuzzy degree area, tracks the moving direction of the instrument end in the area, and generates a fuzzy area motion alignment direction set; The motion trend analysis module extracts motion direction segments based on the fuzzy area motion alignment direction set, identifies the angle values ​​of adjacent direction segments and counts the occurrence frequency, screens the continuous direction sequence, and obtains the repeated trend direction interval; The path response correction module compares the current instrument direction line segment angle with the trend interval angle according to the repeated trend direction interval, selects the line segment position that deviates from the direction and adjusts the motion direction in the corresponding frame to generate an optimized surgical auxiliary instrument motion path; The optimized surgical auxiliary instrument motion path includes a corrected motion trajectory, direction optimization parameters, and a deviation correction position.

2. The microsurgery auxiliary system based on visual recognition according to claim 1, characterized in that: The spatial coordinate set of the instrument end includes spatial position information, image reference offset, and center indication parameters; the instrument action behavior state group includes behavior identification labels, angle change indicators, and motion amplitude parameters; the fuzzy area motion alignment direction set includes direction change characteristics, fuzzy edge attributes, and regional dynamic indicators; the repeated trend direction interval includes continuous direction patterns, angle change distribution, and trend stability parameters.

3. The microsurgery auxiliary system based on visual recognition according to claim 1, characterized in that: The device target positioning module includes: The image sequence extraction submodule identifies the image data of each frame based on the continuous frames of the surgical field image under the microscope, detects the brightness gradient area in each frame, integrates the grayscale difference and texture distribution change, and obtains the image dynamic sequence frame set; The boundary judgment and analysis submodule calls the image dynamic sequence frame set, extracts the closed boundary according to the pixel edge continuity, identifies the boundary contour point set, and analyzes the closure and connection integrity, measures the distance between the contour center of gravity and the image center and classifies it, analyzes the boundary closure and center offset, and uses the formula: ； Calculate the offset correlation value, sort the offset correlation values ​​in each frame image, and filter the front area to obtain the effective boundary positioning interval value; in, represents the offset correlation value, Representative The pixel coordinates of the center of gravity of the contour in the frame, Represents the pixel coordinates of the center point of the image, For the Frame boundary closure value, Representative candidate area The offset pixel value, is the full frame offset reference pixel value, is the number of valid frames, is the number of candidate regions; The spatial coordinate identification submodule selects the pixel density area inside the boundary according to the effective boundary positioning interval value, extracts the geometric center coordinates, identifies the three-dimensional space position by combining the imaging parameters and the scale factor, and generates the spatial coordinate set of the instrument end.

4. The microsurgery auxiliary system based on visual recognition according to claim 3, characterized in that: The behavior feature analysis module includes: The trajectory rate calculation submodule selects the end coordinates of two adjacent frames according to the instrument end space coordinate set, identifies the spatial distance, extracts the rate per unit time according to the time interval, and obtains the line segment angle change rate value; The angle amplitude extraction submodule calls the line segment angle change rate value, identifies the angle according to the vectors of adjacent line segments between three frames, and screens out segments exceeding the rotation amplitude threshold to obtain the line segment angle amplitude interval value; The behavior type determination submodule calculates the line segment action behavior characteristic measurement value according to the line segment angle change rate value, and makes a judgment with the behavior classification boundary value, using the formula: ； Generate a device action behavior state group; in, represents the characteristic measurement value of the line segment action behavior, represents the rate of change of angle, Represents the reference value of angle change, and Respectively represent the upper and lower limits of the angle range, For behavior The stabilizing factor, The number of behaviors.

5. The microsurgery auxiliary system based on visual recognition according to claim 4, characterized in that: The fuzzy dynamic tracking module includes: The boundary image extraction submodule calls the instrument action behavior state group, identifies the current position and posture of the instrument in the microscopic tissue image, extracts the microscopic tissue boundary area image of the instrument in the image, and obtains the instrument boundary registration image set; The jump region screening submodule analyzes the grayscale change amplitude between the device edge and the tissue boundary in adjacent frames based on the device boundary registration image set, screens image regions with drastic grayscale changes and large edge direction differences, and generates a high gradient difference region atlas; The end motion direction extraction submodule tracks the frame-by-frame position of the instrument end in the region according to the high gradient difference region map, extracts the corresponding coordinate changes of the end in consecutive frames and the regional direction vector, identifies the matching degree and angle increment of the end movement direction, and uses the formula: ； Calculate the direction fitting value of the end corresponding area, and generate the fuzzy area motion alignment direction set according to the change trend in the fitting value sequence; in, Represents the fitting value of the end corresponding area direction, Representative The velocity vector at the end of the frame device, Representative Frame and The end coordinate difference vector between frames, Representative Frame and The change value of the direction angle at the end of the frame, Representative The modulus sequence of the frame region direction vector group, The number of frames for continuous tracking.

6. The microsurgery auxiliary system based on visual recognition according to claim 5, characterized in that: The action trend analysis module includes: The direction line segment extraction submodule identifies the continuous coordinate sequence of the moving path of the instrument end based on the fuzzy area motion alignment direction set, connects adjacent coordinates to form direction line segments, removes line segments whose length is less than a specified threshold in the trajectory, and obtains a direction line segment vector set; The angle value statistics submodule analyzes the angle value according to the direction line segment vector set and the direction vector dot product, and divides it into multiple intervals, and calculates the frequency and direction fluctuation of the interval using the formula: ； Calculate the change value of the angle interval, extract the high change area according to the indicator size, and obtain the main angle trend interval; in, Represents the change value of the angle interval, Indicates The frequency of occurrence of the angle interval, Indicates The mean length of the directional line segments within the angle interval, Indicates The frequency change value of the adjacent intervals of the angle interval, Indicates The fluctuation range of the direction angle within the angle interval, Represents the total number of angle intervals; The repetitive interval extraction submodule selects a segment sequence whose angle values ​​continuously fall within the interval in the direction line segment according to the main angle trend interval, integrates the distribution segments of the sequence, and obtains a repetitive trend direction interval.

7. The microsurgery auxiliary system based on visual recognition according to claim 1, characterized in that: The path response correction module includes: The trend interval identification submodule identifies the continuous direction line segments in the instrument motion trajectory according to the repeated trend direction interval, calculates the angle difference between adjacent line segments, analyzes the continuous interval of the angle difference change, determines the start and end boundaries of the trend direction, and obtains the trend direction interval; The deviation direction detection submodule extracts the current frame instrument direction line segment based on the trend direction interval, compares it with the angle difference in the trend direction interval, screens the direction line segment that deviates from the threshold, and obtains the deviation direction line segment; The direction correction generation submodule adjusts the angle according to the deviation direction line segment by identifying the angle difference, corrects the movement direction of the instrument to the direction close to the trend, and obtains the optimized movement path of the surgical auxiliary instrument.

Images