Microscopic Surgery Assistance System Based on Visual Recognition

Abstract

The present invention relates to the field of computer vision technology, specifically a microsurgical assistance system based on visual recognition. The system includes an instrument target positioning module, a behavior feature analysis module, a fuzzy dynamic tracking module, a motion trend analysis module, and a path response correction module. In the present invention, by extracting the closed boundary region and combining the image center offset for determination, the center of the instrument end can be accurately positioned, improving the spatial perception accuracy of the key points in the surgical field. By constructing behavior feature indicators in combination with the angle change rate and the included angle amplitude, the recognition accuracy of complex actions is enhanced. By analyzing the differences in image gray-scale gradient and edge direction, the fuzzy region is extracted and the motion trend is tracked, improving the instrument dynamic capture ability in a low-clarity environment. The repetition trend sequence is extracted by counting the direction change frequency, correcting the path deviation, enhancing the action coherence and controllability, and overall improving the intelligent response of instrument control and the reliability of path planning in the microsurgical operation scenario.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and particularly to a microsurgical assistance system based on visual recognition. Background Art

[0002] The field of computer vision technology includes research and applications in aspects such as image acquisition, image processing, image recognition and understanding, mainly perceiving and analyzing image or video information in the real world by simulating human vision. The core content of this technology field includes the use of image acquisition devices, the extraction of image features, object detection and segmentation, 3D reconstruction, scene understanding, etc. Computer vision is widely applied in scenarios such as autonomous driving, medical image analysis, face recognition, industrial inspection, and intelligent monitoring, relying on the collaborative work of image sensors, neural network models, and image processing algorithms to extract structured information from raw visual data and achieve automatic perception and judgment of the environment.

[0003] Among them, a microsurgical assistance system refers to a system that assists in microsurgical operations based on computer vision technology, mainly used to improve the operation accuracy and visualization degree during the operation. This system covers key technical matters such as high-resolution acquisition of intraoperative images, image registration processing, automatic recognition and tracking of the microsurgical operation area, and three-dimensional structure reconstruction of the surgical area. It accurately extracts the boundary information of the surgical target area by using image segmentation methods, obtains the relative position changes between the surgical instruments and tissues through image sequence analysis, and uses a deep learning model to automatically classify and locate key intraoperative tissue structures, thereby supporting microsurgical doctors to perform operation guidance and auxiliary decision-making in a limited vision environment.

[0004] When the existing technology is applied to the microsurgical assistance scenario, although it has the ability of image acquisition and recognition, when dealing with the evolution trend of instrument behavior in dynamic continuous image frames, it lacks targeted time series analysis means and cannot effectively identify the continuous and trend features in instrument actions, resulting in problems such as fuzzy judgment or response delay in behavior judgment. In scenarios where there are blurred interferences and unclear tissue boundaries in the surgical field image, the existing methods have insufficient ability to identify and dynamically track local abnormal areas in the image, causing instrument position recognition drift and path tracking interruption, affecting the stability and safety of the intraoperative assistance system. The existing system relies on static image features for operation guidance and lacks a long-term behavior modeling mechanism based on the change trend of the instrument movement direction, making the judgment accuracy of repetitive operation paths not high and easily ignoring the cumulative error of subtle path deviations. For example, during a surgical operation with a complex microsurgical field and variable tissue morphology, the existing methods are difficult to achieve stable tracking of continuous operations of the instrument in the blurred area, limiting the effective response of the system to the actual operation intention and affecting the auxiliary ability for doctors' fine operations. Summary of the Invention

[0005] The object of the present invention is to solve the disadvantages existing in the prior art, and a microsurgical assistance system based on visual recognition is proposed.

[0006] To achieve the above object, the present invention adopts the following technical solutions: The microsurgical assistance system based on visual recognition includes:

[0007] The instrument target positioning module extracts the closed boundary region based on the continuous frame images of the surgical field under the microscope, judges the boundary closure degree and the image center offset value, locates the center position of the instrument end in the image, and generates the instrument end spatial coordinate set;

[0008] The behavior feature analysis module analyzes the angular change rate and the included angle amplitude of the connection lines of the end points in the continuous frames according to the instrument end spatial coordinate set, judges the corresponding behavior type of the line segment fragment, and generates the instrument action behavior state group;

[0009] The fuzzy dynamic tracking module calls the instrument action behavior state group, extracts the image of the microsurgical tissue boundary region, identifies the distance between the surgical instrument and the tissue boundary, records the change amplitude of adjacent frames, analyzes the gray gradient and the difference in the instrument edge direction of the jump region, screens the regions with high blur degree, and tracks the movement direction of the instrument end in the region to generate the fuzzy region motion alignment direction set;

[0010] The action trend analysis module extracts the motion direction line segments based on the fuzzy region motion alignment direction set, identifies the included angle values of adjacent direction line segments and counts the occurrence frequencies, screens the continuous direction sequences, and obtains the repeated trend direction intervals.

[0011] As a further solution of the present invention, the instrument end spatial coordinate set includes spatial position information, image reference offset, and center indication parameter, the instrument action behavior state group includes behavior recognition label, angular change index, and motion amplitude parameter, the fuzzy region motion alignment direction set includes direction change feature, fuzzy edge attribute, and region dynamic index, and the repeated trend direction interval includes continuous direction pattern, included angle change distribution, and trend stability parameter.

[0012] As a further solution of the present invention, the instrument target positioning module includes:

[0013] The image sequence extraction sub-module identifies each frame of image data based on the continuous frame images of the surgical field under the microscope, detects the brightness gradient region in each frame of image, integrates the gray difference and the texture distribution change, and obtains the image dynamic sequence frame set;

[0014] The boundary judgment analysis sub-module calls the dynamic image sequence frame set, extracts a closed boundary according to the continuity of pixel edges, identifies the boundary contour point set, analyzes the closure degree and connection integrity, measures the distance between the contour centroid and the image center and classifies it, analyzes the boundary closure degree and the center offset amount, and uses the formula:

[0015] ;

[0016] Calculate the offset correlation degree value, sort the offset correlation degree values in each frame image, and screen the front regions to obtain the effective boundary positioning interval value;

[0017] Among them, represents the offset correlation degree value, represents the pixel coordinates of the contour centroid in the th frame, represents the pixel coordinates of the image center point, is the boundary closure degree value of the th frame, represents the offset pixel value of the candidate region , is the full-frame offset reference pixel value, is the number of valid frames, is the number of candidate regions;

[0018] The spatial coordinate recognition sub-module selects the pixel density region inside the boundary according to the effective boundary positioning interval value, extracts the geometric center coordinates, combines the imaging parameters and the scale factor to recognize the three-dimensional spatial position, and generates the spatial coordinate set of the instrument end.

[0019] As a further solution of the present invention, the behavior feature analysis module includes:

[0020] The trajectory speed calculation sub-module selects the end coordinates of two adjacent frames according to the spatial coordinate set of the instrument end, identifies the spatial distance, and extracts the speed per unit time according to the time interval to obtain the line segment angle change speed value;

[0021] The included angle amplitude extraction sub-module calls the line segment angle change speed value, identifies the included angle based on the vectors of adjacent line segments among three frames and screens out the segments exceeding the rotation amplitude threshold to obtain the line segment included angle amplitude interval value;

[0022] The behavior type determination sub-module calculates the line segment action behavior feature metric value according to the line segment angle change speed value, and judges it with the behavior classification boundary value, and uses the formula:

[0023] ;

[0024] Generate the instrument action behavior state group;

[0025] Among them, Represents the measurement value of the line segment action behavior characteristics, Represents the angular change rate, Represents the angular change reference value, And Respectively represent the upper and lower limits of the included angle amplitude interval, Is the stability factor of the behavior And Is the number of behaviors.

[0026] As a further solution of the present invention, the fuzzy dynamic tracking module includes:

[0027] The boundary image extraction sub-module calls the instrument action behavior state group, identifies the current position and posture of the instrument in the microscopic tissue image, extracts the microscopic tissue boundary region image of the instrument in the image, and obtains the instrument boundary registration image set;

[0028] The jump region screening sub-module analyzes the gray scale change amplitude between the instrument edge and the tissue boundary in adjacent frames based on the instrument boundary registration image set, screens the image regions with large gray scale change and large edge direction difference, and generates a high gradient difference region map;

[0029] The end movement direction extraction sub-module tracks the frame-by-frame position of the instrument end in the region according to the high gradient difference region map, extracts the corresponding coordinate change and region direction vector of the end in consecutive frames, identifies the matching degree and angle increment of the end movement direction, and uses the formula:

[0030] ;

[0031] Calculates the fitting value of the corresponding region direction of the end, and generates a fuzzy region motion alignment direction set according to the change trend in the fitting value sequence;

[0032] Wherein, Represents the fitting value of the corresponding region direction of the end, Represents the Velocity vector of the instrument end in the Represents the Frame and The end coordinate difference vector between frames, Represents the Frame and The end direction angle change value between frames, Represents the Modulus value sequence of the region direction vector group in the Frame, and

[0033] As a further solution of the present invention, the action trend analysis module includes:

[0034] The direction line segment extraction sub-module identifies the continuous coordinate sequence of the movement path of the instrument end based on the fuzzy region movement alignment direction set, connects adjacent coordinates to form direction line segments, eliminates line segments with lengths below a specified threshold in the trajectory, and obtains a set of direction line segment vectors;

[0035] The included angle value statistics sub-module analyzes the included angle value based on the dot product of direction vectors according to the set of direction line segment vectors, divides it into multiple intervals, statistically calculates the frequency and direction fluctuation of the intervals, and uses the formula:

[0036] ;

[0037] Calculate the change value of the included angle interval, extract the high-change region according to the index size, and obtain the main included angle trend interval;

[0038] Among them, represents the change value of the included angle interval, represents the appearance frequency of the included angle interval, represents the average length of the direction line segments within the included angle interval, represents the frequency change value of the adjacent intervals of the included angle interval, represents the fluctuation range of the direction angle within the included angle interval, represents the total number of included angle intervals;

[0039] The repeated interval extraction sub-module filters the segment sequence in which the included angle values continuously fall within the interval in the direction line segments according to the main included angle trend interval, and integrates the distribution segments of the sequence to obtain the repeated trend direction interval.

[0040] As a further solution of the present invention, the system further includes a path response correction module:

[0041] The path response correction module compares the angle of the current instrument direction line segment with the angle of the trend interval according to the repeated trend direction interval, filters the position of the line segment deviating from the direction and adjusts the movement direction in the corresponding frame to generate an optimized movement path of the surgical assistance instrument;

[0042] The optimized movement path of the surgical assistance instrument includes a corrected movement trajectory, direction optimization parameters, and deviation correction positions.

[0043] As a further solution of the present invention, the path response correction module includes:

[0044] The trend interval identification sub-module identifies the continuous direction line segments in the movement trajectory of the instrument according to the repeated trend direction interval, calculates the angle difference between adjacent line segments, analyzes the continuous interval of the change of the angle difference, determines the start and end boundaries of the trend direction, and obtains the trend direction interval;

[0045] Based on the trend direction interval, the deviation direction detection sub-module extracts the instrument direction line segment of the current frame, compares it with the angular difference within the trend direction interval, screens the direction line segments that deviate from the threshold, and obtains the deviation direction line segments.

[0046] According to the deviation direction line segments, the direction correction generation sub-module identifies the adjustment angle through the angular difference, corrects the instrument movement direction to be close to the trend direction, and obtains the optimized surgical assistance instrument movement path.

[0047] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0048] In the present invention, by extracting the region with the closed boundary feature from the microscopic image sequence and combining the image center offset for determination, the center position of the instrument end can be accurately identified, the spatial accurate positioning of the key operation points can be realized, the perception accuracy of the operation target in the surgical field can be enhanced, the behavioral feature index is constructed based on the angular change rate and the included angle amplitude of the end connection lines in the continuous image frames, the recognition accuracy of the complex action features is strengthened, the subtle changes between the instrument operation states can be effectively distinguished, and the accuracy and response speed of the intraoperative behavior judgment can be improved. By combining the image gray gradient and the edge direction difference to extract the highly blurred region and tracking the movement trend of the instrument in the blurred tissue region, the stable capture ability of the instrument dynamics in the low-definition environment can be improved, and the decoding ability of the instrument operation intention in the visually interfering region can be enhanced. By statistically analyzing the direction change frequency and extracting the repeated trend direction sequence, a regular movement trend pattern is constructed, which provides a directional reference for the subsequent path judgment and reduces the interference of the invalid movement path. By comparing the deviation between the current direction line segment and the trend interval angle and correcting the movement direction, the stability of the instrument action and the path consistency are ensured, the coherence and controllability of the overall operation path are improved, and through the technical linkage of multi-dimensional parameter analysis, enhanced recognition of the visual fuzzy region, extraction of the dynamic path law, and adjustment of the deviation direction, the intelligent response ability of the instrument control, the accuracy of the action discrimination, and the reliability of the path planning in the microscopic operation scenario are effectively improved. Brief Description of the Drawings

[0049] Figure 1 It is the system flow chart of the present invention;

[0050] Figure 2 It is the flow chart of the instrument target positioning module in the present invention;

[0051] Figure 3 It is the flow chart of the behavioral feature analysis module in the present invention;

[0052] Figure 4 It is the flow chart of the fuzzy dynamic tracking module in the present invention;

[0053] Figure 5Flow chart of the motion trend analysis module in the present invention;

[0054] Figure 6 Flow chart of the path response correction module in the present invention. Detailed implementation manners

[0055] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0056] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the accompanying drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention. In addition, in the description of the present invention, the meaning of "a plurality of" is two or more, unless otherwise specifically defined.

[0057] Please refer to Figure 1 , the microsurgical assistance system based on visual recognition includes:

[0058] The instrument target positioning module extracts the closed boundary region based on the consecutive frame images of the surgical field under the microscope, judges the boundary closure degree and the image center offset value, locates the center position of the instrument end in the image, and generates the instrument end spatial coordinate set;

[0059] The behavior feature analysis module analyzes the angular change rate and the included angle amplitude of the connecting lines of the end points in the consecutive frames according to the instrument end spatial coordinate set, judges the corresponding behavior types of the line segments, and generates the instrument motion behavior state group;

[0060] The fuzzy dynamic tracking module calls the instrument motion behavior state group, extracts the image of the microsurgical tissue boundary region, identifies the distance between the surgical instrument and the tissue boundary, records the change amplitude of adjacent frames, analyzes the gray gradient and the difference in the instrument edge direction of the jump region, screens the regions with high fuzziness, and tracks the moving direction of the instrument end in the region to generate the fuzzy region motion alignment direction set;

[0061] The motion trend analysis module extracts the motion direction line segments based on the fuzzy region motion alignment direction set, identifies the included angle values of adjacent direction line segments and counts the occurrence frequencies, screens the continuous direction sequences, and obtains the repeated trend direction intervals;

[0062] The path response correction module compares the angle of the current instrument direction line segment with the angle of the trend interval according to the repeated trend direction interval, filters out the line segment positions deviating from the direction, and adjusts the movement direction in the corresponding frame to generate an optimized surgical assistant instrument movement path.

[0063] The set of spatial coordinates of the instrument end includes spatial position information, image reference offset, and center indication parameters. The set of instrument action behavior states includes behavior recognition tags, angle change indicators, and movement amplitude parameters. The set of movement alignment directions in the fuzzy area includes direction change characteristics, fuzzy edge attributes, and regional dynamic indicators. The repeated trend direction interval includes continuous direction patterns, included angle change distributions, and trend stability parameters. The optimized surgical assistant instrument movement path includes corrected movement trajectories, direction optimization parameters, and deviation correction positions.

[0064] Please refer to Figure 2 , the instrument target positioning module includes:

[0065] The image sequence extraction sub-module identifies the image data of each frame based on the continuous frame images of the surgical field under the microscope, detects the brightness gradient regions in each frame image, integrates the changes in gray-scale differences and texture distributions, and obtains a set of image dynamic sequence frames;

[0066] Detect the brightness gradient regions in each frame image, analyze according to the changes in the brightness gradient and the continuity of the texture distribution of the image. The key to the process lies in identifying representative image features. For example, during neurosurgery, identifying the edges of blood vessels and nerve tissues can help doctors accurately judge the position of surgical instruments. By calculating the changes in gray-scale differences and texture distributions in consecutive frames, the dynamic changes during the operation can be effectively tracked. This analysis is achieved by calculating the differences in gray-scale histograms between frames. Integrate the data to construct a detailed relationship between the time frame index and the image frame data. The data set is used for further analysis and operations. For example, in practical applications, the data can be used to evaluate the stability and accuracy during the operation. Through high-precision image processing techniques, a set of image dynamic sequence frames is obtained, providing strong visual support for the operation.

[0067] The boundary judgment and analysis sub-module calls the set of image dynamic sequence frames, extracts closed boundaries according to the pixel edge continuity, identifies the set of boundary contour points, analyzes the closure degree and connection integrity, measures the distance between the contour centroid and the image center and classifies it, analyzes the boundary closure degree and the center offset, and uses the formula:

[0068] ;

[0069] Calculate the offset correlation degree value, sort the offset correlation degree values in each frame image, and filter the front regions to obtain the effective boundary positioning interval value;

[0070] Among them, represents the offset correlation value, represents the pixel coordinates of the centroid of the contour in the th frame, represents the pixel coordinates of the center point of the image, is the closure value of the th frame boundary, represents the offset pixel value of the candidate region is the offset reference pixel value for the entire frame, is the number of valid frames, is the number of candidate regions;

[0071] First, perform an edge continuity extraction operation on each frame of the image to obtain the initial contour of the boundary. During the edge extraction process, the Canny operator is selected for pixel gradient scanning. After calculating the gradient magnitude of the image grayscale through convolution, the pixel points higher than the edge response threshold are recorded as the preliminary boundary point set. For example, select the 12th frame of the intraoperative microscopic image. By setting the gradient change amplitude threshold to 15, a total of 416 edge pixel points are screened to form the preliminary boundary. Subsequently, the boundary point set is connected in position order, and the connection integrity is judged by whether the distance difference between consecutive point pairs is greater than 2 pixels. For each boundary line, the curvature is calculated using the cubic B-spline fitting method, and the tangent vector of the fitted curve is derived. After calculating the absolute value of its second derivative and integrating, it is used as the closed curvature value, which is uniformly normalized to the 0-1 interval range in each frame of the image. For example, the curvature integral value of a certain frame is 238.4, and the closure degree after normalization is 0.87;

[0072] Taking the center coordinate point G of the image plane as the reference point, detect the geometric center coordinates of each frame of the contour area. This point is obtained by the arithmetic mean of all pixel coordinates within the contour. For example, if there are 1372 pixel points within the contour, the mean value of the x coordinates of all points is 103.7, and the mean value of the y coordinates is 119.6, then the centroid of this frame of the contour is (103.7, 119.6), and the image size is 256×256. The geometric center G is (128, 128). Calculate the Euclidean distance between the two points to obtain the offset value of this frame, and perform this process to obtain the offset distance of each frame;

[0073] Furthermore, extract the offset pixel value of the candidate region, take the distance between the centroid of each region and the center of the image as the offset distance, and calculate it using the Euclidean distance formula as well. Then, calculate the average offset within all frames as the reference value . Assume that a total of 8 frames of data are analyzed, and their offset distances are 22.1, 25.7, 19.3, 26.9, 24.2, 28.5, 21.8, 27.4. Then the offset reference value ;

[0074] Call the above parameters and substitute them into the formula for calculation. Set the current number of processed frames \(n = 5\) and the number of candidate regions \(m = 3\). The corresponding data is as follows:

[0075] , ;

[0076] , ;

[0077] , ;

[0078] , ;

[0079] , ;

[0080] , , , ;

[0081] Substitute them into the formula: ;

[0082] Calculate the numerator part of each item respectively:

[0083] , we get ;

[0084] , we get ;

[0085] , we get ;

[0086] , we get ;

[0087] , we get ;

[0088] Calculate the denominator part:

[0089] The sum of offset differences ;

[0090] The denominator ;

[0091] Substitute all the results, the numerator ;

[0092] Final calculation: ;

[0093] The result shows that the comprehensive offset correlation value is 14.47. The higher the value, the more significant the offset of the boundary from the image center. After sorting and screening, it can be used for further determination of the boundary region. By fusing and calculating the boundary closure degree and the contour offset position in a weighted manner, and introducing the offset benchmark of the candidate region to normalize the overall offset impact, it can more reasonably reflect the comprehensive performance of the current frame boundary in two dimensions: spatial offset and structural closure.

[0094] The spatial coordinate recognition sub-module selects the pixel density region inside the boundary according to the effective boundary positioning interval value, extracts the geometric center coordinates, and combines the imaging parameters and the scale factor to recognize the three-dimensional spatial position, generating the spatial coordinate set of the instrument end.

[0095] Obtain the set of pixel points inside the boundary in the corresponding frame image. The process involves the selection of the image region and the precise calculation of pixel points. By selecting the section with the largest density in the boundary point group, its geometric center coordinates can be extracted. The extraction process is realized by calculating the spatial distribution density of each pixel point. Call the index of the corresponding frame of the image and the imaging parameters, and combine the optical axis calibration parameters and the image scale factor, which are all obtained through pre-experimental settings and calibrations. Calculate its position relationship relative to the image plane, which includes a series of geometric transformations and ratio calculations, and obtain its position representation in the three-dimensional space. The position representation is extremely important for accurately navigating the surgical instrument to the target operation area. Establish the spatial coordinate set of the instrument end. This coordinate set provides the accurate position information of the surgical instrument during the operation, thus ensuring the accuracy and safety of the surgery.

[0096] Please refer to Figure 3 , the behavior feature analysis module includes:

[0097] The trajectory rate calculation sub-module selects the end coordinates of two adjacent frames according to the spatial coordinate set of the instrument end, recognizes the spatial distance, and extracts the rate per unit time according to the time interval to obtain the line segment angle change rate value.

[0098] First, obtain the end coordinate points in two adjacent frames. Use the method of spatial geometry calculation to obtain the straight-line distance between the two points, and further calculate the time interval between the two frames. Suppose in a certain instance, the frame interval is 0.02 seconds, and the coordinate points of two adjacent frames are (1, 2, 3) and (4, 6, 8) respectively. Then, the distance between the two points calculated by the spatial distance formula is 5 units. Dividing by the time interval of 0.02 seconds, the obtained rate is 250 units per second. This rate reflects the linear movement rate of the instrument end, similar to the movement speed of a robotic arm. The calculation of the rate value needs to ensure the accuracy of the calculated time interval to reflect the true speed of the instrument movement and ensure that the rate of the line segment fragment will not mislead the behavior analysis due to inaccurate measurement. According to the value of the rate, the coherence and stability of the instrument movement can be further analyzed, and finally the rate value of the line segment angle change is generated for subsequent included angle analysis.

[0099] The included angle amplitude extraction sub-module calls the rate value of the line segment angle change, and based on the vectors of adjacent line segments among three frames, identifies the included angle and filters out the segments exceeding the rotation amplitude threshold to obtain the included angle amplitude interval value of the line segment;

[0100] By calculating the included angle between the vectors of the instrument end, further based on the end vectors of two adjacent line segments within three frames, first calculate the direction vectors of the ends of two adjacent frames. Suppose in a certain instance, the vectors of adjacent line segments are (3, 2, 1) and (5, 6, 2) respectively. Then calculate the inner product of the two vectors, and then calculate the included angle value through the inner product formula. Suppose the result is 45 degrees. In order to filter out outliers, a threshold for the included angle amplitude needs to be set. If the calculated included angle exceeds this threshold, this segment will be excluded. In this way, only the segments that meet the conditions are ensured to be retained. For example, suppose the set included angle threshold is 60 degrees. When the calculated included angle is 45 degrees, this segment meets the conditions, is retained and continues to be analyzed, and finally the included angle amplitude interval value of the line segment is generated. This step helps to reduce the influence of noise and ensures that the subsequent determination of the behavior type is more accurate and reliable.

[0101] The behavior type determination sub-module calculates the measurement value of the line segment action behavior characteristics according to the rate value of the line segment angle change, and makes a judgment with the behavior classification boundary value. The formula is used: ;

[0102] Generate the instrument action behavior state group;

[0103] Among them, represents the measurement value of the line segment action behavior characteristics, represents the rate of angle change, represents the angle change reference value, and represent the upper and lower limits of the included angle amplitude interval respectively, is the stability factor of the behavior ​ is the number of behaviors;

[0104] By obtaining the angular change rate and included angle amplitude of each line segment fragment, calculate the behavior feature metric value of this fragment. Assume that in a certain embodiment, the angular change rate of the instrument's line segment is 250 units per second and the included angle amplitude is 45 degrees. Use the preset behavior classification formula for calculation. The rate value is 250 units per second, and the included angle amplitude is 45 degrees. and respectively represent the upper and lower limits of the included angle amplitude interval of this line segment. Assume = 60 degrees and = 30 degrees;

[0105] Substitute the specific values into the formula: ;

[0106] First, calculate the absolute difference and the square root part: ; ;

[0107] Then, calculate the numerator part of the feature metric value F: ;

[0108] Next, assume that the stability factor of this behavior is 0.5 and the number of behavior types is 3. Then the denominator part in the behavior classification formula is calculated as follows: ;

[0109] Finally, the calculation result of the behavior feature metric value F is: ;

[0110] According to this value, if the boundary of behavior classification is set to 1.0, then since F = 108.83 is greater than 1.0, this fragment is determined to be a high-frequency motion type. In this way, by combining the behavior feature metric values of each fragment, the actions of the instrument can be classified, and finally an instrument action behavior state group is generated. The unification of units is very important to ensure the dimensional consistency of the rate, angle, and behavior stability factor. The rate is expressed in units / second, the angle is in degrees, and the stability factor is a dimensionless value to ensure the rationality and correctness of the formula calculation. In actual operation, the movement rate of the instrument end should be adjusted according to the real-time collected data, and the selection of the included angle amplitude interval should be set according to the movement characteristics and requirements of the instrument to ensure accurate classification of behavior types.

[0111] Please refer to Figure 4 , the fuzzy dynamic tracking module includes:

[0112] The boundary image extraction sub-module calls the instrument action behavior state group to identify the current position and posture of the instrument in the microscopic tissue image, and extracts the microscopic tissue boundary region image of the instrument in the image to obtain the instrument boundary registration image set;

[0113] The position and posture recognition of the instrument in the microscopic tissue image is carried out. Suppose during an ophthalmic surgery, through real-time image recognition technology, the position of the surgical instrument is accurately captured, which helps the doctor perform the surgery more precisely, avoid accidentally touching the sensitive tissues of the patient, and the doctor can adjust the surgical strategy by observing the immediate feedback of the instrument in the field of view. The image extraction includes the interaction between the instrument and the tissue boundary. For example, if the instrument is close to the edge tissue of the eyeball, the edge clarity of that part of the image is automatically enhanced so that the doctor can see the operation area more clearly. By analyzing the relationship between the edge of the instrument and the surrounding tissues, the instrument boundary registration image set is generated, and the image set can be used for subsequent precise operation guidance.

[0114] The jump region screening sub-module analyzes the gray-scale change amplitude between the instrument edge and the tissue boundary in adjacent frames based on the instrument boundary registration image set, screens the image regions with large gray-scale changes and large edge direction differences, and generates a high-gradient difference region map;

[0115] First, it involves calculating the gray-scale change amplitude between the instrument edge and the tissue boundary, which can be used to judge whether the instrument is too close to the key tissue, causing potential surgical risks. For example, during a neurosurgical operation, the gray-scale change amplitude between the edge and the tissue boundary can indicate whether the instrument touches a sensitive area. By setting a threshold for the gray-scale change amplitude, such as a threshold of 15, only when the gray-scale change amplitude exceeds this value, the high-risk area is marked. The marking helps the doctor stay alert to the dangerous area during the real-time surgery, generate a high-gradient difference region map, provide a visual aid tool, and adjust the surgical path according to the high-risk areas marked in the map to avoid unnecessary harm to the patient.

[0116] The end movement direction extraction sub-module tracks the position of the instrument end in the region frame by frame according to the high-gradient difference region map, extracts the corresponding coordinate changes and regional direction vectors of the end in consecutive frames, and identifies the matching degree and angle increment of the end movement direction, using the formula:

[0117] ;

[0118] Calculate the fitting value of the regional direction corresponding to the end, and generate a fuzzy region movement alignment direction set according to the change trend in the fitting value sequence;

[0119] Among them, represents the fitting value of the regional direction corresponding to the end, represents the frame velocity vector of the instrument end, Represents the coordinate difference vector at the end between the th frame and the th frame, represents the change value of the end direction angle between the th frame and the th frame, is the number of consecutive tracking frames;

[0120] Perform position tracking operations on the end of the instrument frame by frame in the regional map. The tracking data is sourced from the microscopic surgery image acquisition sequence, with an interval of 0.05 seconds between each frame to ensure real-time and continuity. Extract the relative displacement of the instrument end in the two-dimensional image coordinate system through the change of the pixel coordinates of the instrument end in consecutive frames, forming , for example, if the coordinates of the instrument end in the t-th frame are (120, 84) and in the (t - 1)-th frame are (117, 81), then , with a modulus length of pixels. Considering an image resolution of 1 pixel corresponding to 2 microns, the displacement length is converted to ;

[0121] The velocity vector is calculated by dividing the end displacement by the time interval between frames, pixels per second, with a modulus length of pixels per second;

[0122] The direction change amount is calculated by the included angle between the unit vectors of the instrument directions in two consecutive frames. Assuming that the unit vector of the instrument direction in the current frame is (0.707, 0.707) and in the previous frame is (0.894, 0.447), and the cosine of the included angle is then , and the modulus of the regional direction vector is the modulus length of the main direction vector of the gray level gradient in the corresponding region within the atlas. Assuming its value is 46.29 pixels per second;

[0123] Substitute the above items into the formula:

[0124] ;

[0125] The calculation results show that in the current consecutive 3 frames, the fitting degree of the instrument end relative to the regional direction is 273.3. The larger the value, the more concentrated the direction change is within the same regional direction range. This result is a representative direction quantity index for the concentrated movement alignment in the fuzzy region, and subsequently, such direction quantities of all frame segments will form a vector set to provide an auxiliary reference for the instrument operation path;

[0126] By jointly participating in non - linear calculations with the velocity vector, position difference, direction change amplitude, and regional direction modulus in multiple dimensions, a quantitative fitting index is formed, which strengthens the expression of the consistency between the instrument movement trajectory and the image region structure direction. The results show that the instrument operation directions are highly concentrated, which is applicable to controlling the judgment of its movement trend and path update.

[0127] Please refer to Figure 5 , the action trend analysis module includes:

[0128] The direction line segment extraction sub - module, based on the fuzzy region motion alignment direction set, identifies the continuous coordinate sequence of the instrument end movement path, connects adjacent coordinates to form direction line segments, and eliminates the line segments in the trajectory whose lengths are lower than the specified threshold to obtain a set of direction line segment vectors;

[0129] Obtain the continuous coordinate sequence of the instrument end movement from the fuzzy region motion alignment direction set. The coordinate sequence represents the specific position of the instrument at different time points. By connecting adjacent coordinates, a set of line segments representing the movement direction is formed. This process is similar to capturing the dynamic path of the instrument through an endoscope during laparoscopic surgery. Assume that during the operation, by real - time tracking the movement of the instrument in the patient's body, each generated line segment is the straight - line movement path of the instrument from one position to another. Any line segment shorter than three pixels is considered a minor movement or a stationary state, so it is eliminated from the dataset to ensure that only the clearly meaningful movement trajectories are analyzed. Such processing improves the accuracy of the data and the reliability of the analysis. Through screening and processing, a set of direction line segment vectors is finally generated, which will be used for further trend analysis and surgical path optimization to improve surgical accuracy and safety.

[0130] The included - angle value statistics sub - module, according to the set of direction line segment vectors, analyzes the included - angle value based on the dot product of direction vectors, divides it into multiple intervals, and statistically calculates the frequency and direction fluctuation of the intervals. Using the formula:

[0131] ;

[0132] Calculate the change value of the included - angle interval, extract the high - change region according to the index size, and obtain the main included - angle trend interval;

[0133] Among them, represents the change value of the included - angle interval, represents the appearance frequency of the th included - angle interval, represents the average length of the direction line segments in the th included - angle interval, represents the frequency change value of the adjacent interval of the th included - angle interval, represents the fluctuation range of the direction angle in the Indicates the total number of included angle intervals;

[0134] First, extract the direction vectors of all adjacent line segment pairs from the set of direction line segment vectors. The direction vector is the unit vector from the starting point to the ending point of the line segment in a two-dimensional coordinate system. For example, if the starting point of line segment A is (12, 20) and the ending point is (16, 24), then its direction vector is (4, 4), which is normalized to the unit vector (0.707, 0.707). If the starting point of line segment B is (16, 24) and the ending point is (20, 22), the direction vector is (4, -2), which is normalized to the unit vector (0.894, -0.447), and their dot product is , and the calculated included angle is . By performing this operation on all line segment pairs, obtain the entire sequence of included angle values. Divide the included angle values into N categories of intervals according to the range. The upper and lower limits of each category of interval are separated by a fixed angle, set to 15 degrees. The occurrence frequency of each included angle interval is . For example, the occurrence frequency of the first category of included angle interval 0° - 15° is 4 times, the second category is 7 times, the third category is 5 times, and so on. The average line segment length is the average value of the lengths of each line segment in this interval, with the unit of pixel. For example, in the second category, the line segment lengths are 6, 7, 6, 5, 7, 6, 8 pixels respectively, and the average value is pixels. The direction fluctuation value is the standard deviation of the included angle direction in this interval. If the calculation result is , the adjacent frequency change value is the absolute value of the difference between the frequencies of adjacent intervals. The difference between the second category and the first category is ;

[0135] Substitute the parameters into the formula: ;

[0136] After performing similar calculations for the remaining k intervals and summing them up, obtain the final value. For example, the sum is 12.63. This value is used as the screening basis for the main included angle trend interval, representing the comprehensive quantitative output of the current included angle frequency concentration and direction stability. Uniformly adopt the standard angle unit, all angle values are expressed in degrees, the length is in pixels, the frequency is in integer times, and all calculation results maintain consistent dimensions. Finally, the value has no unit and is used as the comprehensive trend evaluation index. This result indicates that the current direction change trend is mainly concentrated within a certain included angle range, which can be used as the basis for extracting subsequent repeated direction intervals. By weighted integration of multi-dimensional data such as frequency, direction change, and line segment length, it effectively distinguishes the main intervals where the direction changes are concentrated, helping to clarify the distribution range of the main direction trend.

[0137] The repeated interval extraction sub-module filters the fragment sequences in which the included angle values continuously fall within the interval in the direction line segments according to the main included angle trend interval, integrates the distribution segments of the sequences, and obtains the repeated trend direction interval;

[0138] Filtering continuous direction line segments according to the main included angle trend interval is similar to analyzing the periodic pattern of heart movement during a heart operation. By identifying and tracking the continuous heart movement directions, doctors can better understand the movement law of the heart. By filtering those line segments whose directions continuously fall within the main included angle trend interval, the dominant trend in the instrument movement can be effectively captured. The trend reflects the consistency and regularity of the instrument operation during the operation, which helps the surgical team predict and plan the best movement path of the instrument, thereby reducing the operation time and potential risks. Through this method, useful movement patterns can be extracted from a large amount of surgical dynamic data. The importance of the steps lies in being able to convert complex dynamic information into practical applicable surgical strategies and providing scientific decision-making support for the surgical team.

[0139] Please refer to Figure 6 , the path response correction module includes:

[0140] The trend interval identification sub-module identifies the continuous direction line segments in the instrument movement trajectory according to the repeated trend direction interval, calculates the angle difference between adjacent line segments, analyzes the continuous interval of the angle difference change, determines the start and end boundaries of the trend direction, and obtains the trend direction interval;

[0141] Through the instrument movement trajectory data, which consists of continuous positioning points and direction vectors, the data points can come from sensors, cameras or visual recognition and are used to determine the movement path of the instrument in three-dimensional space. For example, assume that in consecutive frames, the instrument moves from point A(0, 0) to point B(1, 1), then the direction vector is (1, 1), and calculate the angle difference of the direction line segments between every two frames. If the angle difference value between two direction line segments is greater than a certain set threshold, it is considered that a significant change has occurred in these two directions. From point B to point C(2, 0), calculate the angle difference (from 45 degrees to 90 degrees). If the angle difference is greater than the preset 10 degrees, it is considered that this segment has deviated. Extract the trend direction in the instrument movement trajectory, identify the interval of the angle change between its adjacent direction line segments. The interval defines the "trend direction interval" of the instrument movement. Through multiple calculations and adjustments, finally generate the start and end boundaries of the trend direction and accurately correct them so that the subsequent steps can achieve direction adjustment and optimization. Finally, obtain the trend direction interval, which can be used for subsequent deviation direction detection and movement path correction.

[0142] The deviation direction detection sub-module extracts the current frame instrument direction line segment based on the trend direction interval, compares it with the angle difference within the trend direction interval, filters the direction line segments that deviate from the threshold, and obtains the deviation direction line segments;

[0143] Obtain the direction line segment of the instrument from the current frame and compare it with the recognized trend direction interval. For example, in the current frame, the direction vector of the instrument is (0.8, 1.0), and calculate the angle difference with the trend direction interval. Assume the trend direction interval is from 30° to 45°. If the angle of the current direction line segment is 60°, then calculate the angle difference with the trend interval as 15°. If this difference exceeds the preset offset threshold (such as 10°), it is considered that the direction in the current frame has deviated significantly, and then it is screened. This operation ensures that further adjustment is only made when the direction deviates significantly, avoiding unnecessary error adjustment, and obtaining the deviated direction line segment, which will be used for direction correction in subsequent steps to ensure that the movement path of the instrument can conform to the predetermined trend.

[0144] The direction correction generation sub-module identifies the adjustment angle based on the deviated direction line segment through the angle difference, and corrects the movement direction of the instrument to be close to the trend direction, obtaining the optimized movement path of the surgical assistance instrument;

[0145] First, calculate the angle difference between it and the trend direction interval. Assume the angle of the deviated direction line segment is 60° and the trend direction interval is from 30° to 45°, so the angle difference is 15°. Based on this difference, the minimum angle adjustment method is used for correction. For example, if the angle difference is greater than 5°, then calculate the minimum adjustment angle to correct the instrument direction, and the adjusted angle should be between the minimum angle and the maximum angle of the trend direction interval. In this way, the direction of the original path will be adjusted to be closer to the trend direction, thus forming a corrected movement trajectory of the instrument. When performing this operation, further refinement adjustments will also be made according to factors such as the movement speed and path density of the instrument. For example, if the speed of the instrument is fast and the distance is far, the adjustment step size can be appropriately enlarged, obtaining the optimized movement path of the surgical assistance instrument. This path ensures that the instrument can always maintain a movement trajectory that is more in line with the predetermined trend when performing tasks, thereby effectively improving the path accuracy.

[0146] The above is only the preferred embodiment of the present invention and does not limit the present invention in other forms. Any person skilled in the art may use the disclosed technical content to make changes or modifications into equivalent embodiments with equivalent changes and apply them to other fields. However, as long as it does not depart from the technical solution content of the present invention, any simple modification, equivalent change, and modification made to the above embodiments based on the technical essence of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A microsurgical assistance system based on visual recognition, characterized in that, The system includes: Based on consecutive frame images of the operative field under the microscope, the instrument target positioning module extracts the closed boundary region, judges the boundary closure degree and the image center offset value, locates the center position of the instrument end in the image, and generates a set of spatial coordinates of the instrument end; According to the set of spatial coordinates of the instrument end, the behavior feature analysis module analyzes the angular change rate and the included angle amplitude of the connection lines of the end points in consecutive frames, judges the corresponding behavior type of the line segment fragment, and generates a set of instrument action behavior states; The fuzzy dynamic tracking module calls the set of instrument action behavior states, extracts the image of the microscopic tissue boundary region, identifies the distance between the surgical instrument and the tissue boundary, records the change amplitude of adjacent frames, analyzes the gray gradient and the difference in the instrument edge direction in the jumping region, screens the regions with high fuzziness, and tracks the moving direction of the instrument end in the region to generate a set of fuzzy region motion alignment directions; The fuzzy dynamic tracking module includes: The boundary image extraction sub-module calls the set of instrument action behavior states, identifies the current position and posture of the instrument in the microscopic tissue image, extracts the image of the microscopic tissue boundary region of the instrument in the image, and obtains a set of instrument boundary registration images; Based on the set of instrument boundary registration images, the jumping region screening sub-module analyzes the gray change amplitude between the instrument edge and the tissue boundary in adjacent frames, screens the image regions with large gray change and large difference in edge direction, and generates a map of high gradient difference regions; According to the map of high gradient difference regions, the end movement direction extraction sub-module tracks the position of the instrument end in the region frame by frame, extracts the coordinate change and the region direction vector corresponding to the end in consecutive frames, identifies the matching degree and the angle increment of the end movement direction, and uses the formula: ； Calculate the fitting value of the direction corresponding to the end in the region. According to the change trend in the fitting value sequence, generate a set of fuzzy region motion alignment directions; Among them, represents the fitting value of the end corresponding area direction, represents the velocity vector of the instrument end at the represents the frame, and the end coordinate difference vector between the represents the frame and the end direction angle change value of the represents the modulus sequence of the area direction vector group at the is the number of consecutive tracking frames; Based on the set of fuzzy region motion alignment directions, the action trend analysis module extracts the motion direction line segments, identifies the included angle values between adjacent direction line segments and counts the occurrence frequencies, screens the continuous direction sequences, and obtains the repeated trend direction intervals; According to the repeated trend direction intervals, the path response correction module compares the angle of the current instrument direction line segment with the angle of the trend interval, screens the line segment positions deviating from the direction and adjusts the motion direction in the corresponding frames to generate an optimized motion path of the surgical assistance instrument; The optimized motion path of the surgical assistance instrument includes a corrected motion trajectory, a direction optimization parameter, and a deviation correction position.

2. The microsurgical assistance system based on visual recognition according to claim 1, wherein The set of spatial coordinates of the instrument end includes spatial position information, an image reference offset, and a center indication parameter. The set of instrument action behavior states includes a behavior recognition label, an angular change index, and a motion amplitude parameter. The set of fuzzy region motion alignment directions includes a direction change feature, a fuzzy edge attribute, and a region dynamic index. The repeated trend direction intervals include a continuous direction pattern, an included angle change distribution, and a trend stability parameter.

3. The microsurgical assistance system based on visual recognition according to claim 1, wherein The instrument target positioning module includes: Based on consecutive frame images of the operative field under the microscope, the image sequence extraction sub-module identifies the image data of each frame, detects the brightness gradient region in each frame image, integrates the gray difference and the change in texture distribution, and obtains a set of dynamic sequence frames of the image; The boundary judgment analysis sub-module calls the dynamic image sequence frame set, extracts a closed boundary according to the continuity of pixel edges, identifies the boundary contour point set, analyzes the closure degree and connection integrity, measures the distance between the contour centroid and the image center and classifies it, analyzes the boundary closure degree and the center offset, and uses the formula: ； Calculate the offset correlation degree value, sort the offset correlation degree values in each frame of the image, and screen the front regions to obtain the effective boundary positioning interval value; Among them, represents the offset correlation value, represents the pixel coordinates of the center of gravity of the contour in the th frame, represents the pixel coordinates of the center point of the image, is the closure value of the th frame boundary, represents the offset pixel value of the candidate region, is the full-frame offset reference pixel value, is the number of valid frames, is the number of candidate regions; The spatial coordinate recognition sub-module selects the pixel density region inside the boundary according to the effective boundary positioning interval value, extracts the geometric center coordinates, combines the imaging parameters and the scale factor to recognize the three-dimensional spatial position, and generates the instrument end spatial coordinate set.

4. The microsurgical assistance system based on visual recognition according to claim 3, wherein, The behavior feature analysis module includes: The trajectory speed calculation sub-module selects the end coordinates of two adjacent frames according to the instrument end spatial coordinate set, identifies the spatial distance, and extracts the speed within the unit time according to the time interval to obtain the line segment angle change speed value; The included angle amplitude extraction sub-module calls the line segment angle change speed value, identifies the included angle based on the vectors of adjacent line segments among three frames and screens out the segments exceeding the rotation amplitude threshold to obtain the line segment included angle amplitude interval value; The behavior type determination sub-module calculates the line segment action behavior feature measurement value according to the line segment angle change speed value, and judges it with the behavior classification boundary value, using the formula: ； Generate the instrument action behavior state group; Among them, represents the measurement value of the line segment action behavior feature, represents the angular change rate, represents the angular change reference value, and represent the upper and lower limits of the included angle amplitude interval respectively, is the stability factor of the behavior , is the number of behaviors.

5. The microsurgical assistance system based on visual recognition according to claim 1, wherein The action trend analysis module includes: The direction line segment extraction sub-module identifies the continuous coordinate sequence of the instrument end movement path based on the fuzzy region movement alignment direction set, connects adjacent coordinates to form a direction line segment, and eliminates the line segments with lengths lower than the specified threshold in the trajectory to obtain the direction line segment vector set; The included angle value statistics sub-module analyzes the included angle value according to the direction line segment vector set based on the dot product of the direction vectors, divides it into multiple intervals, and counts the frequency and direction fluctuation of the intervals, using the formula: ； Calculate the included angle interval change value, extract the high change region according to the index size to obtain the main included angle trend interval; Among them, represents the change value of the included angle interval, represents the appearance frequency of the included angle interval, represents the average length of the direction line segments within the included angle interval, represents the frequency change value of the adjacent interval of the included angle interval, represents the fluctuation range of the direction angle within the included angle interval, represents the total number of included angle intervals; The repeated interval extraction sub-module screens the segment sequences in which the included angle values of the direction line segments continuously fall within the interval according to the main included angle trend interval, and integrates the distribution segments of the sequences to obtain the repeated trend direction interval.

6. The microsurgical assistance system based on visual recognition according to claim 1, wherein The path response correction module includes: The trend interval recognition sub-module identifies the continuous direction line segments in the instrument movement trajectory according to the repeated trend direction interval, calculates the angle difference between adjacent line segments, analyzes the continuous interval of the angle difference change, determines the start and end boundaries of the trend direction, and obtains the trend direction interval; The deviation direction detection sub-module extracts the current frame instrument direction line segment based on the trend direction interval, compares it with the angle difference within the trend direction interval, and screens out the direction line segments with deviation thresholds to obtain the deviation direction line segments; The direction correction generation sub-module identifies the adjustment angle according to the deviation direction line segment through the angle difference, and corrects the instrument movement direction to be close to the trend direction to obtain the optimized surgical auxiliary instrument movement path.

Images