Surgical instrument abnormal state recognition method based on image recognition

Abstract

The invention discloses a surgical instrument abnormal state recognition method based on image recognition, particularly relates to the field of image recognition, and is used for solving the problem that an existing surgical instrument structure is abnormal and detained. Instrument identification is executed by collecting image stream data covering an operation field space; dividing a shielding area in the surgical field according to an image shielding form, and counting a high-risk part of an instrument structure in combination with medical surgical accident data; identifying a start-stop image frame when the instrument enters and exits from the sheltered area, constructing a polar coordinate system based on the main axis direction and the center of the instrument, performing angle sorting and sliding comparison on the edge of the instrument under the coordinate system, and focusing and identifying the structure defect of the high-risk part; and carrying out overlapping analysis on the identified structure loss abnormity and the surgical field shielding area, and outputting a closed area set in which the component may be retained. According to the scheme, abnormal focusing of the instrument fracture event in the operation and part retention risk space positioning are achieved, and the accuracy and efficiency of risk management and control in the operation are improved.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and more specifically, to a method for identifying abnormal states of surgical instruments based on image recognition. Background Technology

[0002] In minimally invasive and laparoscopic surgeries, the problem of abnormal instrument retention poses a serious safety hazard in clinical practice. Its essence does not stem from failed instrument identification or forgotten inventory, but rather from instruments breaking or detaching after entering obstructed areas of the surgical field during surgery, or from temporary instruments failing to be physically removed without being identified as hazardous, resulting in some instrument structures remaining in the body post-surgery. This type of retention often lacks significant visual characteristics, showing neither violent tissue reactions nor necessarily being marked as abnormal by the intraoperative procedure. Furthermore, factors such as the alternating use of multiple instruments, frequent instrument switching, and misalignment of the assistant's and surgeon's coordination paths during surgery easily lead to situations where the main instrument has been removed but components remain. Adding to the complexity, this type of instrument retention is often not caused by the operator's subjective actions, but rather passively falling into tissue gaps, sliding into deep cavities, or failing to identify its unretrievable location due to obstruction, posing a high degree of danger.

[0003] Currently, there is a lack of intraoperative identification mechanisms for such dangerous retention scenarios, especially when using instruments with complex structures and detachable components. These highly concealed instruments, whose abnormalities are not visible in images, are more easily overlooked. Most existing methods involve manually counting or using auxiliary scanning to check the integrity of instruments postoperatively. However, these postoperative remedial methods are inefficient and cannot detect dangers during surgery in a timely manner, requiring a second surgery to remove them. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide a surgical instrument abnormality identification method based on image recognition to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A method for identifying abnormal states of surgical instruments based on image recognition includes the following steps: S1: Acquire image stream data of the surgical field space during surgery and identify surgical instruments, extracting the temporal position of each instrument in each frame of the image; S2: Based on the image occlusion pattern within the surgical field space, the surgical field space occlusion area is divided; S3: Obtain archived medical surgical accident data, count the locations in the accident data where instrument breakage or component detachment has occurred, and generate a set of high-risk parts of the instrument structure. S4: Identify and extract the start and end image frames of the instrument entering and exiting the occluded area, and establish a polar coordinate system based on the principal axis center; S5: In the polar coordinate system, the start and end image frames of the occlusion area are sorted by angle to generate a spiral edge sequence and a sliding matching comparison is performed to target and focus on high-risk areas. S6: Acquire all instruments with structural loss anomalies, and output the residual area after component loss in the surgical field space in combination with the occlusion area.

[0006] In a preferred embodiment, in S1, acquiring image stream data of the surgical field space during surgery and identifying surgical instruments, specifically extracting the temporal position of each instrument in each frame of the image includes: Acquire image stream data covering the surgical field space during surgery, and convert the image stream data into an image frame sequence; Based on a deep semantic feature encoding network, we identify medical device instances and classify medical devices in image frames, and remove image frames with blurred edges of medical device instances from the image frame sequence. Two-dimensional image spatial envelopes of different instruments are extracted from the image frame sequence. The two-dimensional position coordinates of the instruments in the surgical field coordinate system are calculated. The image planar motion trajectory of the instruments is constructed based on the two-dimensional position coordinates of the instruments.

[0007] In a preferred embodiment, the calculation of the two-dimensional position coordinates of the instrument in the surgical field coordinate system is specifically performed by performing image registration on consecutive frames in the image frame sequence, extracting static texture points in adjacent image frames, constructing a surgical field reference plane based on the distribution of static texture points in the non-instrument background area in the surgical field space, setting the origin, horizontal axis and vertical axis to establish a surgical field coordinate system, mapping the spatial envelope of the instrument image appearing in all image frames to the surgical field reference plane, and marking the two-dimensional planar position of the instrument in the surgical field coordinate system.

[0008] In a preferred embodiment, in S2, dividing the surgical field occlusion region based on the image occlusion pattern within the surgical field specifically includes: Based on the determination of brightness gradient direction breakage and texture continuity interruption, the spatial overlap between the instrument image spatial envelope and the non-instrument background area in the surgical field space is identified in the image frame, and the occlusion candidate area in the image is delineated. Connectivity is determined for occlusion candidate regions in the image frame sequence, and segments with continuously increasing overlapping areas and closed edges are selected as effective occlusion segments; All effective occluded segments are reconstructed into spatial occluded blocks in the surgical field coordinate system, and a set of spatial occluded regions in the surgical field is generated based on the contours of the spatial occluded blocks.

[0009] In a preferred embodiment, in S3, archived medical surgical accident data is obtained, and the locations in the accident data where instrument breakage or component detachment has occurred are statistically analyzed. A high-risk component tag set for instrument structures is generated, specifically including: Retrieve archived medical malpractice reports containing data on surgical instrument breakage and foreign body residue incidents; The structure of the instruments in the accident data collection of instruments broken and foreign objects left behind is deconstructed to mark the specific fracture points and the locations of the detached components. Based on the types and models of equipment involved in the accident, the specific locations on the structures of various types of equipment that repeatedly break or detach are statistically analyzed to generate a high-risk structural node index set. The high-risk structural node index set is back-labeled to the instrument structure to form a high-risk structural part label set.

[0010] In a preferred embodiment, in S4, identifying and extracting the start and end image frames of the instrument entering and exiting the obstructed area, and establishing a polar coordinate system based on the principal axis center specifically includes: The first overlapping frame and the last de-overlapping frame between the spatial envelope of each instrument image and the set of occluded regions in the surgical field space are identified in the image frame sequence and labeled as the occlusion entry frame and occlusion exit frame, respectively. In the occlusion entry frame and occlusion exit frame, the device image must be determined to have the same device posture. If they are not consistent, continuous matching and retrieval will be performed before the first overlapping frame and after the last unoverlapping frame. Extract the two-dimensional image spatial envelope of the instrument in the occlusion entry frame and occlusion exit frame, and combine it with the position trajectory in the surgical field coordinate system to perform instrument image principal axis direction fitting and structural center point localization; Using the direction of the instrument's main axis and the center point of the structure, a two-dimensional polar coordinate system is defined with the center of the structure as the pole and the direction of the main axis as the polar axis.

[0011] In a preferred embodiment, in S5, in the polar coordinate system, the start and end image frames entering and exiting the occlusion area are sorted by angle to generate a spiral edge sequence and a sliding matching comparison is performed. Targeted focusing identification of high-risk areas specifically includes: In the constructed polar coordinate system, the spatial envelope of the instrument image in the occlusion entry frame and the occlusion exit frame is expanded along the angular direction to generate a spiral edge distribution sequence. Arrange the spiral edge distribution sequence in ascending order of polar angle to form two sets of spiral edge vectors with the same direction. Perform sliding window matching on the two sets of spiral edge vectors, calculate the edge offset index under the same angular direction, and output the structural difference map; Extract the polar angle segments corresponding to the label set of high-risk structural parts from the structural difference map, and perform structural difference analysis to determine whether there are defective segments with texture breaks or shortened contours. If so, mark them as abnormal instrument structure loss.

[0012] In a preferred embodiment, in S6, all instruments with structural loss anomalies are acquired, and the area where the component remains after loss is output within the surgical field space, combined with the obstruction area, specifically includes: The defective segment corresponding to the structurally missing instrument is reverse-mapped to the two-dimensional image space envelope of the instrument in the image frame to generate the image space missing region. Based on the overlapping location of the obscured area and the missing area in the surgical field, the possible breakage and detachment range of the component can be estimated. Perform connectivity and closure tests on all fractured and detached areas, and output a set of structurally closed component retention areas.

[0013] The technical effects and advantages of the image recognition-based surgical instrument abnormality identification method of the present invention are as follows: By constructing a pre- and post-operative image comparison mechanism based on occlusion zones, real-time identification of potential breakage or component detachment anomalies that may occur when instruments enter occluded areas of the surgical field is achieved. This significantly improves the intraoperative image analysis system's ability to identify instrument risks under occlusion conditions. Through image frame alignment in the surgical field coordinate system and the construction of a polar coordinate system at the principal axis center, spatial consistency and structural focus during the comparison process are ensured. Combined with a targeted identification strategy using a high-risk structural part label set, the false positive rate in non-critical areas can be effectively reduced, enhancing the accuracy and engineering practicality of identification. Especially in cases where complete occlusion prevents continuous observation of the instrument's integrity, this solution analyzes the edge evolution path of images entering and exiting the occluded area, achieving sensitive detection of instrument structural loss. This avoids the risk of missed identification due to occlusion failure in traditional methods, enabling early detection and location locking of intraoperative instrument breakage and component detachment. This provides accurate evidence for postoperative foreign body removal, significantly reducing residual risks and the probability of postoperative complications, demonstrating significant safety value and clinical application significance. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of a surgical instrument abnormality identification method based on image recognition according to the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0016] Example 1 Figure 1 This invention provides a method for identifying abnormal states of surgical instruments based on image recognition, comprising the following steps: S1: Acquire image stream data of the surgical field space during surgery and identify surgical instruments, extracting the temporal position of each instrument in each frame of the image; S2: Based on the image occlusion pattern within the surgical field space, the surgical field space occlusion area is divided; S3: Obtain archived medical surgical accident data, count the locations in the accident data where instrument breakage or component detachment has occurred, and generate a set of high-risk parts of the instrument structure. S4: Identify and extract the start and end image frames of the instrument entering and exiting the occluded area, and establish a polar coordinate system based on the principal axis center; S5: In the polar coordinate system, the start and end image frames of the occlusion area are sorted by angle to generate a spiral edge sequence and a sliding matching comparison is performed to target and focus on high-risk areas. S6: Acquire all instruments with structural loss anomalies, and output the residual area after component loss in the surgical field space in combination with the occlusion area.

[0017] In S1, image stream data of the surgical field space during surgery is acquired and surgical instruments are identified, and the temporal position of each instrument in each frame of the image is extracted.

[0018] During the surgery, multi-view image recording methods are deployed to acquire image stream data covering the entire surgical field. The acquired data must meet a time sampling frequency of at least 25 frames per second and be saved as a continuous image frame sequence in a uniform format. To ensure subsequent recognition accuracy, the image frames need to undergo preliminary preprocessing, including resolution unification (e.g., unification to 1920×1080 pixels) and illumination histogram adjustment, to ensure that image contrast and tone distribution are within a controllable range, avoiding edge recognition misjudgment caused by image brightness fluctuations or shadows.

[0019] The task of instrument recognition is performed on the image frame sequence. A structured semantic feature encoding network model is used to perform semantic segmentation and instance recognition for each image frame. The structured semantic feature encoding network model is based on a multi-task branch architecture of Mask R-CNN, with a backbone of ResNet-101 and FPN to extract spatial semantic features at different scales. The training dataset comes from multiple surgical scene image annotation sets, covering the categories and structural partitioning information of commonly used surgical instruments. During the recognition process, all surgical instruments appearing in the image are labeled with their category tags (e.g., forceps, scissors, retractors, etc.), and the corresponding image spatial envelope region, i.e., the boundary contour of the instrument in that frame, is extracted in the 2D image. The recognition results are output in the form of instances, along with the pixel position index of each instance in the image frame. Image preprocessing operations are performed on the image frame sequence. Specifically, this includes: unifying the spatial size of the image frames (e.g., scaling to a uniform resolution), standardizing the image grayscale range, performing histogram equalization to unify illumination contrast, and initially screening the image frame sharpness based on edge sharpness gradients. Image sharpness is determined by using the Sobel operator to calculate the mean of gradient magnitude in the image edge region. Frames with a gradient magnitude below a set threshold (e.g., frames with a mean edge gradient below 15) are identified as blurry frames and removed to avoid incomplete feature extraction in subsequent recognition.

[0020] A clear spatial localization representation is established for each identified instrument in each image frame. Specifically, a two-dimensional image spatial envelope is extracted for each instrument target in the image frame to define the boundary range of each instrument in that frame. The image spatial envelope is defined as the smallest closed boundary structure of the visible area of ​​the instrument in the current image frame, often represented by polygonal or elliptical edge fitting results. The envelope extraction process uses the target mask from the instrument instance identification results in the image, and performs edge extraction operations on the mask area to eliminate contour burrs and low-contrast background interference. After completing the instrument envelope extraction, the envelope position in the image space is mapped to a unified reference system in the surgical field space to establish a two-dimensional localization model in the surgical field coordinate system. To achieve this mapping, the surgical field coordinate system is first constructed. The surgical field coordinate system should have two characteristics: first, the spatial reference object should be the background area in the image that does not change significantly over time; second, the definition of the coordinate axes should cover the entire surgical field plane and have spatial consistency. To this end, image registration is first performed on consecutive images in the image frame sequence. The purpose of image registration is to correct spatial offsets caused by factors such as shooting angle, parallax, and vibration in multiple images, thereby achieving alignment of the same location across multiple frames. In image registration, a feature point detection algorithm is used to extract highly stable texture points from image frames, especially in the background region (i.e., the non-instrument area), selecting widely distributed, clearly defined corner points, stripes, and texture boundaries as static texture points. To ensure these texture points possess static properties, their spatial position stability needs to be verified across image frames at multiple time points. The stability criterion is: the center position of a texture point in the image coordinate system shifts by no more than 2 pixels in five adjacent frames. Texture points meeting this condition constitute a static point set, serving as spatial reference points for the surgical field background area. Based on this static texture point set, its overall distribution density and directional consistency across image frames are analyzed. The image frame with the most uniform texture point distribution and the largest coverage area is selected as the surgical field reference frame, and the set of texture points in this frame is used to construct the surgical field reference plane. After construction, the centroid of the texture point in the reference plane is used as the origin of the surgical field coordinate system. The direction from the centroid to the horizontal boundary of the image is defined as the horizontal axis (x-axis), and the vertical direction is defined as the y-axis, forming a complete two-dimensional surgical field coordinate system.

[0021] After establishing the surgical field coordinate system, the center point of the spatial envelope of the instrument image in all image frames is mapped to this coordinate system. The image coordinates in each image frame are then uniformly transformed to planar coordinates within the surgical field coordinate system. The two-dimensional image position of any instrument in an image frame is uniformly described as a planar point in the surgical field coordinate system, possessing absolute positional reference significance. As the image frame sequence progresses, the above operations are continuously performed to obtain the two-dimensional position point set of each identified instrument in the surgical field coordinate system across consecutive image frames. These points are then organized chronologically to construct the planar motion trajectory of the instrument. Each node in the trajectory consists of the following fields: image frame number, position in the surgical field coordinate system (x, y), and the boundary of the instrument's spatial envelope in the corresponding image frame. This planar trajectory will provide a unified spatial-temporal reference for subsequent identification of entry and exit points of occluded areas, establishment of polar coordinates, and matching and comparison of spiral edges.

[0022] In S2, the occlusion region of the surgical field is divided based on the image occlusion pattern within the surgical field space.

[0023] Because instruments frequently move between tissue structures during operation, they are prone to partially entering or exiting the field of view, or being obscured by tissue. These obscurations can mask abnormal conditions of the instrument structure. Therefore, it is essential to identify the location and time period of obscuration in the image and establish a set of regions in the surgical field space corresponding to the obscuration behavior. Using image frames in the surgical field coordinate system as input, the spatial overlap area between the calibrated two-dimensional image space envelope region of the instrument and the non-instrument background region in the surgical field space is calculated. Specifically, a boundary comparison method is used to detect the image transition features between the edge of the instrument image space envelope and the background region to determine whether there are signs of obscuration.

[0024] The identification of occlusion here is based on two main indicators: the break in the brightness gradient direction and the interruption of texture continuity. The identification of brightness gradient direction breakage is based on the image grayscale gradient field, calculated by performing Sobel calculations on local images at the envelope edge. If there is a significant angle change (exceeding 30°) between the brightness gradient direction at the edge of an image segment and the interior of the envelope, or if the gradient direction is discontinuous, it is identified as a gradient breakage. Typically, this breakage indicates occlusion interference between the instrument and tissue at the boundary, rather than a complete edge. The determination of texture continuity interruption uses an image texture direction consistency measure. Image texture vectors within a 10-pixel range inside and outside the envelope edge are extracted, and their direction distribution density and direction distribution dispersion indices are calculated. If there is a significant break in the texture direction inside and outside the envelope (e.g., an increase in direction standard deviation exceeding 20%), it is marked as a texture interruption region. This processing logic ensures that occlusion judgment not only relies on brightness changes but also incorporates the physical logic of tissue texture continuity. Based on the above two types of breakage regions, image morphological processing methods (such as dilation, erosion, and boundary tracking) are used to delineate complete and closed occlusion candidate regions. All regions that meet the criteria of boundary closure greater than 85% and occlusion area greater than 2% of the total image area are retained as preliminary candidates.

[0025] After candidate region identification, connectivity is assessed within the image frame sequence. Specifically, each image frame's occlusion candidate region is numbered, and spatial connectivity is analyzed based on the continuity of the trajectory of the occlusion region's center position in the surgical field coordinate system. If a candidate region exhibits a constant or increasing area trend in adjacent frames, and its boundary closure continuously increases (e.g., boundary integrity consistently exceeds 90%), it is designated as a valid occlusion segment. After filtering valid occlusion segments, spatial reconstruction is performed in the surgical field coordinate system. Based on the two-dimensional boundary points of the valid occlusion region in each frame's surgical field coordinate system, occlusion boundaries in consecutive frames are superimposed in temporal order to form spatial occlusion and reconstruct its contour on the surgical field plane, resulting in an occlusion region representation. Spatial occlusion is achieved using temporal envelope merging, which involves fusing the occlusion boundary point sets of adjacent frames to generate a unified occlusion boundary within the time window. The final result is an occlusion region representation that includes attributes such as the occlusion start frame, end frame, occlusion duration, occlusion boundary coordinate set in the surgical field coordinate system, maximum occlusion area, and center position.

[0026] In S3, archived medical surgical accident data is retrieved, and the locations in the accident data where instrument breakage or component detachment has occurred are identified, generating a set of high-risk parts of the instrument structure.

[0027] The existing medical malpractice archives were centrally organized. This included publicly available records of intraoperative surgical instrument abnormalities from medical institutions at all levels, such as instrument breakage, component loosening, and foreign body residue. Examples include "fractured gripper tip," "dislodgement of the mandible from tissue scissors," and "damage to the insulation layer of electrocautery hooks." A time frame (e.g., within the last five years) was set as a screening criterion for data collection. Structural analysis was then performed on the screened data on instrument breakage and component dislodgement. This analysis focused on the physical structure of the surgical instruments. The accident records all included subjective descriptions of the damaged parts. Combined with publicly available structural diagrams of the instruments, the specific location of the abnormality was mapped to the structural composition hierarchy. For example, in double-bend grippers, common breakage points were the distal gripper head connection or the lower gripper claw fixing rivet point; in laparoscopic tissue scissors, common occurrences included blade breakage or detachment of the movable pivot at the blade root. Each incident required the specific instrument model, the structural name of the damaged part, and a description of its physical location (e.g., "outer edge of the gripper head connecting arm"), and was recorded as a structural location point record.

[0028] All analyzed structural anomalies are summarized and categorized by instrument type and model to form a list of high-risk structural parts. This statistical process uses the frequency of occurrence of instrument parts as a basis to filter out structural nodes with high repetition and overlap rates. For example, in the statistical analysis of gripping forceps, the inner connection point of the lower jaw was found to have broken 16 times in 20 accidents, classifying this connection point as a typical high-risk structural part. The statistical threshold can be set as more than 30% of the total number of accidents within the same model, or a structural position repetition rate exceeding 40% across models, as high-risk criteria. Based on the statistical results, a high-risk structural node index set is constructed. This index set uses the instrument type as the primary key, with several structural parts labeled under each instrument type. Each structural part includes the following fields: structural name (e.g., "gripper movement hub"), node number (named according to the sequence identifier of the component in the structural diagram), frequency count, list of involved models, and accident example index. This structural node index set is then mapped to a standardized instrument structural diagram model to form a high-risk structural part label set. The tag set is coded in the form of structural diagram node component descriptions, and each high-risk structural part is marked with a prominent identifier in the corresponding device structural diagram.

[0029] In S4, the start and end image frames of the instrument entering and exiting the occluded area are identified and extracted, and a polar coordinate system is established based on the principal axis center.

[0030] For the identified instrument image spatial envelope and surgical field spatial occlusion region set in the surgical image frame sequence, a frame-by-frame spatial overlap analysis is performed. First, according to the image frame numbering order, the spatial overlap between the instrument image spatial envelope and the boundary of the occlusion region in each frame is calculated. Based on the percentage of the overlapping area in the pixel area of ​​the surgical field coordinate system, the image frame that first exceeds a set overlap rate threshold is extracted and marked as the occlusion entry frame for that instrument. The occlusion rate threshold is set to 20% of the instrument image spatial envelope area. That is, when the portion of the instrument envelope area that overlaps with the occlusion region first exceeds 20% of the total envelope area, it is judged as entering an occlusion state; conversely, when this percentage first drops below the threshold in a continuous frame sequence, the image frame is marked as an occlusion exit frame. The above operations can be performed independently for each instrument to ensure accurate differentiation of occlusion entry and exit nodes in the case of concurrent occlusion of multiple instruments. After the occlusion entry and occlusion exit frames are marked, it is necessary to further confirm whether the instrument posture in the two frames is consistent. To perform this judgment, the principal axis direction angle and the centroid position of the spatial envelope of the two instrument images are extracted, and the angle between the direction vectors and the centroid offset magnitude are calculated. The principal axis direction is the linear fitting direction of the main extension direction in the spatial envelope of the instrument image. The centroid position (image centroid) is the geometric center point of the contour region of the instrument image in the image space; it is the positional center of the instrument on the image, regardless of quality or color, based solely on the shape of the region.

[0031] When the angle between the principal axes is less than 5 degrees and the centroid position offset is less than a set pixel difference (e.g., 15 pixels) in the image coordinate space, the two frames are considered to have the same pose. If either indicator exceeds the set range, out-of-frame matching is initiated. Specifically, a maximum of 10 frames are traced back before the occlusion enters the frame, and a maximum of 10 frames are traced back after the occlusion exits the frame. The pose of the instrument image in each frame is compared and matched with the pose of the target frame. The image frame with the closest pose and image clarity not lower than the average of the preceding and following frames is selected as the replacement frame. This replacement frame serves as the base image for subsequent principal axis extraction and polar coordinate system construction, ensuring consistency in the analysis.

[0032] After acquiring occlusion entry and exit frames with consistent poses, the principal axis direction fitting and structural center point localization operations are performed. The fitting operation is based on the contour boundary point set of the instrument image envelope region in image space, and the principal axis direction is extracted using the least squares linear fitting method, outputting an orientation vector in image space. The geometric centroid of the image envelope region is used as the initial estimate of the structural center point, and the initial centroid is corrected for stability by combining it with low-speed displacement segments in the continuous frame trajectory of the instrument in the surgical field coordinate system, finally determining the two-dimensional coordinates of the instrument structural center point in the surgical field coordinate system. After the principal axis direction and structural center point localization are completed, polar coordinate system construction is performed. The specific method is as follows: with the structural center point as the pole and the principal axis direction vector as the polar axis direction, a 0° angle starting line is set along the positive polar axis direction, and the 0 to 360 degree full rounded corner range is divided at equal intervals (default 5°).

[0033] In S5, in the polar coordinate system, the start and end image frames of the occlusion area are sorted by angle to generate a spiral edge sequence and a sliding matching comparison is performed to target and focus on high-risk areas.

[0034] In the constructed polar coordinate system, the spatial envelope of the instrument image, as defined in the occlusion entry and exit frames, undergoes an angle unfolding operation. Specifically, for the contour edge of the spatial envelope of the instrument image in each frame, the center point of the established polar coordinate system is used as the pole, and the principal axis direction is used as the polar axis. A polar projection transformation is performed on the contour edge points. The position of each discrete pixel on the contour edge in the image coordinate system is converted to polar coordinates with the pole as the origin, forming a polar radius and polar angle pair for the edge points. By setting a polar angle resolution interval, all edge points are grouped in ascending order of polar angle, and the maximum value of the polar radius in each angle group is counted as the edge distribution index under that polar angle direction. The above process is applied to the occlusion entry and exit frames respectively, forming two sets of spiral edge distribution sequences with consistent directions. To avoid the impact of angle errors on matching accuracy, a high-resolution angle granularity is used when dividing the polar angles, and the starting point of the polar angle is uniformly set to ensure the consistency of correspondence between sequences. Subsequently, a sliding window matching operation is performed on the spiral edge vector set composed of the above two sets of spiral edge distribution sequences. In this operation, a fixed-length polar angle window (e.g., 10 degrees) is selected for local alignment between two sets of spiral edge vectors, and the edge distance offset of corresponding polar angle segments within the window is calculated. This edge distance offset is defined as the sum of the absolute values ​​of the differences in polar radius values ​​corresponding to the same polar angle, reflecting the degree of edge contour change in the two frames in that direction. During the matching process, sliding analysis is performed on all polar angle segments to obtain a complete edge difference distribution map, i.e., a structural difference map.

[0035] After constructing the structural difference map, the polar angle segments corresponding to the high-risk structural parts label set are extracted for focused analysis. The high-risk structural part label set is constructed in the previous steps using the structural node index set of frequently fractured areas in the instrument structural model; here, it needs to be converted into polar angle intervals. The conversion method is as follows: for each high-risk structural node, component correspondence is established, and polar coordinate transformation is performed at the two-dimensional image envelope position in the instrument space. The corresponding polar angle positions of the structural node edge in the occlusion entry and exit frames are calculated, and a structural tolerance range (default ±5 degrees) is set to expand and form a high-risk polar angle segment set. Based on this set, the corresponding segments are located in the structural difference map, and the edge offset trend, gradient abruptness phenomenon, and local polar diameter shortening phenomenon are judged.

[0036] In the extracted structural difference map, if multiple consecutive polar points (e.g., more than 5 consecutive angular units) within a high-risk polar angle segment show a continuous increase in edge offset values, and this increase exceeds a set threshold (e.g., the polar radius offset between consecutive frames is greater than 20% of the original polar radius value), then the edge structure in that direction is considered to have undergone abnormal deformation. Simultaneously, within the same polar angle segment, if a sudden drop in polar radius value is observed—that is, the farthest point of the edge in that direction suddenly shrinks inward, and the average polar radius is more than 30% lower than the historical polar radius value of the corresponding polar angle segment under normal operating conditions (a reference polar radius can be constructed from unoccluded frames)—it indicates that the edge region has experienced length compression, which is usually closely related to structural breakage or component detachment. For the instrument edge region corresponding to this polar angle segment in the occlusion exit frame, image texture continuity analysis is performed. This analysis extracts the grayscale gradient direction map and gradient intensity distribution of the device contour region, and performs linear projection statistics on the edge texture corresponding to the polar angle direction. If there is a significant abrupt change in gradient direction or a break in texture line segments, and texture holes or blurred texture bands are formed in the edge region (regional gradient intensity is lower than a set threshold, such as less than 10 average grayscale gradient), then the texture continuity of that region is considered interrupted. When the above three conditions—significant edge offset, sudden drop in polar diameter value, and interruption of texture continuity—are met simultaneously, it can be determined that the device structure region corresponding to the polar angle segment has a significant anomaly and is no longer in a complete configuration state. This anomaly can be caused by factors such as structural fracture, detachment of end components, or surface peeling. Based on this, the polar angle segment is recorded as the location of structural defect and matched with the label set of high-risk structural parts of the device for confirmation. If the polar angle segment belongs to a high-risk structural part, it can be directly marked as an abnormality of device structural loss.

[0037] In S6, all instruments with structural loss anomalies are acquired, and the occlusion area is combined with the output area of ​​the lost component in the surgical field space.

[0038] After an instrument structure loss anomaly is determined, the edge contour information of the missing segment in the polar coordinate system should be back-projected onto the corresponding two-dimensional image space envelope of the image frame based on the polar angle localization result of the missing segment. Specifically, firstly, the position of the instrument structure center point used to construct the polar coordinate system in the image coordinate system is extracted, and the corresponding principal axis direction vector is retained. Then, the boundary vector covered by the polar angle segment is rotated and mapped back to the image space along the principal axis direction. The contour path of the missing segment in the image frame is reconstructed in the image space using the polar radius length and polar angle direction. This path should intersect the contour of the instrument image space envelope and form a closed or approximately closed region within the contour. This region is defined as the missing region in the image space and marked on the image frame using a binary mask for occlusion relationship and retention estimation.

[0039] After annotating the missing regions in the image space, spatial relationship analysis is performed between the missing regions and the occluded regions in the surgical field to determine the possible locations where the defective parts may remain or detach after detachment. Specifically, the following steps are taken: First, the constructed mask of the missing region and the set of occluded regions in the surgical field are aligned in the image frame space. Since the occluded regions in the surgical field have already been converted into two-dimensional polygonal regions with clear boundaries and coverage in the surgical field coordinate system during the previous processing, the image frame containing the missing region is mapped to the surgical field coordinate system through image registration, and the spatial overlap between the two regions is determined by coordinate overlap. When the surgical field coordinates corresponding to any pixel point in the missing region in the image space are within the boundary of the occluded region, that point can be considered a possible landing point for the broken or detached part. Further, all missing region point sets with overlapping relationships are aggregated to form the possible landing range of the part. If multiple missing segments span multiple occluded regions, multiple landing range sets should be generated separately.

[0040] After estimating all possible areas where components might remain, the structural closure and connectivity of these areas are verified to ensure the physical conditions for actual retention of the instrument components are met. For each area, connectivity detection using binary graphics is performed, employing standard 4-neighbor or 8-neighbor algorithms to identify connected regions and excluding isolated points or discrete regions that are too small (area below a set threshold, such as less than 100 pixels in the surgical field). Subsequently, boundary extraction and contour tracking are performed on each connected region to determine if it constitutes a closed structure. If the region boundary can be closed into a closed polygon without edge breaks, the region is considered structurally closed and can form a potential retention area for components. If the region boundary has significant gaps (e.g., closure less than 90%), the region is excluded. Finally, all regions satisfying connectivity and closure are included in the component retention area set and archived according to instrument number and image frame timestamp, serving as a signal for intraoperative surgical instrument breakage and foreign body retention, as well as a basis for postoperative traceability.

[0041] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0042] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0043] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0044] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0045] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0046] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0047] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0048] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0049] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0050] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for identifying abnormal states of surgical instruments based on image recognition, characterized in that, Includes the following steps: S1: Acquire image stream data of the surgical field space during surgery and identify surgical instruments, extracting the temporal position of each instrument in each frame of the image; S2: Based on the image occlusion pattern within the surgical field space, the surgical field space occlusion area is divided; S3: Obtain archived medical surgical accident data, count the locations in the accident data where instrument breakage or component detachment has occurred, and generate a set of high-risk parts of the instrument structure. S4: Identify and extract the start and end image frames of the instrument entering and exiting the occluded area, and establish a polar coordinate system based on the principal axis center; S5: In the polar coordinate system, the start and end image frames of the occlusion area are sorted by angle to generate a spiral edge sequence and a sliding matching comparison is performed to target and focus on high-risk areas. S6: Acquire all instruments with structural loss anomalies, and output the residual area after component loss in the surgical field space in combination with the occlusion area.

2. The method for identifying abnormal states of surgical instruments based on image recognition according to claim 1, characterized in that, In S1, image stream data of the surgical field space is acquired during the operation, and surgical instruments are identified. The temporal position of each instrument in each frame of the image is extracted, specifically including: Acquire image stream data covering the surgical field space during surgery, and convert the image stream data into an image frame sequence; Based on a deep semantic feature encoding network, we identify medical device instances and classify medical devices in image frames, and remove image frames with blurred edges of medical device instances from the image frame sequence. Two-dimensional image spatial envelopes of different instruments are extracted from the image frame sequence. The two-dimensional position coordinates of the instruments in the surgical field coordinate system are calculated. The image plane motion trajectory of the instruments is constructed based on the two-dimensional position coordinates of the instruments.

3. The method for identifying abnormal states of surgical instruments based on image recognition according to claim 2, characterized in that, Specifically, the calculation of the two-dimensional position coordinates of the instrument in the surgical field coordinate system involves performing image registration on consecutive frames in the image frame sequence, extracting static texture points in adjacent image frames, constructing a surgical field reference plane based on the distribution of static texture points in the non-instrument background area of ​​the surgical field space, setting the origin, horizontal axis and vertical axis to establish the surgical field coordinate system, mapping the spatial envelope of the instrument image appearing in all image frames to the surgical field reference plane, and marking the two-dimensional planar position of the instrument in the surgical field coordinate system.

4. The method for identifying abnormal states of surgical instruments based on image recognition according to claim 1, characterized in that, In S2, based on the image occlusion pattern within the surgical field space, the division of the surgical field occlusion region specifically includes: Based on the determination of brightness gradient direction breakage and texture continuity interruption, the spatial overlap between the instrument image spatial envelope and the non-instrument background area in the surgical field space is identified in the image frame, and the occlusion candidate area in the image is delineated. Connectivity is determined for occlusion candidate regions in the image frame sequence, and segments with continuously increasing overlapping areas and closed edges are selected as effective occlusion segments; All effective occluded segments are reconstructed into spatial occluded blocks in the surgical field coordinate system, and a set of spatial occluded regions in the surgical field is generated based on the contours of the spatial occluded blocks.

5. The method for identifying abnormal states of surgical instruments based on image recognition according to claim 1, characterized in that, In S3, archived medical surgical accident data is retrieved, and the locations where instrument breakage or component detachment occurred are identified from the accident data. A high-risk component tag set for instrument structures is generated, specifically including: Retrieve archived medical malpractice reports containing data on surgical instrument breakage and foreign body residue incidents; The structure of the instruments in the accident data collection of instruments broken and foreign objects left behind is deconstructed to mark the specific fracture points and the locations of the detached components. Based on the types and models of equipment involved in the accident, the specific locations on the structures of various types of equipment that repeatedly break or detach are statistically analyzed to generate a high-risk structural node index set. The high-risk structural node index set is back-labeled to the instrument structure to form a high-risk structural part label set.

6. The method for identifying abnormal states of surgical instruments based on image recognition according to claim 1, characterized in that, In S4, identifying and extracting the start and end image frames of the instrument entering and exiting the occluded area, and establishing a polar coordinate system based on the principal axis center specifically includes: The first overlapping frame and the last de-overlapping frame between the spatial envelope of each instrument image and the set of occluded regions in the surgical field space are identified in the image frame sequence and labeled as the occlusion entry frame and occlusion exit frame, respectively. In the occlusion entry frame and occlusion exit frame, the device image must be determined to have the same device posture. If they are not consistent, continuous matching and retrieval will be performed before the first overlapping frame and after the last unoverlapping frame. Extract the two-dimensional image spatial envelope of the instrument in the occlusion entry frame and occlusion exit frame, and combine it with the position trajectory in the surgical field coordinate system to perform instrument image principal axis direction fitting and structural center point localization; Using the direction of the instrument's main axis and the center point of the structure, a two-dimensional polar coordinate system is defined with the center of the structure as the pole and the direction of the main axis as the polar axis.

7. The method for identifying abnormal states of surgical instruments based on image recognition according to claim 1, characterized in that, In S5, in the polar coordinate system, the start and end image frames of the occlusion area are sorted by angle to generate a spiral edge sequence and a sliding matching comparison is performed. Targeted focusing identification of high-risk areas specifically includes: In the constructed polar coordinate system, the spatial envelope of the instrument image in the occlusion entry frame and the occlusion exit frame is expanded along the angular direction to generate a spiral edge distribution sequence. Arrange the spiral edge distribution sequence in ascending order of polar angle to form two sets of spiral edge vectors with the same direction. Perform sliding window matching on the two sets of spiral edge vectors, calculate the edge offset index under the same angular direction, and output the structural difference map; Extract the polar angle segments corresponding to the label set of high-risk structural parts from the structural difference map, and perform structural difference analysis to determine whether there are defective segments with texture breaks or shortened contours. If so, mark them as abnormal instrument structure loss.

8. The method for identifying abnormal states of surgical instruments based on image recognition according to claim 1, characterized in that, In S6, all instruments with structural loss anomalies are acquired, and the residual area after component loss is output within the surgical field space, combined with the occlusion area. Specifically, this includes: The defective segment corresponding to the structurally missing instrument is reverse-mapped to the two-dimensional image space envelope of the instrument in the image frame to generate the image space missing region. Based on the overlapping location of the obscured area and the missing area in the surgical field, the possible breakage and detachment range of the component can be estimated. Perform connectivity and closure tests on all fractured and detached areas, and output a set of structurally closed component retention areas.

Images