A surgical instrument abnormal state recognition method based on image recognition

Abstract

The application discloses a kind of based on image recognition surgical instrument abnormal state identification method, specifically related to image recognition field, for solving the existing surgical instrument structure abnormal retention problem;It is through the image stream data collector covering operation field space to execute instrument identification;According to image occlusion form, divide the occlusion area in operation field, combine medical operation accident data to count the high-risk part of instrument structure;Identify the start and stop image frame of instrument entering and exiting occlusion area, based on its main shaft direction and center to construct polar coordinate system, and under the coordinate system, angle sorting and sliding comparison are carried out to instrument edge, focus on identifying the structural defect of high-risk part;The structure loss anomaly identified is overlapped with the occlusion area of operation field, and the closed area set that component can be retained is output.The scheme realizes the abnormal focusing of intraoperative instrument fracture event and the positioning of component retention risk space, improves the precision and efficiency of intraoperative risk control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and more particularly, to a surgical instrument abnormal state recognition method based on image recognition. BACKGROUND

[0002] In minimally invasive surgery and endoscopic surgery, the problem of abnormal instrument retention becomes a serious safety hazard in clinical practice. The essence of the problem is not due to instrument identification failure or forgetting to count, but due to the fact that the instrument is broken and the parts are dropped after entering the occlusion area in the operation field, or part of the temporary placement type instrument is not removed physically, but the risk state is not identified, so that part of the instrument structure is still left in the body after the operation. Such retention often does not have significant visual features, neither severe tissue reaction nor being marked as abnormal by the operation process. At the same time, due to the factors such as the use of multiple instruments alternately, frequent instrument switching, and the misalignment of the assistant and the surgeon's collaborative path, it is easy to form an instrument main body that has been removed but the parts are retained. More complexly, such instrument retention is often not caused by the subjective operation of the operator, but is caused by the passive falling into the tissue gap, sliding into the deep cavity area, or being out of control due to occlusion and failing to identify the unrecycled position, which has a high degree of danger.

[0003] At present, there is still a lack of intraoperative identification mechanism for such dangerous retention situations, especially when using instruments with complex structure and detachable parts. Such hidden and image-unvisible state of instrument abnormalities is more likely to be ignored. The existing methods mostly check the integrity of the instrument after the operation through manual counting or auxiliary scanning, but such post-event remedial methods are inefficient and cannot timely discover the danger during the operation, requiring a second operation to remove. SUMMARY

[0004] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present application provide a surgical instrument abnormal state recognition method based on image recognition to solve the problems raised in the background art.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0006] A surgical instrument abnormal state recognition method based on image recognition, comprising the following steps:

[0007] S1: Collecting image stream data of the operation field space during the operation and performing surgical instrument recognition to extract the time sequence position of each instrument in each image;

[0008] S2: Dividing the occlusion area of the operation field space based on the occlusion shape of the image within the operation field space range;

[0009] S3: Obtaining the archived medical operation accident data, counting the positions where instrument breakage and part drop occur in the accident data, and generating a high-risk part label set of the instrument structure;

[0010] S4: Identify and extract the start and end image frames of the instrument in and out of the occlusion area, and establish a polar coordinate system based on the center of the main shaft;

[0011] S5: In the polar coordinate system, sort the start and end image frames of the instrument in and out of the occlusion area by angle to generate a convolution edge sequence and perform sliding matching comparison, and target focus identification on high-risk parts;

[0012] S6: Obtain all structural loss anomalies of the instrument, and combine the occlusion area to output the retention area after the loss of the component in the surgical field space.

[0013] In a preferred embodiment, in S1, the image stream data of the intraoperative surgical field space is collected and the surgical instrument recognition is performed, and the time sequence position of each instrument in each image frame is extracted, which specifically includes:

[0014] Collecting image stream data covering the surgical field space in the operation, and converting the image stream data into an image frame sequence;

[0015] Performing instrument instance recognition and instrument category labeling in the image frame based on a deep semantic feature coding network, and removing image frames with blurred instrument instance edges in the image frame sequence;

[0016] Extracting the two-dimensional image space envelope of different instruments in the image frame sequence, calculating the two-dimensional position coordinates of the instruments in the surgical field coordinate system, and constructing the image plane motion trajectory of the instruments according to the two-dimensional position coordinates of the instruments.

[0017] In a preferred embodiment, the calculation of the two-dimensional position coordinates of the instruments in the surgical field coordinate system specifically includes performing image registration on consecutive frames in the image frame sequence, extracting static texture points in adjacent image frames, constructing a surgical field reference plane based on the distribution of static texture points in the non-instrument background area in the surgical field space, and setting an origin, a horizontal axis and a vertical axis to establish a surgical field coordinate system., map the instrument image space envelope appearing in all image frames to the surgical field reference plane, and label the two-dimensional plane position of the instrument in the surgical field coordinate system.

[0018] In a preferred embodiment, in S2, based on the image occlusion pattern within the surgical field space, the surgical field space is divided into occlusion areas, which specifically includes:

[0019] According to the judgment of brightness gradient direction fracture and texture coherence interruption, the spatial overlap between the instrument image space envelope and the non-instrument background area in the surgical field space in the image frame is identified, and the occlusion candidate area in the image is determined;

[0020] The connectivity of the occlusion candidate area in the image frame sequence is judged, and the segment with continuous growth of overlapping area and closed edge is selected as the effective occlusion segment;

[0021] Reconstruct all effective occlusion segments into spatial occlusion blocks in the surgical field coordinate system, and generate a set of surgical field spatial occlusion regions based on the outlines of the spatial occlusion blocks.

[0022] In a preferred embodiment, in S3, the archived medical operation accident data is obtained, the positions where instrument fracture and component shedding occur in the accident data are counted, and the generated high-risk position label set of the instrument structure specifically includes:

[0023] Retrieve the instrument fracture and foreign body retention accident data set in the archived medical accident report;

[0024] Disassemble the instrument structure in the instrument fracture and foreign body retention accident data set, and mark the specific fracture point position and the shedding component position;

[0025] Combine the instrument types and models in the accident, count the specific positions where repeated fracture or shedding occurs on the structures of various instruments, and generate a high-risk structure node index set;

[0026] Map the high-risk structure node index set back to the instrument structure to form a high-risk structure position label set.

[0027] In a preferred embodiment, in S4, the start and end image frames of the instrument entering and exiting the occlusion region are identified and extracted, and a polar coordinate system is established based on the center of the main shaft, specifically including:

[0028] Identify the first overlapping frame and the last non-overlapping frame between the spatial envelope of each instrument image in the image frame sequence and the set of surgical field spatial occlusion regions, and label them as occlusion entry frame and occlusion exit frame, respectively;

[0029] The instrument images in the occlusion entry frame and the occlusion exit frame need to determine consistent instrument poses, and if they are inconsistent, continuous matching retrieval is performed before the first overlapping frame and after the last non-overlapping frame;

[0030] Extract the two-dimensional image spatial envelope of the instrument in the occlusion entry frame and the occlusion exit frame, and combine the position trajectory under the surgical field coordinate system to perform main shaft direction fitting and structure center point positioning of the instrument image;

[0031] Use the instrument main shaft direction and structure center point to define a two-dimensional polar coordinate system with the structure center as the pole and the main shaft direction as the polar axis.

[0032] In a preferred embodiment, in S5, in the polar coordinate system, the start and end image frames of the instrument entering and exiting the occlusion region are sorted by angle to generate a convolution edge sequence and perform sliding matching comparison, and the high-risk positions are targeted and focused on for identification, specifically including:

[0033] In the constructed polar coordinate system, the instrument image spatial envelope in the occlusion entry frame and the occlusion exit frame is expanded along the angle direction to generate a convolution edge distribution sequence;

[0034] arranging the sequence of the edge of rotation in order of the polar angle from small to large, to form two sets of edge of rotation vector sets with consistent direction;

[0035] performing sliding window matching on the two sets of edge of rotation vector sets, calculating the edge offset index under the same angle direction, and outputting a structure difference atlas;

[0036] extracting the polar angle section corresponding to the high-risk structure part label set in the structure difference atlas, and performing structure difference analysis to determine whether there is a defect section of texture fracture or contour shortening, and if so, marking as an abnormality of instrument structure loss.

[0037] In a preferred embodiment, in S6, all instruments with structure loss abnormalities are obtained, and the specific output of the retention area of the component after the loss in the surgical field space in combination with the occlusion area includes:

[0038] mapping the defect section corresponding to the instrument with structure loss abnormalities to the two-dimensional image space envelope of the instrument in the image frame in reverse, to generate an image space missing area;

[0039] estimating the possible fracture, shedding and retention range of the component according to the overlapping position of the occlusion area and the missing area in the surgical field space;

[0040] performing connectivity and closedness detection on all fracture, shedding and retention ranges, and outputting a set of component retention areas with structure closure.

[0041] The technical effects and advantages of the surgical instrument abnormal state recognition method based on image recognition are as follows:

[0042] By constructing an instrument image before and after state comparison mechanism based on the occlusion interval, the real-time recognition of the fracture or component shedding abnormality that may occur after the instrument enters the occlusion area in the surgical field is realized, and the recognition ability of the intraoperative image analysis system for the instrument risk under the occlusion state is significantly improved. Through the alignment of the image frame position under the surgical field coordinate system and the construction of the main shaft center polar coordinate system, the spatial consistency and structure focusing of the comparison process are ensured, and the target recognition strategy of the high-risk structure part label set is combined, which can effectively reduce the misjudgment rate of non-critical areas and enhance the accuracy and engineering practicability of the recognition. Especially for the situation that complete occlusion leads to continuous observation of the instrument complete state, the present scheme realizes sensitive detection of the instrument structure loss by analyzing the edge evolution path of the image in and out of the occlusion area, avoids the missing risk caused by the failure of the traditional method due to occlusion, and realizes early detection and position locking of the intraoperative instrument fracture and component shedding, provides accurate basis for postoperative foreign body removal, significantly reduces the risk of leaving and the probability of postoperative complications, and has important safety value and clinical application significance. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 Fig. 1 is a schematic diagram of a surgical instrument abnormal state recognition method based on image recognition according to the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0045] Embodiment 1

[0046] Figure 1 The present application provides a surgical instrument abnormal state recognition method based on image recognition, comprising the following steps:

[0047] S1: collecting image stream data of an intraoperative surgical field space and performing surgical instrument recognition to extract the time sequence position of each instrument in each image frame;

[0048] S2: dividing a surgical field space occlusion area based on the image occlusion pattern in the surgical field space range;

[0049] S3: obtaining archived medical surgical accident data, counting the positions of instrument fracture and component shedding in the accident data, and generating a high-risk position label set of the instrument structure;

[0050] S4: recognizing and extracting the start and end image frames of the instrument entering and exiting the occlusion area, and establishing a polar coordinate system based on the main shaft center;

[0051] S5: in the polar coordinate system, the start and end image frames of the instrument entering and exiting the occlusion area are sorted by angle to generate a convolution edge sequence and perform sliding matching comparison, and the high-risk position is targeted and focused for recognition;

[0052] S6: obtaining all the instruments with structural loss anomaly, and combining the occlusion area to output the retention area of the component loss in the surgical field space.

[0053] In S1, the image stream data of the intraoperative surgical field space is collected and surgical instrument recognition is performed to extract the time sequence position of each instrument in each image frame.

[0054] During the surgery, a multi-view image recording device is deployed to collect image stream data covering the entire surgical field space. The collected data needs to meet a temporal sampling frequency of at least 25 frames per second and be saved as a continuous image frame sequence in a unified format. To ensure subsequent recognition accuracy, the image frames need to undergo preliminary preprocessing, including resolution unification (e.g., unified to 1920x1080 pixels) and illumination histogram adjustment to ensure that the image contrast and tone distribution are within a controllable range, avoiding edge recognition errors caused by image brightness fluctuations or shadows.

[0055] An instrument recognition task is performed on the image frame sequence. A structured semantic feature encoding network model is used for semantic segmentation and instance recognition of each image frame. The structured semantic feature encoding network model is based on the multi-task branch architecture of Mask R-CNN structure, with ResNet-101 and FPN combined as the backbone to extract spatial semantic features at different scales. The training data set is derived from multiple surgical scene image annotation sets, covering the categories and structural partition information of commonly used surgical instruments. During the recognition process, all surgical instruments appearing in the image are labeled with their class labels (such as forceps, scissors, retractors, etc.), and the corresponding image space envelope area in the two-dimensional image is extracted, i.e., the boundary contour of the instrument in the frame image. The recognition result is output in the form of instances, with the pixel position index of each instance in the image frame. Image preprocessing is performed on the image frame sequence. Specifically, it includes unifying the spatial size of the image frame (such as scaling to a unified resolution), standardizing the image grayscale range, performing histogram equalization to unify the illumination contrast, and performing preliminary screening of image frame sharpness based on edge sharpness gradient. Image sharpness is calculated using the average gradient amplitude of the Sobel operator in the image edge region. If the average edge gradient is below a certain threshold (e.g., frames with an average edge gradient below 15), the frame is considered to be blurred and is removed to avoid incomplete feature extraction in subsequent recognition.

[0056] A clear spatial positioning representation of the recognized instruments is established in each image frame. Specifically, a two-dimensional image space envelope is extracted for each instrument target in the image frame, clearly defining the boundary range of each instrument in the frame image. The image space envelope is defined as the minimum closed boundary structure of the visible area of the instrument in the current image frame, often represented by a polygon or elliptical edge fitting result. The envelope extraction process uses the target mask in the instrument instance recognition result in the image to perform edge extraction on the mask area, excluding profile burrs and low-contrast background interference. After completing the instrument envelope extraction, the envelope position in the image space is mapped to a unified reference system in the surgical field space to establish a two-dimensional positioning model in the surgical field coordinate system. To achieve this mapping, a surgical field coordinate system is first constructed. The surgical field coordinate system should have two characteristics: first, the spatial reference object should be the background area in the image that does not change significantly over time; second, the definition of the coordinate axes should cover the entire surgical field plane and have spatial consistency. To this end, first perform image registration on consecutive images in the image frame sequence. The purpose of image registration is to correct the spatial offset caused by factors such as shooting angle, parallax, vibration, etc. in multiple images, thereby achieving position alignment of the same position in multiple images. In image registration, a feature point detection algorithm is used to extract high-stability texture points in the image frame, especially in the background area (i.e., non-instrument area) to select widely distributed, clear-edged corner points, stripes, and texture boundaries as static texture points. To ensure that these texture points have static properties, their spatial positions need to be verified for stability in multiple image frames at different time points. The stability judgment criterion is that a certain texture point has a center position offset of no more than 2 pixels in the image coordinate system in adjacent 5 frames. Texture points that meet this condition form a static point set, which serves as the spatial reference points of the surgical field background area. Based on the above static texture point set, analyze the overall distribution density and direction consistency of the texture points in the image frame, select the image frame with the most uniform texture point distribution and the largest coverage range as the surgical field reference frame, and construct the surgical field reference plane based on the texture point set in this frame. After the construction is completed, the centroid of the texture points in the reference plane is taken as the origin of the surgical field coordinate system, the direction from the centroid to the horizontal boundary of the image is defined as the horizontal axis (x-axis), and the vertical direction is defined as the y-axis, forming a complete two-dimensional surgical field coordinate system.

[0057] After the coordinate system of the surgical field is established, the center points of the instrument image space envelopes in all image frames are mapped to the coordinate system. The image coordinates in the image frames are uniformly transformed into planar coordinates under the surgical field coordinate system. The two-dimensional image positions of any instrument in the image frames are uniformly described as a planar point under the surgical field coordinate system, which has an absolute position reference significance. With the time advancement of the image frame sequence, the above operation is continuously performed to obtain the two-dimensional position point set of each identified instrument in the surgical field coordinate system in the continuous image frames, and the points are organized in time sequence to construct the image plane motion trajectory of the instrument. Each node in the trajectory is composed of the following fields: image frame number, position (x, y) under the surgical field coordinate system, and instrument space envelope boundary in the corresponding image frame. The planar trajectory will provide a unified space-time reference for the subsequent identification of occlusion area entry and exit points, polar coordinate establishment, and convolution edge matching comparison.

[0058] In S2, based on the image occlusion pattern in the surgical field space range, the surgical field space occlusion area is divided.

[0059] Since the instrument frequently penetrates between the tissue structures during the operation process, it is easy to occur that part of the instrument enters or exits the field of view, is occluded by the tissue, etc. These occlusion phenomena may mask the abnormal state of the instrument structure. Therefore, it is necessary to identify the position and time period of the occlusion in the image, and establish a region set corresponding to the occlusion behavior in the surgical field space. With the image frame under the surgical field coordinate system as input, based on the calibrated two-dimensional image space envelope region of the instrument and the non-instrument background region in the surgical field space, the spatial overlap region between the two is calculated. Specifically, the boundary contrast method is used to detect the image transition features between the instrument image space envelope edge and the background region, and it is determined whether there is occlusion evidence.

[0060] The judgment of the occlusion sign here is based on two main indicators: one is the break of the brightness gradient direction, and the other is the interruption of the texture continuity. The recognition of the break of the brightness gradient direction is based on the image gray gradient field, and the gray gradient direction is calculated by performing Sobel calculation on the local image at the envelope edge. If there is an obvious angle change (more than 30°) or gradient direction discontinuity between the brightness gradient direction at the edge of a certain image segment and the inside of the envelope, it is determined that the gradient is broken. Usually, this broken phenomenon indicates that the instrument and the tissue have been interfered by occlusion at the boundary, rather than a complete edge. The interruption judgment of the texture continuity uses the consistency of the image texture direction. The image texture vectors in the range of 10 pixels inside and outside the envelope edge are extracted, and the direction distribution density and the direction distribution dispersion index are calculated. If there is an obvious break in the texture direction inside and outside the envelope (for example, the standard deviation of the direction increases by more than 20%), it is marked as a texture interruption area. This processing logic ensures that the occlusion judgment not only depends on the brightness change, but also combines the physical logic of the continuity of the tissue texture. Based on the above two types of broken areas, the image morphological processing method (such as dilation, erosion and boundary tracking) is used to delineate the complete closed occlusion candidate area. All areas that meet the boundary closure greater than 85% and the occlusion area greater than 2% of the total image area are retained as preliminary candidates.

[0061] After the candidate area is identified, its connectivity in the image frame sequence is judged. The specific method is: the occlusion candidate area in each image frame is numbered, and the spatial connectivity analysis is performed through the trajectory continuity of the center position of the occlusion area in the surgical field coordinate system. If a certain candidate area exists in the adjacent frames with constant or increasing area trend, and its boundary closure continues to rise (for example, the boundary integrity continuously exceeds 90%), it is marked as an effective occlusion segment. After the selection of the effective occlusion segment is completed, the spatial reconstruction is performed in the surgical field coordinate system. Based on the two-dimensional boundary points of the effective occlusion area in each image in the surgical field coordinate system, the occlusion boundaries in the continuous frames are superimposed in time sequence, the spatial occlusion is formed, and the contour of the occlusion area in the surgical field plane is reconstructed, forming the occlusion area representation. The spatial occlusion is realized by time envelope merging, that is, the point set fusion of the occlusion boundary points in adjacent frames is performed to generate a unified occlusion boundary in the time window. Finally, the occlusion area representation is formed, which includes the starting frame, the ending frame, the occlusion duration, the occlusion boundary coordinate set in the surgical field coordinate system, the maximum occlusion area, the center position and other attributes.

[0062] In S3, the archived medical surgery accident data is obtained, the parts where the instrument fracture and component shedding occur in the accident data are counted, and a high-risk part label set of the instrument structure is generated.

[0063] Centralize and organize existing medical malpractice archival data. This includes abnormal intraoperative events such as intraoperative instrument breakage, component loosening, and foreign body retention recorded in the intraoperative surgical instrument abnormal event records disclosed by medical institutions at all levels, such as "grasping forceps tip breakage", "lower jaw loss of tissue scissors", "insulation layer damage of electrocoagulation hook", etc. At the same time, set the data collection time range (such as within the past five years) as the screening condition. Perform structural deconstruction analysis on the screened instrument breakage and component shedding accident data. This operation is carried out around the physical structure of surgical instruments. The accident records all contain subjective descriptions of the damaged parts of the instruments. Combined with the public structure diagram of the instrument, the specific location of the abnormality is mapped to the structure composition level, such as in the double-bend grasping forceps, the common breakage points are the distal end of the forceps head connection or the lower jaw fixed rivet point; in the laparoscopic tissue scissors, the common breakage points are the shear blade breakage or the shear blade root activity hinge breakage. In each event, the specific model of the instrument, the structure name of the damaged part, and the physical position description (such as "the outer edge of the forceps head connection arm") are marked and recorded as structure position point records.

[0064] All the analyzed structure abnormal points are summarized and classified by instrument category and model to form a list of high-risk structure parts. In this statistical process, the frequency of instrument part occurrence is used as the basis to screen out high-repetition and high-overlap structure nodes. For example, in the statistics of grasping forceps instruments, it is found that the lower jaw inside connection point appears 16 times in 20 accidents, which can be determined as a typical high-risk structure part. The statistical threshold can be set to more than 30% of the total number of accidents in the same model, or the structure position repetition rate is more than 40% across models as the high-risk determination condition. Based on the statistical results, an index set of high-risk structure nodes is constructed. The index set takes the instrument category as the index primary key, and labels several structure parts under each instrument category. Each structure part contains the following fields: structure name (such as "jaw activity hinge"), node serial number (named by the sequence identification of the structure diagram component), frequency count, involved model list, and accident example index. The structure node index set is structure aligned back to the standardized instrument structure diagram model to form a high-risk structure part label set. The label set is encoded in the form of structure diagram node component description, and each high-risk structure part is labeled in the corresponding instrument structure diagram with a prominent identifier.

[0065] In S4, identify and extract the start and end image frames of the instrument entering and exiting the occlusion area, and establish a polar coordinate system based on the main shaft center.

[0066] For the identified instrument image spatial envelope and the set of surgical field spatial occlusion regions in the sequence of surgical image frames, a frame-by-frame spatial overlap analysis operation is performed. First, according to the image frame number order, the spatial overlap between the instrument image spatial envelope and the occlusion region boundary in each frame is calculated, and according to the pixel area ratio of the overlap area in the surgical field coordinate system, the first image frame that exceeds the set overlap rate threshold is extracted, which is marked as the occlusion entry frame of the instrument. The occlusion rate threshold is set to 20% of the area of the instrument image spatial envelope, that is, when the part of the instrument envelope region overlapping with the occlusion region first exceeds 20% of the total envelope area, it is judged to be in the occlusion state; otherwise, in the continuous frame sequence, the proportion first drops below the threshold, and the image frame is marked as the occlusion exit frame. The above operation can be independently performed for each instrument to ensure accurate distinction of occlusion entry and exit nodes in the case of concurrent occlusion of multiple instruments. After the occlusion entry frame and the occlusion exit frame are marked, it is necessary to further confirm whether the instrument poses in the two frames are consistent. To complete this judgment, the direction angle of the main axis of the instrument image spatial envelope and the contour barycenter position are extracted, and the direction vector angle and the centroid offset amplitude are calculated. Among them, the main axis direction is the linear fitting direction of the main extension direction of the instrument image spatial envelope, and the contour barycenter position (image centroid) is the geometric center point of the contour region of the instrument image in the image space, which is the position center of the instrument on the image, without considering mass or color, only according to the shape of the region.

[0067] When the main axis direction angle is less than 5 degrees and the centroid position offset is less than a set pixel difference (such as 15 pixels) in the image coordinate space, it is considered that the poses of the two frames are consistent; if any index exceeds the set range, frame-out matching is started. Specifically, at most 10 frames before the occlusion entry frame are traced back, and at most 10 frames after the occlusion exit frame are tracked, and the instrument image pose in each frame is compared and matched with the target frame pose, and the image frame with the closest pose and the image clarity not lower than the average of the previous and next frames is selected as the replacement frame. The replacement frame is used as the basis image for subsequent structure main axis extraction and polar coordinate system construction, ensuring analysis consistency.

[0068] After the occlusion entering and exiting frames with consistent poses are obtained, the fitting of the main axis direction and the positioning of the structure center point are performed. The fitting operation is based on the contour boundary point set of the instrument image envelope region in the image space, and the least square straight line fitting method is used to extract the main axis direction, and the output is a directional vector in the image space. The geometric center of the image envelope region is taken as the initial estimate of the structure center point, and combined with the low-speed displacement segment in the continuous frame trajectory of the instrument in the surgical field coordinate system, the initial gravity center is corrected for stability, and the two-dimensional coordinates of the instrument structure center point in the surgical field coordinate system are finally determined. After the main axis direction and the structure center point are positioned, the polar coordinate system is constructed. The specific method is: taking the structure center point as the pole point, the main axis direction vector as the polar axis direction, setting 0° angle starting line along the positive direction of the polar axis, and dividing the 0 to 360 degree full circle angle range with equal interval angle (default 5°).

[0069] In S5, in the polar coordinate system, the start and end image frames entering and exiting the occlusion region are sorted by angle to generate a convolution edge sequence and perform sliding matching comparison to focus on identifying high-risk parts.

[0070] In the constructed polar coordinate system, the instrument image space envelope labeled in the occlusion entering frame and the occlusion exiting frame is performed angle expansion operation. Specifically, for the contour edge of the instrument image space envelope in each frame of image, relying on the structure center point of the established polar coordinate system as the pole point and the main axis direction as the polar axis, the contour edge points are subjected to polar angle projection transformation. The position of each discrete pixel point on the contour edge in the image coordinate system is converted into the polar coordinate form with the pole point as the origin, forming the polar radius and polar angle pair of the edge point. By setting the polar angle resolution interval, all edge points are grouped in order of polar angle from small to large, and the maximum value of the polar radius in each angle group is counted as the edge distribution index in the polar angle direction. The above process is applied to the occlusion entering frame and the occlusion exiting frame respectively, forming two groups of convolution edge distribution sequences with consistent direction. In order to avoid the influence of angle error on matching accuracy, high resolution angle granularity should be used when dividing the polar angle, and the starting point of the polar angle should be uniformly set to ensure the correspondence consistency between the sequences. Then, the sliding window matching operation is performed on the convolution edge vector set formed by the above two groups of convolution edge distribution sequences. In this operation, a fixed length polar angle window (such as 10 degrees) is selected to locally align between the two groups of convolution edge vector sets, and the edge distance offset of the corresponding polar angle segment in the window is calculated. The edge distance offset is defined as the absolute value sum of the difference between the polar radius values corresponding to the same polar angle, which reflects the edge contour change degree of the two frames of images in the direction. All polar angle segments are analyzed by sliding during the matching process, and the complete edge difference distribution map, i.e. the structure difference atlas, is obtained.

[0071] After the construction of the structure difference map, the polar angle section corresponding to the high-risk structure site label set in the map is extracted for key analysis. The high-risk structure site label set is composed of the structure node index set of the high-risk fracture site in the instrument structure model in the previous step, which needs to be converted to a polar angle interval here. The conversion method is: for each high-risk structure node, perform part correspondence, and perform polar coordinate conversion at the two-dimensional image envelope position in the instrument space. Calculate the corresponding polar angle position of the edge of the structure node in the occlusion entering frame and the exiting frame, and set the structure tolerance range (default ±5 degrees) to expand to form a high-risk polar angle section set. According to the set in the structure difference map, the corresponding section is located, and the edge offset trend, gradient mutation phenomenon and local polar radius shortening phenomenon in the section are judged.

[0072] In the extracted structure difference map, if there are continuous multiple polar angle points (such as more than 5 angle units) in a high-risk polar angle section with continuous rising of the edge offset value, and the rising amplitude exceeds the set threshold (such as the polar radius offset amount of the front and rear frames is greater than 20% of the original polar radius value), it is considered that the edge structure in this direction has abnormal deformation. At the same time, in the same polar angle section, if the polar radius value is observed to suddenly drop, that is, the farthest point of the edge in this direction suddenly shrinks, and the average value of the polar radius is more than 30% lower than the historical polar radius value of the corresponding polar angle section in the normal working state of the instrument (which can be constructed by the unoccluded frame), it indicates that the edge region has length compression, which is usually closely related to the instrument structure fracture or component shedding. For the instrument edge region corresponding to the occlusion exit frame in the polar angle section, perform image texture continuity analysis. The analysis extracts the gray gradient direction graph and gradient intensity distribution of the instrument contour region, and performs linear projection statistics on the edge texture in the corresponding polar angle direction. If there is a significant mutation in the gradient direction or a texture line segment fracture phenomenon, and a texture void or texture blur band (the area gradient intensity is lower than the set threshold, for example, less than 10 average gray gradient) is formed in the edge region, it is considered that the texture continuity in the region is interrupted. When the above three conditions, i.e. significant edge offset, polar radius value sudden drop, and texture continuity interruption, are met at the same time, it is determined that the instrument structure region corresponding to the polar angle section has obvious abnormalities and is no longer in the complete configuration state. The abnormality can be caused by factors such as structure fracture, end component shedding or surface peeling. On this basis, the polar angle section is recorded as the structure defect occurrence position, and it is matched and confirmed with the instrument high-risk structure site label set. If it belongs to the polar angle section in the high-risk structure site, it can be directly marked as an instrument structure loss anomaly.

[0073] In S6, all instruments with structure loss anomalies are obtained, and the retention area of the occluded region in the surgical field space after the loss of the part is output.

[0074] After the instrument structure loss anomaly is determined, the edge profile information of the segment in the polar coordinate system should be inversely projected into the two-dimensional image space envelope corresponding to the image frame according to the polar angle positioning result of the missing segment. In specific implementation, first, the position of the instrument structure center point used to construct the polar coordinate system in the image coordinate system is extracted, and the corresponding principal axis direction vector is retained; then, the boundary vector covered by the polar angle segment is rotated and mapped back to the image space according to the principal axis direction, and the profile path of the missing segment in the image frame is reconstructed in the image space through the polar radius length and the polar angle direction. The path should intersect with the instrument image space envelope profile and form a closed or approximately closed region inside the profile. This region is defined as the image space missing region and is marked in a binary mask manner on the image frame, which is used for occlusion relationship and retention estimation.

[0075] After the image space missing region is marked, the image space missing region and the surgical field space occlusion region are analyzed in terms of spatial relationship to determine the position where the missing part may stay after being possibly retained or falling off. The specific operation is as follows: first, the missing region mask constructed is aligned with the surgical field occlusion region set in the image frame space. Since the surgical field occlusion region has been converted into a two-dimensional polygonal region with clear boundaries and coverage in the surgical field coordinate system in the foregoing processing flow, the image frame where the missing region is located is mapped to the surgical field coordinate system through image registration, and whether the two regions have spatial overlap is judged in a coordinate overlapping manner. When the surgical field coordinates corresponding to any pixel point of the image space missing region are located within the boundary of the occlusion region, it is considered that the point is the possible falling point of the broken or fallen part. Further, all the missing region point sets having overlapping relationship are aggregated to form the possible staying range of the part. If there are multiple missing segments spanning multiple occlusion regions, multiple staying range sets should be generated respectively.

[0076] After all the possible part staying ranges are estimated, it is confirmed whether these regions have structural closure and connectivity, to ensure that they have the closed physical conditions for actual retention of the instrument assembly. The connectivity of the binary graph is detected for each staying range region, the connected regions are identified using the standard 4-neighborhood or 8-neighborhood algorithm, and isolated points or discrete regions with a size smaller than a set threshold (for example, less than 100 pixels in the surgical field image) are excluded. Subsequently, boundary extraction and contour tracking are performed on each connected region to determine whether it constitutes a closed structure. If the region boundary can be closed into a closed polygon and has no edge breakage, it is considered that the region has structural closure and can form a potential staying region of the part. If the region boundary has obvious gaps (for example, the closure degree is less than 90%), the region is excluded. Finally, all the regions satisfying the connectivity and closure are included in the part staying region set, and are archived according to the instrument number and the image frame time stamp, as the signal prompt basis for the retention of the broken foreign matter of the surgical instrument during surgery and the basis for the postoperative tracing.

[0077] The above formulas are all de-dimensioned to calculate their numerical values. The formulas are obtained by collecting a large amount of data to simulate a formula of the most recent real situation. The preset parameters and threshold values in the formulas are set by a person skilled in the art according to actual conditions.

[0078] The above embodiments can be implemented wholly or partially by software, hardware, firmware, or any other combination. When implemented by software, the above embodiments can be implemented wholly or partially in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server, or data center to another by wired (for example, infrared, wireless, microwave, etc.) or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0079] Those of ordinary skill in the art can realize that the modules and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0080] Those of ordinary skill in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device, and module can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0081] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the division of the above-described device embodiment is merely a logical function division, and there can be another division manner for the actual implementation, for example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different modules can be indirect couplings or communication connections through some interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0082] The modules illustrated as separated components can or can not be physically separated, and the components illustrated as modules can or can not be physical modules, and can be located in one place, or can be distributed on multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment.

[0083] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can be physically present alone, or two or more modules can be integrated into one module.

[0084] If the functions are realized in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program codes that can be stored in the medium.

[0085] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, and all should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0086] Finally: the above only for the preferred embodiments of the present application, and not for limiting the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application, should be included in the scope of protection of the present application.

Claims

1. A method for identifying abnormal states of surgical instruments based on image recognition, characterized in that, Includes the following steps: S1: Acquire image stream data of the surgical field space during surgery and identify surgical instruments, extracting the temporal position of each instrument in each frame of the image; S2: Based on the image occlusion pattern within the surgical field space, the surgical field space occlusion area is divided; S3: Obtain archived medical surgical accident data, count the locations in the accident data where instrument breakage or component detachment has occurred, and generate a set of high-risk parts of the instrument structure. S4: Identify and extract the start and end image frames of the instrument entering and exiting the occluded area, and establish a polar coordinate system based on the principal axis center; S5: In the polar coordinate system, the start and end image frames of the occlusion area are sorted by angle to generate a spiral edge sequence and a sliding matching comparison is performed to target and focus on high-risk areas. S6: Obtain all instruments with structural loss anomalies, and output the residual area after component loss in the surgical field space in combination with the occlusion area; In S4, identifying and extracting the start and end image frames of the instrument entering and exiting the occluded area, and establishing a polar coordinate system based on the principal axis center specifically includes: The first overlapping frame and the last de-overlapping frame between the spatial envelope of each instrument image and the set of occluded regions in the surgical field space are identified in the image frame sequence and labeled as the occlusion entry frame and occlusion exit frame, respectively. In the occlusion entry frame and occlusion exit frame, the device image must be determined to have the same device posture. If they are not consistent, continuous matching and retrieval will be performed before the first overlapping frame and after the last unoverlapping frame. Extract the two-dimensional image spatial envelope of the instrument in the occlusion entry frame and occlusion exit frame, and combine it with the position trajectory in the surgical field coordinate system to perform instrument image principal axis direction fitting and structural center point localization; Using the direction of the instrument's main axis and the center point of the structure, a two-dimensional polar coordinate system is defined with the center of the structure as the pole and the direction of the main axis as the polar axis.

2. The method for identifying abnormal states of surgical instruments based on image recognition according to claim 1, characterized in that, In S1, image stream data of the surgical field space is acquired during the operation, and surgical instruments are identified. The temporal position of each instrument in each frame of the image is extracted, specifically including: Acquire image stream data covering the surgical field space during surgery, and convert the image stream data into an image frame sequence; Based on a deep semantic feature encoding network, we identify medical device instances and classify medical devices in image frames, and remove image frames with blurred edges of medical device instances from the image frame sequence. Two-dimensional image spatial envelopes of different instruments are extracted from the image frame sequence. The two-dimensional position coordinates of the instruments in the surgical field coordinate system are calculated. The image planar motion trajectory of the instruments is constructed based on the two-dimensional position coordinates of the instruments.

3. The method for identifying abnormal states of surgical instruments based on image recognition according to claim 2, characterized in that, Specifically, the calculation of the two-dimensional position coordinates of the instrument in the surgical field coordinate system involves performing image registration on consecutive frames in the image frame sequence, extracting static texture points in adjacent image frames, constructing a surgical field reference plane based on the distribution of static texture points in the non-instrument background area of ​​the surgical field space, setting the origin, horizontal axis and vertical axis to establish the surgical field coordinate system, mapping the spatial envelope of the instrument image appearing in all image frames to the surgical field reference plane, and marking the two-dimensional planar position of the instrument in the surgical field coordinate system.

4. The method for identifying abnormal states of surgical instruments based on image recognition according to claim 1, characterized in that, In S2, based on the image occlusion pattern within the surgical field space, the division of the surgical field occlusion region specifically includes: Based on the determination of brightness gradient direction breakage and texture continuity interruption, the spatial overlap between the instrument image spatial envelope and the non-instrument background area in the surgical field space is identified in the image frame, and the occlusion candidate area in the image is delineated. Connectivity is determined for occlusion candidate regions in the image frame sequence, and segments with continuously increasing overlapping areas and closed edges are selected as effective occlusion segments; All effective occluded segments are reconstructed into spatial occluded blocks in the surgical field coordinate system, and a set of spatial occluded regions in the surgical field is generated based on the contours of the spatial occluded blocks.

5. The method for identifying abnormal states of surgical instruments based on image recognition according to claim 1, characterized in that, In S3, archived medical surgical accident data is retrieved, and the locations where instrument breakage or component detachment occurred are identified from the accident data. A high-risk component tag set for instrument structures is generated, specifically including: Retrieve archived medical malpractice reports containing data on surgical instrument breakage and foreign body residue incidents; The structure of the instruments in the accident data collection of instruments broken and foreign objects left behind is deconstructed to mark the specific fracture points and the locations of the detached components. Based on the types and models of equipment involved in the accident, the specific locations on the structures of various types of equipment that repeatedly break or detach are statistically analyzed to generate a high-risk structural node index set. The high-risk structural node index set is back-labeled to the instrument structure to form a high-risk structural part label set.

6. The method for identifying abnormal states of surgical instruments based on image recognition according to claim 1, characterized in that, In S5, in the polar coordinate system, the start and end image frames of the occlusion area are sorted by angle to generate a spiral edge sequence and a sliding matching comparison is performed. Targeted focusing identification of high-risk areas specifically includes: In the constructed polar coordinate system, the spatial envelope of the instrument image in the occlusion entry frame and the occlusion exit frame is expanded along the angular direction to generate a spiral edge distribution sequence. Arrange the spiral edge distribution sequence in ascending order of polar angle to form two sets of spiral edge vectors with the same direction. Perform sliding window matching on the two sets of spiral edge vectors, calculate the edge offset index under the same angular direction, and output the structural difference map; Extract the polar angle segments corresponding to the label set of high-risk structural parts from the structural difference map, and perform structural difference analysis to determine whether there are defective segments with texture breaks or shortened contours. If so, mark them as abnormal instrument structure loss.

7. The method for identifying abnormal states of surgical instruments based on image recognition according to claim 1, characterized in that, In S6, all instruments with structural loss anomalies are acquired, and the residual area after component loss is output within the surgical field space, combined with the occlusion area. Specifically, this includes: The defective segment corresponding to the structurally missing instrument is reverse-mapped to the two-dimensional image space envelope of the instrument in the image frame to generate the image space missing region. Based on the overlapping location of the obscured area and the missing area in the surgical field, the possible breakage and detachment range of the component can be estimated. Perform connectivity and closure tests on all fractured and detached areas, and output a set of structurally closed component retention areas.

Images