Surgical instrument identification method and system based on artificial intelligence

Abstract

The invention discloses a surgical instrument identification method and system based on artificial intelligence, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a multi-modal original input data set; feature extraction fusion and dynamic identification tracking; state evaluation cleanliness; intelligent decision recommendation; and data storage interacts with the system. The system comprises a multi-modal acquisition module, a preprocessing and enhancing module, a feature extraction fusion and dynamic identification tracking module, a state evaluation and cleanliness detection module, an intelligent decision recommendation module and a data management integration module. The problem that existing surgical instrument recognition is tedious and time-consuming is solved, the robustness and precision of surgical instrument recognition can be improved in a complex environment, dynamic tracking, counting and state evaluation are achieved, cleanliness detection and intelligent decision recommendation are supported, and the efficiency and safety of the surgical process are improved through data sharing and system integration.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence, and particularly relates to a surgical instrument identification method and system based on artificial intelligence. BACKGROUND

[0002] With the continuous progress of global medical health, the safety and efficiency of surgical operation as a key diagnosis and treatment method are increasingly valued. Surgical instruments, as the core tools in surgical operation, are not only of various types and complex specifications, but also have a fast update speed and are difficult to manage. In the traditional operation room operation process, the counting, identification, tracing and maintenance of instruments are important foundations to ensure the safety and smoothness of the operation. Especially in the clinical environment with high operation intensity and high risk, accurate identification and efficient management of instruments have become an important research direction to improve medical quality and optimize resource allocation.

[0003] Currently, the identification and management of surgical instruments still mainly rely on manual methods, which are inefficient and prone to errors. Staff may miss or misjudge the counting due to fatigue or distraction, especially when dealing with similar shapes, multiple types or slightly damaged instruments, which further increases the difficulty of identification. On the other hand, although traditional machine vision or early artificial intelligence algorithms have been tried to be applied in this field, their performance in real surgical scenarios is still not satisfactory. The reflection, bloodstains, occlusion and shadow changes on the surface of the instrument, as well as the rapid movement of the target in the dynamic environment, will significantly interfere with the identification effect, resulting in insufficient accuracy and robustness of the system. In addition, existing technologies generally cannot achieve continuous tracking and accurate counting of instruments, and have limited capacity when dealing with large-scale instrument data. The integration level between the system and external platforms such as hospital information systems or sterilization supply systems is low, and a complete instrument life cycle management and real-time risk warning mechanism cannot be established, thereby limiting the overall improvement of the operation room management level. Therefore, the industry urgently needs a new solution for efficient, accurate and intelligent adaptive surgical instrument identification and management. SUMMARY

[0004] The present application provides a video automatic labeling method and system based on background difference and contour detection to solve the problems of tedious process, long time consumption, large workload, poor adaptability to multi-category instruments, complex environment, high similarity instruments, metal reflection and dynamic tracking in the prior art.

[0005] To solve the above technical problems, the present application provides the following technical scheme: a video automatic labeling method based on background difference and contour detection, comprising the following steps:

[0006] S100: Obtain a multi-modal raw input data set of surgical instruments, which includes a visible light image sequence, a multi-spectral image sequence, a three-dimensional depth map sequence, and environmental context data;

[0007] S200: Based on the multi-modal raw input data set, a deep learning model is used to extract multi-scale features;

[0008] S300: Based on the multi-scale features, an identification and classification module is used to identify and classify the surgical instruments, outputting the instrument category, model, unique identifier, and confidence, and generating the corresponding bounding box coordinates;

[0009] S400: Based on the identification and classification of the surgical instruments, further dynamic tracking and state evaluation of the identified instruments are performed to form a comprehensive monitoring result of instrument use and maintenance;

[0010] S500: Based on the comprehensive monitoring result, after completing the identification and state evaluation of the surgical instruments, the relevant results are centralized stored and managed, and through the interface with the external medical information system, an intelligent decision support and visual interaction mechanism is formed.

[0011] Visible light image data of surgical instruments is obtained through a high-resolution visible light imaging unit, which uses a complementary metal oxide semiconductor sensor with not less than 12 million pixels and a frame rate of not less than 30 frames / second;

[0012] Multi-spectral image data of surgical instruments is obtained through a multi-spectral imaging unit, which collects image data in the visible light and near-infrared wave bands;

[0013] Three-dimensional depth data of surgical instruments is obtained through a depth perception unit, which is based on the structure light or time of flight principle, with a depth measurement accuracy of not higher than 1 millimeter.

[0014] Alignment is achieved through a timestamp mechanism, ultimately forming a multi-modal raw input data set covering visible light image sequences, multi-spectral image sequences, three-dimensional depth map sequences, and environmental context data.

[0015] A convolutional neural network module based on a residual network or an efficient network as a backbone network architecture extracts multi-level visual features;

[0016] A multi-scale feature pyramid network structure is used to fuse the multi-level visual features extracted by the convolutional neural network module;

[0017] A self-attention mechanism module is used to capture global context information and long-distance dependencies of surgical instruments to enhance the ability to distinguish similar instruments;

[0018] Through the fusion network based on the adaptive gating mechanism, the features of different modalities and different scales are integrated, and the integrated features are mapped nonlinearly through a multi-layer fully connected network or a Transformer architecture to obtain a multi-dimensional fusion representation vector.

[0019] Based on the multi-dimensional fusion representation vector, a fully connected layer structure based on residual connection or a convolutional network with global average pooling is used to output the category prediction result of the instrument; the category prediction result includes the unique name, standard model, department to which the instrument belongs, and identification confidence;

[0020] A target detection algorithm module based on real-time object detection series algorithms or regional convolutional neural network series algorithms is used for real-time bounding box detection and category classification of surgical instruments in the enhanced multi-modal perception data.

[0021] While completing the category recognition, the target detection model is used to spatially locate the instrument, and the target detection model adopts the structure of a multi-stage detector or a single-stage detector to output the bounding box coordinates in a two-dimensional image or the minimum bounding box in a three-dimensional space; the bounding box generation process is optimized by a bounding box regression loss function.

[0022] A tracking framework combining Kalman filtering and Hungarian algorithm or a deep learning tracking network structure is used to match the instruments identified in the current frame with the trajectories of historical frames to realize continuous frame tracking and counting.

[0023] Through image segmentation and edge detection methods, the scratch, crack, rust, and deformation defect regions are located and compared with the standard model of the instrument to quantify the severity and distribution of the defects; for blade-type instruments, the wear depth and range are measured to determine whether they meet the repair or scrap criteria, thereby achieving objective evaluation of the service life and functional state of the instrument.

[0024] The data is stored in a structured form using the table structure of a relational database, and each instrument entry, exit, use, cleaning, repair, and scrap event forms a complete record in the database.

[0025] Based on the complete record, combined with the context information provided by the external medical system, the decision support and human-computer interaction module outputs intelligent decision suggestions and real-time warnings.

[0026] Through the human-computer interaction interface unit, the identification results, positioning information, counting results, state evaluation, and warning information of the instrument are visually presented in an intuitive manner.

[0027] The multi-dimensional state results, i.e., identification results, positioning information, counting results, state evaluation and early warning information of the instrument, are integrated into a single index through a health score mechanism.

[0028] An artificial intelligence-based surgical instrument identification system comprises the following modules:

[0029] A data perception module is configured to collect multi-modal raw data of surgical instruments to form a multi-modal raw input data set.

[0030] A data preprocessing module is electrically connected to the data perception module, receives the multi-modal raw input data set transmitted by the data perception module, and performs cleaning, standardization, denoising and registration operations.

[0031] A multi-scale feature extraction module is electrically connected to the data preprocessing module, and uses a deep learning model to extract discriminative multi-scale features from the preprocessed multi-modal data.

[0032] An identification and classification module is responsible for accurate identification and classification of surgical instruments, receives the fusion feature vector output by the multi-scale feature extraction module, and inputs it into a pre-trained classifier.

[0033] A state evaluation module comprehensively evaluates the use state of the identified instrument, including cleanliness, integrity and wear condition.

[0034] A database management module is responsible for storing and managing relevant data of all instruments, adopts a distributed database architecture, uses a non-relational database based on columnar storage or key-value storage, and supports fast read-write and efficient retrieval of large-scale data.

[0035] A system integration module provides standardized data interfaces and communication protocols for interfacing with hospital information systems, operating room management systems and material management systems, supports transmission control protocol and user datagram protocol network communication methods, and provides application programming interfaces based on representational state transfer.

[0036] The state evaluation module comprises:

[0037] A cleanliness detection unit uses the waveband features sensitive to proteins and hemoglobin in multi-spectral images, combined with a machine learning classifier, to determine whether there are organic residues on the surface of the instrument.

[0038] An integrity checking unit detects whether there is deformation by comparing the real-time acquired three-dimensional point cloud data with the standard instrument three-dimensional model.

[0039] A wear detection unit uses high-resolution visible light images and depth images, combined with edge detection and texture analysis algorithms, to identify minor defects and quantify the range and severity of the defects.

[0040] Compared with the prior art, the present application has the following advantages:

[0041] The present application effectively overcomes the limitations of single visual modalities being easily disturbed by reflections, bloodstains and shadows by introducing multi-modal perception data acquisition technology, fusing visible light, multispectral and three-dimensional depth information, and significantly enhancing the robustness of instrument recognition in complex surgical environments.

[0042] The present application adopts a multi-scale feature extraction and fusion network based on deep learning, combining the powerful local feature extraction capability of convolutional neural networks and the global context modeling advantage of self-attention mechanisms, to efficiently process surgical instruments with diverse shapes and various categories.

[0043] The present application integrates instrument state evaluation and cleanliness detection functions, relying on high-resolution image analysis and multispectral quantitative technology, to automatically identify physical defects such as wear, cracks and corrosion of instruments, and detect cleanliness indicators such as surface protein residues or bloodstains, thereby ensuring instrument quality and sterilization qualification rate.

[0044] The present application uses knowledge graph and recommendation algorithm to comprehensively consider surgical type, patient pathological information and instrument inventory status, intelligently generates the optimal instrument matching scheme, real-time alerts instrument missing or misplacement, significantly shortens preoperative preparation time, and improves surgical efficiency and accuracy, especially providing fast and reliable decision support in emergency scenarios.

[0045] The present application establishes a unified surgical instrument database and standardized interface data management and system integration module, realizing seamless connection and data sharing with external platforms such as hospital information systems and operating room management systems, overcoming the previous data island and system integration problems.

[0046] The present application solves the problems of complexity, time-consuming and heavy workload in surgical instrument management from multiple levels of perception, recognition, tracking, state evaluation to intelligent decision-making, realizes end-to-end automated management, and improves surgical quality and patient safety, having significant clinical value and application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, and the following drawings only show some embodiments of the present application, therefore should not be regarded as limiting the scope, for those skilled in the art, other related drawings can also be obtained without creative labor.

[0048] Figure 1 is a schematic diagram of the surgical instrument recognition method of the present application based on artificial intelligence;

[0049] Figure 2 is a schematic diagram of a surgical instrument recognition system architecture based on artificial intelligence according to the present application. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the scope of protection of the present application. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only for selected embodiments of the present application.

[0051] Please refer to Figure 1 , Figure 1 is a schematic diagram of a surgical instrument recognition method flow based on artificial intelligence according to the present application, comprising the following steps:

[0052] S100: Obtain a multi-modal original input data set of a surgical instrument, the multi-modal original input data set comprising a visible light image sequence, a multi-spectral image sequence, a three-dimensional depth map sequence and environmental context data, specifically comprising:

[0053] S101: High-resolution visible light image data acquisition, through a high-resolution visible light imaging unit, the high-resolution visible light imaging unit adopts a complementary metal oxide semiconductor sensor with not less than 12 million pixels, and a frame rate of not less than 30 frames / second, to continuously acquire a two-dimensional image sequence of the instrument at a preset frame rate in a surgical area or an instrument cleaning area; during the acquisition process, the influence of environmental light fluctuation is suppressed in real time in combination with an adaptive lighting system, to ensure that clear and stable input can still be obtained under complex lighting conditions, the obtained image is stored in a lossless compression format, and an accurate time stamp is attached, thereby realizing stable high-quality visible light image output.

[0054] S102: Multi-spectral image data acquisition, while visible light acquisition is performed, the system utilizes a multi-spectral imaging unit to acquire image data in the visible light band and the near-infrared band, specifically: synchronously acquiring instrument images under specific narrow-band spectra such as 470nm, 530nm, 660nm and 850nm, the imaging results of each band reflect the absorption and reflection characteristics of the instrument material under the corresponding spectrum; among them, blood stains present obvious absorption peaks in the infrared band, and metal surfaces have significant reflection differences in the visible light and near-infrared regions; by introducing the multi-spectral dimension, effective differentiation of the instrument body and attachments is realized, and the accuracy of pollution detection and identification is improved.

[0055] S103: Three-dimensional depth information collection. In addition to two-dimensional and spectral information, three-dimensional point cloud data or depth map of the instrument surface is further obtained through the depth perception unit composed of a structured light projector and an infrared camera. The depth measurement precision is not higher than 1 mm, which is realized based on the structured light or time-of-flight principle. The obtained data can accurately describe the geometric shape, size parameter and spatial position of the instrument. Even in the case of multiple instruments stacking or partial occlusion, the reliability of identification can still be ensured. The precision of depth information can reach sub-millimeter level, which can capture fine structural features and provide a basis for distinguishing instruments with similar shapes but different sizes.

[0056] S104: Environmental context information collection. In addition to the perception data of the instrument itself, environmental context information related to identification is also collected, including illumination intensity, background clutter distribution and instrument placement density, etc. Such information can be used to adaptively adjust image enhancement and segmentation parameters after entering the preprocessing module, such as dynamically optimizing the reflection elimination strategy or background removal method, so as to maintain the stability and robustness of the identification process in different working environments.

[0057] The data collected by the above-mentioned various sensors is aligned through the time stamp mechanism after preliminary caching, and finally forms a multi-modal raw input data set covering the visible light image sequence, the multi-spectral image sequence, the three-dimensional depth map sequence and the environmental context data.

[0058] S200: Multi-scale feature extraction of multi-modal surgical instrument perception data. Multi-scale feature extraction is performed on the multi-modal raw input data set. Deep learning models are used to extract multi-scale features with rich semantic information and high discriminability from data of different modalities, which specifically include:

[0059] S200: Multi-scale feature extraction of multi-modal surgical instrument perception data. Multi-scale feature extraction is performed on the multi-modal raw input data set. Deep learning models are used to extract multi-scale features with rich semantic information and high discriminability from data of different modalities, which specifically include:

[0060] S201: Multi-scale feature extraction of visible light image, input the visible light image sequence obtained in S101 into a deep convolutional neural network model, which can adopt the architecture form of residual network or dense connection network; through the layer-by-layer stacking of multiple convolutional layers, activation functions and pooling layers, the network captures local fine-grained features such as edges and textures in the low layer, identifies component-level features such as joints or handle shapes in the middle layer, and extracts overall structure and semantic features in the high layer. In order to highlight the key recognition area, channel attention or spatial attention mechanism is introduced in the network to dynamically adjust the weights of different feature channels or spatial regions. After multi-scale feature fusion realized by the feature pyramid network, a visible light multi-scale feature pyramid containing rich spatial details and high-level semantics is finally formed, which is used to comprehensively represent the significant features of the instrument under two-dimensional visible light imaging.

[0061] S202: Joint extraction of spectral and spatial features of multi-spectral image, input the multi-spectral image data obtained in S102 into the multi-spectral branch of the convolutional neural network, which first performs independent convolution operation on each waveband image to extract the specific response features of each narrowband waveband. Then, through a cross-waveband fusion layer such as three-dimensional convolution or attention weighting mechanism, different waveband features are combined to capture the collaborative information between different wavebands. This process not only strengthens the absorption performance of bloodstains in the infrared band and the reflection difference of metal surfaces in the visible light and near-infrared regions, but also takes into account local texture and shape and other spatial features. The multi-spectral joint features generated thereby have high sensitivity to the material properties, pollution state and minor defects of the instrument surface.

[0062] S203: Geometric and topological feature extraction of three-dimensional depth information, input the three-dimensional depth information obtained in S103 into a three-dimensional convolutional neural network or a point cloud processing network to extract the geometric features and topological structure of the instrument. Such network can accurately capture the size, volume, surface curvature, concave-convex degree and relative position relationship between components of the instrument. For instruments with complex curved surfaces or slender structures, three-dimensional depth information can provide accurate physical size and shape representation, thereby maintaining the reliability of recognition in the case of partial occlusion or stacking of instruments, and effectively distinguishing instruments with similar appearance but minor size differences.

[0063] S204: Deep fusion of multi-modal features, input the visible light multi-scale features, multispectral joint features and three-dimensional geometric topology features obtained in steps S201, S202 and S203 into a fusion network for deep fusion of multi-modal features; the fusion network is based on an adaptive gating mechanism to dynamically evaluate the importance and reliability of different modal features and assign weights to each modal feature; when the visible light feature is affected by reflection, the multispectral feature weight is increased; when the signal-to-noise ratio of the multispectral image is reduced, the three-dimensional depth feature weight is enhanced. The fused features are nonlinearly mapped through a multi-layer fully connected network or a Transformer architecture to form a unified, high semantic density, multi-modal fusion representation vector for comprehensively describing the digital features of the instrument, which is represented by the following formula:

[0064]

[0065] wherein, represents the fused feature vector, , , represent the visible light, multispectral and depth feature vectors respectively; , , represent the weight matrix of the corresponding modal feature, and B represents the bias vector, represents a nonlinear activation function; the above weight matrix and bias vector are obtained through an end-to-end training process of the fusion network.

[0066] S300: Surgical instrument recognition and classification based on multi-scale feature coding, after completing the multi-modal feature extraction and fusion of S200, the generated multi-dimensional fusion representation vector of the surgical instrument is input into a classification and detection model to realize the classification of the instrument, robust recognition in complex environment and spatial positioning, which specifically includes:

[0067] S301: Instrument type discrimination and fine-grained classification, input the multi-dimensional fusion representation vector obtained in S204 into a multi-label classification neural network, which uses a fully connected layer structure based on residual connection or a convolutional network with global average pooling to output the class prediction result of the instrument; for instruments with extremely high appearance similarity, the classification module introduces a fine-grained recognition strategy, which constrains the distance between similar instruments to be smaller than that between dissimilar instruments in the feature space through a contrastive learning loss function, thereby enhancing the model's ability to distinguish subtle differences; the final output classification result includes the unique name, standard model, department to which the instrument belongs and recognition confidence, realizing complete recognition from coarse-grained class discrimination to fine-grained model differentiation.

[0068] S302: Robust Recognition Under Reflection and Shadow Interference. To address the interference of reflection and shadow on recognition performance, the classifier introduces adversarial training and domain adaptation mechanisms during the training phase. Through generative adversarial networks, the feature extraction module is forced to maintain high robustness under various lighting conditions and interference environments, ensuring that the extracted representation features are not distorted due to lighting fluctuations. At the same time, the classifier adaptively optimizes samples in complex scenarios through reinforcement learning strategies, enabling the system to stably output reliable recognition results in practical applications.

[0069] Furthermore, in conjunction with the confidence assessment mechanism, when the confidence level of identification is lower than the set threshold, the system marks the device as "high-risk identification" and automatically triggers the manual review process to ensure safety and reliability in critical scenarios.

[0070] S303: Instrument Localization and Bounding Box Generation. While performing category recognition, this function uses an object detection model to spatially locate the instrument. The object detection model employs a multi-stage or single-stage detector structure, outputting the bounding box coordinates in the 2D image or the minimum bounding box in 3D space. This localization result not only provides spatial reference for instrument counting and dynamic tracking but also offers intuitive geometric information for human-computer interaction and automated operation. The bounding box generation process is optimized using a bounding box regression loss function, as follows:

[0071]

[0072] in, It is the bounding box regression loss. and These are weighting coefficients. It is the number of grid cells. It is the number of bounding boxes predicted for each grid cell. This indicates that the j-th prediction box in the i-th grid cell is responsible for detecting a target. , , , )and( , , , These are the center coordinates and width / height of the predicted and actual bounding boxes, respectively.

[0073] S400: Dynamic tracking and status assessment of surgical instruments. After identifying and classifying surgical instruments, further dynamic tracking and status assessment are performed on the identified instruments to generate comprehensive monitoring results regarding instrument use and maintenance, specifically including:

[0074] S401: Real-time dynamic tracking of instruments. For instruments that continuously exist within the sensing area, continuous localization in the time series is achieved through a multi-target tracking algorithm. The multi-target tracking algorithm adopts a scheme based on a combination of Kalman filtering and Hungarian algorithm, or a deep learning tracking network structure. By matching the trajectory of the instrument identified in the current frame with that in historical frames, the system assigns a unique tracking identifier to each instrument. When an instrument is occluded or briefly leaves the field of view, the tracking module uses motion models and appearance features to predict its possible location and associates it when it reappears, ensuring the continuity and stability of the tracking process and realizing continuous frame tracking and counting.

[0075] S402: Instrument Counting and Integrity Verification. Based on real-time dynamic tracking results, the system can accurately count all instruments within the sensing area and compare them with a pre-set surgical instrument list. Before, during, and after surgery, the system automatically completes instrument counting and verification. This process can be combined with laser-etched codes or color markings on the instrument surface for identification, thus avoiding double counting or omissions. When the actual quantity does not match the list, or when an instrument not on the list is detected, the system immediately triggers an early warning mechanism, prompting relevant personnel to verify and handle the situation.

[0076] S403: Instrument Cleanliness Detection. The system utilizes the multispectral features extracted in S202 combined with the multispectral image data acquired in S102. Through image analysis algorithms and a pre-trained classification model, it detects the cleanliness of instrument surfaces. This process can identify residual substances that are difficult to detect with the naked eye, including bloodstains, tissue debris, and disinfectant residues. By analyzing the spectral reflectance and absorptivity in specific wavelength bands, the system can quantify the degree of surface cleanliness and classify instruments into multiple cleanliness levels according to set standards. This detection mechanism reliably ensures the hygienic condition of instruments before reuse.

[0077] S404: Instrument Surface Defect and Wear Assessment. This further utilizes the high-resolution visible light multi-scale features extracted in S201, the three-dimensional geometric topological features extracted in S203, and the original high-resolution visible light image to automatically assess instrument surface defects and wear. Through image segmentation and edge detection methods, the system can accurately locate defect areas such as scratches, cracks, corrosion, and deformation, and compare them with a standard model of the instrument to quantify the severity and distribution of defects. For cutting edge instruments, the system can measure the wear depth and extent to determine whether they meet the criteria for repair or scrapping. This achieves an objective assessment of the instrument's service life and functional status.

[0078] S500: Instrument information management and system linkage based on identification and evaluation results. After completing the identification and status evaluation of surgical instruments, the system centrally stores and manages the relevant results, and forms an intelligent decision support and visual interaction mechanism by connecting with external medical information systems, specifically including:

[0079] S501: Instrument Identification and Status Information Storage. The instrument identification results generated in S300, including instrument name, model, unique identifier, location information, and identification confidence level, and the status assessment results generated in S400, including cleanliness level, integrity status, and descriptions of defects and wear, along with timestamps and operator information, are written to the database management module in real time via the system integration module. This data is stored in a structured format, using a relational database table structure to ensure traceability and queryability. Every instrument entry, exit, use, cleaning, maintenance, and disposal event is recorded completely in the database, ensuring systematic retention of information throughout its entire lifecycle.

[0080] S502: Interface and Information Sharing with External Medical Systems. Through standardized interfaces, the system enables bidirectional data exchange with hospital information systems, surgical scheduling systems, and material management systems. The system can obtain upcoming surgical types and related patient information from the surgical scheduling system, thereby pre-loading the corresponding equipment list. Simultaneously, it synchronizes real-time updates on inventory status, cleanliness information, and equipment availability to the material management system, enabling automated inventory checks and replenishment recommendations. In emergency surgical scenarios, the system can quickly generate a list of available equipment and indicate missing items, reducing the risk of surgical delays due to insufficient equipment preparation.

[0081] S503: Intelligent Decision Support and Early Warning. Based on the identification results and status data stored in S501, combined with contextual information provided by external medical systems, the system's decision support and human-computer interaction module can output intelligent decision suggestions and real-time early warnings. When unqualified instrument cleanliness or serious defects are detected, the early warning management unit immediately generates a high-priority warning, prompting relevant medical staff to suspend the use of the instrument. During the preoperative preparation stage, the system recommends the optimal combination and specifications of instruments based on the type of surgery and patient characteristics, combined with instrument library information. For orthopedic surgery, it automatically matches the length and angle of the blades to shorten preparation time and improve efficiency. During the operation, the system monitors the instruments in use in real time and dynamically updates the count results; if a discrepancy in quantity or abnormal removal of instruments is found, an audible and visual alarm is immediately triggered, and the location of potentially lost instruments is marked to prevent the risk of surgical instruments remaining in the body.

[0082] S504: Human-Computer Interaction and Visualization. Through the human-computer interaction interface, the system visually presents the device identification results, location information, counting results, status assessment, and warning information in an intuitive manner. The interface can overlay device identification bounding boxes on the display, highlighting devices at risk. It also provides historical data query functionality, allowing healthcare professionals to trace device usage frequency, maintenance records, and status evolution. The interface interaction also allows healthcare professionals to perform direct operations, such as confirming inventory results, marking specific devices, or retrieving device lifecycle information. To facilitate a quick understanding of the overall device status, the system integrates multi-dimensional status results into a single indicator through a health scoring mechanism, calculated as follows:

[0083]

[0084] in, For device health scoring, To score cleanliness, For completeness score, Score the degree of wear and tear. , , These are the corresponding weights, and .

[0085] Through the above steps, centralized storage, cross-system sharing, intelligent decision support, and visual interaction of identification and evaluation results are achieved, which can comprehensively solve various challenges in surgical instrument management and significantly improve the level of medical quality and safety.

[0086] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the surgical instrument recognition system architecture based on artificial intelligence of the present invention, which includes the following modules:

[0087] The system includes a data perception module, a data preprocessing module, a multi-scale feature extraction module, a recognition and classification module, a state assessment module, a database management module, a system integration module, and a decision support and human-computer interaction module.

[0088] The data sensing module is used to collect multimodal raw data from surgical instruments and integrates multiple sensor units, including a high-resolution visible light imaging unit, a multispectral imaging unit, a depth sensing unit, and an optional thermal imaging unit.

[0089] The high-resolution visible light imaging unit uses a megapixel-level digital camera with high frame rate acquisition capability and is equipped with autofocus and zoom functions to capture the macroscopic morphological features and surface details of instruments, adapting to different working distances and instrument sizes.

[0090] Multispectral imaging units acquire images within a specific wavelength range using narrowband filters or tunable filters. This effectively distinguishes surface residues such as bloodstains and water stains, reduces interference from metal reflections, and reveals material properties and surface defects that are difficult to detect under visible light conditions.

[0091] The depth sensing unit generates 3D point cloud data or depth maps of the instrument based on structured light or time-of-flight principles, providing spatial information for size measurement, attitude estimation, and identification under occlusion conditions.

[0092] Thermal imaging units are used in specific scenarios to monitor the temperature distribution on the surface of instruments in order to help determine whether they have been sterilized at high temperatures or whether there is any abnormal heating.

[0093] All sensor units are aligned using a unified time synchronization mechanism, and the acquired results are transmitted to the data preprocessing module via a high-speed data bus. The module's parameter settings include: sensor hardware calibration parameters, adaptive exposure adjustment information provided by the ambient light sensor, and user-defined sampling frequency and resolution. Its verification mechanism encompasses sensor self-testing, data integrity checksums, and cross-modal data consistency comparison. Typical performance parameters are: visible light resolution up to 8 megapixels, multispectral band coverage from 400nm to 1000nm, and depth sensing accuracy up to 1mm.

[0094] The data preprocessing module receives the raw multimodal data transmitted from the data sensing module and performs operations such as cleaning, standardization, noise reduction, registration, and enhancement to improve data quality and consistency, including:

[0095] Image denoising unit: Employs a deep learning-based convolutional neural network denoising model to effectively suppress random noise generated during the acquisition process and ensure clear image details.

[0096] Reflection elimination unit: Based on multispectral imaging data or polarized light technology, it analyzes and separates the specular reflection and diffuse reflection components through a reflection model to eliminate strong reflective interference on the surface of metal instruments.

[0097] Shadow correction unit: Combining depth information, it uses local contrast enhancement and global brightness equalization algorithms to compensate for shadow areas and restore the true texture and color of the instrument.

[0098] Data registration unit: Spatially aligns visible light images, multispectral images, and depth maps to ensure that different modal data correspond to the same physical region, laying the foundation for subsequent feature fusion.

[0099] Data augmentation unit: During the model training phase, it performs operations such as rotation, scaling, cropping, and color perturbation on the original data to expand the diversity of the dataset and improve the model's generalization ability and adaptability to complex environments.

[0100] All preprocessing steps are executed based on a parallel computing architecture to ensure that real-time processing requirements are met.

[0101] The multi-scale feature extraction module is the core intelligent component of the system. It uses a deep learning model to extract discriminative multi-scale features from preprocessed multimodal data, including:

[0102] Feature extraction network: Composed of deep convolutional neural networks, it can adopt residual network architecture or convolutional network with attention mechanism. The network contains multiple levels of downsampling and upsampling layers, which can extract local texture features, edge features and global structural features under different receptive fields.

[0103] Modal feature branch: For visible light images, multi-scale features from low-level edges to high-level semantics are extracted by convolution layer by layer; for multispectral data, independent channel branches are set up to extract spectral response features of each band. These features are extremely sensitive to residues such as bloodstains and rust; for depth data, the three-dimensional geometric shape and surface undulation features of the instrument are extracted to ensure the accuracy of shape recognition.

[0104] Feature fusion unit: Employing cross-modal attention or gating mechanisms, features from different modalities and scales are dynamically weighted and fused to generate a unified high-dimensional feature encoding vector. This encoding can comprehensively characterize the physical properties, geometric shape, and surface condition of the device.

[0105] Small target detection mechanism: By fusing features at different scales through a feature pyramid network, the detection accuracy of small instruments and densely distributed instruments is improved.

[0106] The identification and classification module is responsible for the accurate identification and classification of surgical instruments. It receives the fused feature vector output by the multi-scale feature extraction module and inputs it into a pre-trained classifier. The classifier is based on a fully connected neural network or support vector machine and learns from a large-scale labeled dataset to identify more than 1,000 types of instruments, including blades of different specifications, forceps of different functions, and suture needles of different models.

[0107] To address the challenge of distinguishing highly similar medical devices, the classifier employs a fine-grained recognition mechanism. This involves introducing a metric learning loss function or a multi-task learning strategy into the network, forcing the model to capture subtle differences between similar devices, such as handle coding and minute variations in blade curvature. The module also includes a confidence evaluation unit, providing a confidence score for each recognition result and triggering manual review or further analysis when the score falls below a set threshold. The final output includes the device category name, a unique identifier, and the bounding box coordinates of the recognized region.

[0108] The condition assessment module provides a comprehensive evaluation of the usage status of identified instruments, covering cleanliness, integrity, and wear and tear, including:

[0109] Cleanliness detection unit: Utilizing the spectral features of multispectral images that are sensitive to proteins and hemoglobin, combined with a machine learning classifier, it determines whether there are organic residues on the surface of the instrument.

[0110] Integrity check unit: By comparing the real-time acquired 3D point cloud data with the standard instrument 3D model, it detects whether there are deformations such as bending or breakage.

[0111] Wear detection unit: Utilizing high-resolution visible light images and depth images, combined with edge detection and texture analysis algorithms, it identifies minute defects such as scratches, wear, and cracks, and quantifies the extent and severity of the defects; for the cutting edge of surgical instruments, it can accurately measure its dullness and determine whether replacement is necessary.

[0112] The status assessment results include cleanliness level, integrity status, wear level and location, and are stored in correspondence with the instrument's unique identifier.

[0113] The database management module is responsible for storing and managing all medical device-related data. It employs a distributed database architecture, based on a non-relational database with columnar or key-value storage, to support fast read / write and efficient retrieval of large-scale data. The data stored in the database is divided into:

[0114] Static attributes: Name, Model, Manufacturer, Purchase Date, Value, Lifetime, etc.;

[0115] Dynamic attributes include: historical identification records, number of uses, cleaning and disinfection records, maintenance records, status assessment reports, and location information.

[0116] To ensure consistency and security, the module supports data encryption, access control, and regular backups. The system implements data CRUD operations through a database interface. After each identification or status assessment is completed, the relevant data is written to the database in real time, forming a complete lifecycle traceability chain.

[0117] The system integration module provides standardized data interfaces and communication protocols for interfacing with hospital information systems, operating room management systems, and material management systems. It supports network communication methods such as Transmission Control Protocol (TCP) and User Datagram Protocol (UDP), and provides an application programming interface based on representational state transitions to enable bidirectional data exchange.

[0118] Through system integration, this system can acquire patient imaging data and surgical scheduling information, providing a basis for equipment recommendation and preparation. Simultaneously, it can transmit identification, inventory, and status assessment results back to the materials management system, enabling automated inventory updates and precise management. When equipment is detected as missing or in abnormal condition, the system can generate alerts and notify relevant medical staff through the hospital information system.

[0119] The decision support and human-computer interaction module, based on the recognition and evaluation results, provides medical staff with an intelligent decision-making and visual interactive interface, including:

[0120] Intelligent recommendation unit: Combining patient image data, surgical type and instrument library information, it uses collaborative filtering or rule-based recommendation algorithms to automatically generate the best combination and specifications of instruments, shortening preoperative preparation time.

[0121] Early warning management unit: Based on the results of instrument inventory and status assessment, it monitors the usage of instruments in real time before, during and after surgery. When missing, count discrepancies, substandard cleanliness or serious defects are found, a high-priority early warning is immediately generated and relevant personnel are notified through audible and visual alarms, mobile terminals or desktop pop-ups.

[0122] Human-Computer Interaction Interface Unit: Provides a graphical interface that intuitively presents device identification results, location information, status reports, and historical query functions, supporting quick searching, verification, and management. The interface can overlay identification bounding boxes, highlight abnormal devices, and allow users to input commands, such as confirming inventory results, marking specific devices, or querying their complete lifecycle information.

[0123] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations will be apparent to those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A surgical instrument recognition method based on artificial intelligence, characterized in that, include: S100: Acquire the multimodal raw input dataset of surgical instruments, which includes visible light image sequences, multispectral image sequences, three-dimensional depth map sequences, and environmental context data; S200: Based on the aforementioned multimodal original input dataset, a deep learning model is used to extract multi-scale features; S300: Based on the multi-scale features, the surgical instruments are identified and classified using the identification and classification module, and the instrument category, model, unique identifier and confidence level are output, and the corresponding bounding box coordinates are generated. S400: Based on the identification and classification of the surgical instruments, further dynamic tracking and status assessment of the identified instruments are performed to form a comprehensive monitoring result on the use and maintenance of the instruments; S500: Based on the comprehensive monitoring results, after completing the identification and status assessment of surgical instruments, the relevant results are centrally stored and managed, and an intelligent decision support and visual interaction mechanism is formed by connecting with external medical information systems.

2. The artificial intelligence-based surgical instrument recognition method according to claim 1, characterized in that, The acquisition of the multimodal raw input dataset of surgical instruments includes: Visible light image data of surgical instruments are acquired through a high-resolution visible light imaging unit, which employs a complementary metal-oxide-semiconductor sensor with no less than 12 million pixels and a frame rate of no less than 30 frames per second. Multispectral image data of surgical instruments is acquired by a multispectral imaging unit, which acquires image data in the visible light band and the near-infrared band. The three-dimensional depth data of the surgical instruments is acquired through a depth sensing unit, which is based on the principle of structured light or time-of-flight, and the depth measurement accuracy is no higher than 1 mm.

3. The artificial intelligence-based surgical instrument recognition method according to claim 1, characterized in that, The multimodal raw input dataset for acquiring surgical instruments includes: Alignment is achieved through a timestamp mechanism, ultimately forming a multimodal raw input dataset that includes visible light image sequences, multispectral image sequences, 3D depth map sequences, and environmental context data.

4. The artificial intelligence-based surgical instrument recognition method according to claim 1, characterized in that, The extraction of multi-scale features using a deep learning model includes: Convolutional neural network modules based on residual networks or high-efficiency networks as the backbone network architecture extract multi-level visual features. A multi-scale feature pyramid network structure is used to fuse multi-level visual features extracted by the convolutional neural network module; The self-attention mechanism module is used to capture global contextual information and long-distance dependencies of surgical instruments to enhance the ability to distinguish highly similar instruments. By using a fusion network based on an adaptive gating mechanism, features of different modalities and scales are integrated. The integrated features are then nonlinearly mapped through a multi-layer fully connected network or a Transformer architecture to obtain a multi-dimensional fusion representation vector.

5. The artificial intelligence-based surgical instrument recognition method according to claim 1, characterized in that, The method of using an identification and classification module to identify and classify surgical instruments includes: Based on the multidimensional fusion representation vector, a fully connected layer structure based on residual connections or a convolutional network with global average pooling is used to output the category prediction result of the device; the category prediction result includes: the unique name of the device, the standard model, the department to which it belongs, and the recognition confidence level; A target detection algorithm module based on a series of real-time object detection algorithms or a series of regional convolutional neural network algorithms is used to perform real-time bounding box detection and category classification of surgical instruments in the enhanced multimodal perception data; While completing category recognition, the target detection model is used to spatially locate the instrument. The target detection model adopts a multi-stage detector or a single-stage detector structure and outputs the bounding box coordinates in the two-dimensional image or the minimum bounding box in the three-dimensional space. The bounding box generation process is optimized by the bounding box regression loss function.

6. The artificial intelligence-based surgical instrument recognition method according to claim 1, characterized in that, The condition assessment and cleanliness detection of the identified surgical instruments include: A tracking framework combining Kalman filtering and the Hungarian algorithm or a deep learning tracking network structure is adopted to achieve continuous frame tracking and counting by matching the trajectory of the device identified in the current frame with that in historical frames. By using image segmentation and edge detection methods, areas of scratches, cracks, corrosion, and deformation defects are located and compared with the standard model of the instrument to quantify the severity and distribution of defects. For cutting edge instruments, the wear depth and range are measured to determine whether they meet the criteria for repair or scrapping, thus achieving an objective assessment of the instrument's service life and functional status.

7. The artificial intelligence-based surgical instrument recognition method according to claim 1, characterized in that, The intelligent decision support and visualization interaction mechanism includes: The data is stored in a structured form, using a relational database table structure. Every event of equipment entering, leaving, being used, being cleaned, being repaired, or being scrapped is recorded in the database. Based on the complete record and combined with the contextual information provided by the external medical system, the decision support and human-computer interaction module outputs intelligent decision suggestions and real-time warnings. The human-computer interaction interface unit visually presents the device's identification results, positioning information, counting results, status assessment, and early warning information in an intuitive way.

8. The artificial intelligence-based surgical instrument recognition method according to claim 1, characterized in that, The method of visually presenting the device's identification results, positioning information, counting results, status assessment, and early warning information in an intuitive manner includes: The health scoring mechanism integrates multi-dimensional status results into a single indicator, which includes the device identification results, positioning information, counting results, status assessment, and early warning information.

9. An artificial intelligence-based surgical instrument recognition system, applied to the artificial intelligence-based surgical instrument recognition method according to any one of claims 1-8, characterized in that, include: The data perception module is used to collect multimodal raw data of surgical instruments to form a multimodal raw input dataset. The data preprocessing module is electrically connected to the data sensing module, receives the multimodal raw input dataset transmitted by the data sensing module, and performs cleaning, standardization, noise reduction, and registration operations. The multi-scale feature extraction module is electrically connected to the data preprocessing module and uses a deep learning model to extract discriminative multi-scale features from the preprocessed multimodal data. The identification and classification module is responsible for accurately identifying and classifying surgical instruments. It receives the fused feature vector output by the multi-scale feature extraction module and inputs it into the pre-trained classifier. The status assessment module provides a comprehensive evaluation of the usage status of identified instruments, covering cleanliness, integrity, and wear. The database management module is responsible for storing and managing all medical device-related data. It adopts a distributed database architecture, based on a non-relational database with columnar or key-value storage, and supports fast reading and writing and efficient retrieval of large-scale data. The system integration module provides standardized data interfaces and communication protocols for interfacing with hospital information systems, operating room management systems, and material management systems. It supports Transmission Control Protocol and User Datagram Protocol network communication methods and provides an application programming interface based on representational state transitions.

10. The artificial intelligence-based surgical instrument recognition system according to claim 9, characterized in that, The status assessment module includes: The cleanliness detection unit uses the spectral features of multispectral images that are sensitive to proteins and hemoglobin, combined with a machine learning classifier, to determine whether there are organic residues on the surface of the instrument. The integrity check unit detects whether deformation exists by comparing the real-time acquired 3D point cloud data with the 3D model of the standard instrument. The wear detection unit uses high-resolution visible light images and depth images, combined with edge detection and texture analysis algorithms, to identify minute defects and quantify the extent and severity of the defects.

Images