Real-time intraoperative surgical instrument identification system and method based on AR devices

Abstract

This invention discloses a real-time intraoperative surgical instrument recognition system and method based on AR devices, belonging to the field of medical device management technology. The system includes: an AR glasses terminal, an edge computing unit, a surgical instrument library, a text matching module, and a highlight rendering module. The AR glasses terminal includes a camera module, a built-in display, and a voice input module, for use by surgical nurses. The edge computing unit receives a video stream and performs frame-by-frame decomposition, identifying instruments in the image using a surgical instrument recognition algorithm. This invention employs a hierarchical recognition algorithm to segment instruments into multiple layers, accurately matching each independent part with the corresponding feature information in the surgical instrument library, and then combining the matching probabilities of each part to determine the final recognition result. This invention effectively solves the problem of distinguishing similar instruments, significantly reduces the recognition error rate, and provides a guarantee for the safe and smooth conduct of surgery.

Description

Technical Field

[0001] This invention relates to the field of medical device management technology, and in particular to a real-time intraoperative surgical instrument identification system and method based on AR devices. Background Technology

[0002] In the continuous development of medical technology, augmented reality (AR) technology, with its unique advantage of blending the virtual and the real, has gradually penetrated into multiple aspects of the medical field, bringing new perspectives and operating methods to medical practice. As a key part of the medical process, surgery places extremely high demands on the precise and efficient management of surgical instruments. The accurate identification and rapid transfer of surgical instruments are not only related to the smoothness of the operation, but also directly affect the patient's surgical safety and treatment effect.

[0003] In current surgical procedures, scrub nurses play a crucial role in quickly and accurately delivering surgical instruments to the surgeon according to the progress of the operation. However, the current work model heavily relies on the nurse's individual professional knowledge, memory, and long-term accumulated experience, which presents several significant problems. For example, patent CN115661509A discloses a surgical instrument identification and classification method based on a 3D point cloud ICP registration algorithm. This method utilizes point cloud segmentation and data processing algorithms to achieve precise segmentation of the 3D point cloud data of surgical instruments, obtaining a target surgical instrument point cloud set. By globally coarsely registering this set with a template point cloud and then performing fine ICP registration, high-precision identification of surgical instruments with similar appearances or different models of the same type is achieved. However, during surgery, surgical instruments are often stacked, and after 3D scanning, multiple instruments form a single 3D model that cannot be completely segmented, making it difficult to accurately match them with instruments in a 3D model library. Summary of the Invention

[0004] To address the technical problems existing in the prior art, the present invention provides a real-time intraoperative surgical instrument identification system and method based on AR devices, the technical solution of which is as follows:

[0005] On the one hand, an intraoperative surgical instrument real-time recognition system based on AR devices is provided. The system includes: an AR glasses terminal, an edge computing unit, a surgical instrument library, a text matching module, and a highlight rendering module.

[0006] The AR glasses terminal includes a camera module, a built-in display, and a voice input module, and is designed for use by scrub nurses.

[0007] The edge computing unit receives the video stream and performs frame-by-frame decomposition, and identifies the instruments in the image using a surgical instrument recognition algorithm;

[0008] The surgical instrument library stores feature information of various surgical instruments to support matching operations of the recognition algorithm;

[0009] The text matching module receives text instructions converted by the voice input module and pairs them with the instrument text results output by the recognition algorithm;

[0010] The highlight rendering module performs contour drawing and highlighting on the successfully matched target device, and transmits the processed image to the built-in display of the AR glasses to achieve virtual-real overlay.

[0011] Optionally, the AR glasses terminal includes a camera module, a built-in display, and a voice input module. The camera module is relatively fixed in the user's field of vision and synchronously collects real-time video streams of the operating table area as the user's head moves. The built-in display adopts an optical waveguide display structure to present augmented reality images. The voice input module integrates a noise-canceling microphone and bone conduction headphones. The microphone is used to collect voice signals, and the bone conduction headphones are used to output audio signals. It also has a built-in speech-to-text processing component.

[0012] Optionally, the surgical instrument library stores feature information of various surgical instruments, including partial feature data, complete feature data, and associated data. The partial feature data includes the storage of size parameters, contour features, and texture features for the grip part, working part, and connecting part of the instrument, respectively. The complete feature data includes the instrument's three-dimensional model data, standard name information, specification and model information, department information, and applicable surgical type information. The associated data includes the instrument's common combinations, usage frequency information, and sterilization expiration information.

[0013] On the other hand, a method for real-time intraoperative surgical instrument identification based on AR devices is provided, the method comprising:

[0014] S1. The nurse wears AR glasses and connects to the edge computing unit. The nurse uses voice to trigger the calibration of the camera, monitor and voice sensitivity, and loads the current surgical instrument feature data into the local cache.

[0015] S2 and AR glasses cameras move with the nurse's perspective to capture video streams from the operating table, which are then wirelessly transmitted to the edge computing unit.

[0016] S3, the edge computing unit decomposes the video stream into single images, calls the recognition algorithm to complete region segmentation, multi-layer segmentation, feature matching and comprehensive judgment, and outputs the recognition results and text;

[0017] S4. The voice input module collects nurses' instructions on the names of instruments and converts them into text instructions using a pre-trained model.

[0018] S5. The text matching module compares the text command with the recognition result text and calculates the matching degree. If the threshold is reached, the text matching module outputs a matching success signal to the highlight rendering module.

[0019] S6. The highlight rendering module generates highlight data of the successfully matched device and transmits it to the AR glasses display to achieve virtual-real overlay and guide the grasping process.

[0020] S7. If a nurse's voice is misrecognized, the system records the data; postoperative correction data is uploaded to the cloud, and the recognition model is optimized through transfer learning.

[0021] Optionally, in step S3, the edge computing unit decomposes the video stream frame by frame into individual images, which are then processed by a surgical instrument recognition algorithm. The specific steps are as follows:

[0022] S31. Perform multi-level segmentation on the equipment in the image, decomposing it into multiple independent parts;

[0023] S32. Match each segment with the segment feature information in the surgical instrument library to generate the matching probability of each segment;

[0024] S33. Based on the matching probabilities of all parts, output the device recognition result with the highest probability and the corresponding text label.

[0025] Optionally, in step S32, the formula for calculating the matching probability is: ,in, It is the first Class I medical devices Partial matching probability, It is the first Class I medical devices The feature vectors of the segmented instrument images are processed using a feature extraction network, which performs layer-by-layer convolution operations. First, 16 depthwise separable convolutional layers extract basic features, including texture, shape, and structural features. Then, 3 residual connection layers enhance key features and filter out interfering features such as intraoperative blood contamination and glare. Finally, a global average pooling layer compresses the high-dimensional features to 256 dimensions, followed by fully connected layers and L2 normalization, ultimately outputting a 128-dimensional feature vector. This ensures that the feature dimensions are consistent and distinctive. It is the first in the instrument library The standard feature vectors of the similar instruments are pre-generated and stored through a large number of samples, serving as a matching benchmark; yes and The included angle.

[0026] Optionally, in step S33, the device identification result with the highest probability is output, and its probability calculation formula is as follows: ,in, The device to be identified is the first The overall probability of such devices It is the first The weighting coefficients of each functional part, It is the first Class I medical devices The significance coefficient of some features It refers to the number of functional components that can be broken down into parts.

[0027] Optionally, the This refers to the number of functional components that can be broken down into parts, for scissor-type instruments. =4. Tweezers =3. Pliers and similar instruments =3.

[0028] Optionally, in step S5, the text matching module compares the text instruction with the recognition result text and calculates their matching degree. The formula for calculating the matching degree is: ,in, It is the text matching degree. It is text With text Edit distance, yes , It is the text of the device identification result. , It is text , The length of the characters, It is the semantic correlation coefficient of the text.

[0029] Optionally, the It is the text matching degree, when A match is considered successful when the value is ≥0.9, triggering the highlighting of the target device; if 0.7 ≤ <0.9, a list of suspected medical devices is displayed via AR glasses for nurses to select from; if <0.7 indicates that no matching device was found.

[0030] By collecting a large amount of real operating room voice data, including different accents, speaking speeds, and background noise, and combining it with similarity analysis of commonly used instrument names, a high confidence threshold of 0.9 is set to enable automatic highlighting without manual confirmation, ensuring fast and accurate instrument transfer; a low confidence threshold of 0.7 is set, and when the matching degree is in this range, the system will provide a list of suspected cases for manual selection to deal with uncertainties such as unclear pronunciation; values ​​below 0.7 are directly judged as non-matches to avoid affecting surgical safety due to mismatches.

[0031] The beneficial effects of the technical solutions provided by the embodiments of the present invention include at least the following:

[0032] I. This invention employs a hierarchical recognition algorithm to segment instruments into multiple layers. Each independent part is then precisely matched with the corresponding feature information in the surgical instrument library. Finally, the matching probabilities of each part are combined to determine the final recognition result. This effectively solves the problem of distinguishing similar instruments, significantly reduces the recognition error rate, and provides a guarantee for the safe and smooth conduct of surgery.

[0033] Second, this invention supports voice interaction, which can quickly respond to and highlight the target instrument based on the voice command of the instrument name. At the same time, through the highlighting guidance, the target instrument is clearly visible in the nurse's field of vision, and can be quickly located and grasped without additional actions, which greatly improves the smoothness of the surgical procedure and allows the nurse to focus more on the surgical cooperation. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a block diagram of an AR-based real-time intraoperative surgical instrument recognition system.

[0036] Figure 2 This is a flowchart of a method for real-time intraoperative surgical instrument identification based on AR devices;

[0037] Figure 3 It is a schematic diagram of all the instruments in the image, located using segmentation technology. Detailed Implementation

[0038] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0039] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0040] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0041] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0042] This invention provides a real-time intraoperative surgical instrument identification system and method based on AR devices, such as... Figure 1 As shown, it includes an AR glasses terminal, an edge computing unit, a surgical instrument library, a text matching module, and a highlight rendering module.

[0043] The AR glasses terminal includes a camera module, a built-in display, and a voice input module. The camera module uses a 1080P resolution infrared camera with a frame rate of 30fps, featuring autofocus and anti-fog functions. Its integrated design with the headband ensures synchronized movement with the nurse's field of vision. The built-in display utilizes waveguide technology, offering a field of view ≥40° and adjustable brightness. The voice input module is equipped with a noise-canceling microphone and bone conduction headphones, supporting voice command acquisition. The edge computing unit features a built-in NVIDIA Jetson AGX Xavier processor, supporting real-time image processing at 30 frames per second, and connects to the AR glasses wirelessly via 5G. The surgical instrument library stores 3D models and partial feature data of over 1000 commonly used surgical instruments. The highlight rendering module uses a red dashed outline with semi-transparent fill for highlighting.

[0044] By employing AR glasses, the camera moves synchronously with the nurse's perspective, ensuring full coverage of the surgical field of view, and the built-in display overlays information. The edge computing unit possesses powerful real-time processing capabilities and connects to the AR glasses via high-speed wireless connection, ensuring timely data transmission. In terms of intelligent recognition, a hierarchical recognition algorithm is used to segment instruments into multiple layers, and the comprehensive matching of features of each part improves recognition accuracy, significantly increasing the accuracy of identifying similar instruments. In terms of interaction, it supports voice query of instrument names, meeting the requirements of aseptic operation in the operating room, and has a fast response speed. The nurse only needs to issue a voice command, and the system can quickly respond and complete the location of the target instrument. In terms of highlighting guidance, the outline of the successfully matched target instrument is highlighted to intuitively indicate the position of the instrument, reducing the nurse's recognition time and improving surgical efficiency.

[0045] The system works as follows: the surgical nurse wears AR glasses and uses a camera module to capture video streams from the operating table, which are then transmitted to the edge computing unit. The edge computing unit decomposes the video stream frame by frame, calls the surgical instrument recognition algorithm to perform multi-layer segmentation and partial feature matching of the image, and outputs the recognition result based on the matching probability of each part. At the same time, the voice input module converts the nurse's instrument name instructions into text, which is then compared with the recognition result by the text matching module. If a match is successful, the highlight rendering module generates highlight data and transmits it to the AR glasses display, enabling the virtual and real superposition of the target instrument for guided grasping. The nurse can correct recognition errors through voice, and the system optimizes the model using postoperative data.

[0046] The specific implementation process includes:

[0047] 1. Preoperative preparation: Nurses wear AR glasses to complete equipment calibration, and the system loads the current surgical instrument feature data into the local cache;

[0048] 2. Intraoperative recognition: The camera captures video streams in real time and transmits them to the edge computing unit. After frame-by-frame decomposition, the instruments are divided into independent parts such as the gripping part and the working part through a multi-layer segmentation algorithm. Each part is matched with the feature data of the sub-parts in the surgical instrument library to generate matching probabilities. After comprehensive calculation, the recognition result is output.

[0049] 3. Interactive Matching: After the nurse's voice commands are converted, they are compared with the recognized text. Based on the matching degree threshold (≥0.9, direct highlighting; 0.7-0.9, list selection; <0.7, prompt not found), the corresponding interaction is triggered.

[0050] 4. Postoperative optimization: Nurses' voice correction data is recorded and uploaded to the cloud, and the recognition model is continuously optimized through transfer learning.

[0051] This system improves recognition accuracy through a hierarchical recognition architecture and solves the problem of distinguishing similar instruments by partial feature matching. Experimental data shows that the recognition error rate is reduced.

[0052] The three-level threshold matching mechanism enables precise interaction, and the voice command response delay is controlled within 0.3 seconds, improving the efficiency of nurses in grasping instruments.

[0053] The model was optimized through transfer learning, resulting in a 2.3% improvement in the system's monthly recognition accuracy, thus adapting to the rapid iteration needs of new medical devices.

[0054] The combination of bone conduction headphones and noise-canceling microphones ensures the stability of intraoperative voice interaction and has a high voice recognition accuracy in noisy environments.

[0055] Example 1

[0056] This embodiment is for laparoscopic cholecystectomy, in which the instrument nurse of this invention can quickly complete instrument identification and transfer.

[0057] refer to Figure 2 As shown in the procedure, the scrub nurse wears customized AR glasses (equipped with an anti-fog camera, bone conduction headphones, and noise-canceling microphone). The glasses frame can be wrapped in a sterile cover (meeting the sterility standards of the operating room). The system automatically connects to the edge computing unit in the operating room to complete the camera focus calibration and display parameter adjustment. At the same time, the system automatically detects the AR glasses' battery level and synchronizes it with the surgical timing system to ensure a stable power supply throughout the process.

[0058] The circulating nurse selects laparoscopic cholecystectomy on the system touchscreen. The system automatically loads a list of instruments specific to this surgery from the instrument library (including 32 instruments such as curved hemostats, straight hemostats, dissecting forceps, and titanium clips), and caches the partial characteristic data of each instrument (such as the 15° bending feature of the curved hemostat head and the 0° straight feature of the straight hemostat head) to the edge computing box. In addition, the system will also load the instrument usage sequence suggestion for this surgery (such as preparing dissecting forceps first, then preparing hemostats), and display the current suggestion to prepare: dissecting forceps in the corner of the AR glasses screen, assisting the nurse to prepare the instrument table in advance.

[0059] After the surgery began, the surgeon performed gallbladder dissection. As the nurse's gaze swept across the instrument table, the AR glasses' camera (30fps) captured the image synchronously with the field of view, transmitting one frame of image to the edge computing box every 33 milliseconds. The computing unit then processed the images.

[0060] All instruments in the image are located using segmentation technology, such as Figure 3 As shown, the overall area distinguishes different instruments such as curved hemostats and straight hemostats;

[0061] Each instrument is divided into three parts: the head (working part, i=2), the joint (connecting part, i=3), and the handle (gripping part, i=1). A 128-dimensional feature vector is extracted from each part. (e.g., the characteristic vector of the bending angle of the tip of a curved hemostat) , the bending angle feature vector of the straight hemostat head In this context, where j represents the j-th type of instrument, a lightweight feature extraction network is employed, adapted to the computing power of edge computing units. For each segmented instrument portion image (e.g., the head region of a curved hemostat, with its size normalized to 224×224 pixels to eliminate the influence of size differences), layer-by-layer convolution operations are performed. First, basic features are extracted through 16 depthwise separable convolutional layers, including texture, shape, and structural features. Then, key features are enhanced through 3 residual connection layers, including the 12° bending angle of the curved hemostat head (angle error ≤0.5° calculated using Hough transform) and the 0° straight linear feature of the straight hemostat head (linear fitting error ≤0.1°). Simultaneously, an attention mechanism filters out interfering features such as intraoperative blood contamination and surgical light reflection. Finally, a global average pooling layer compresses the high-dimensional features to 256 dimensions, followed by a fully connected layer combined with L2 normalization. The normalized feature vector has a magnitude of 1, facilitating similarity calculation, ultimately outputting a 128-dimensional feature vector. This ensures that the feature dimensions are consistent and distinctive.

[0062] The probability formula for matching single-part features Compare the standard feature vectors of the corresponding parts in the instrument library. (such as the standard bending characteristics of the head of a curved hemostat) ), calculate the matching probability of each part, such as the matching probability of the curved hemostat head. , ( and With a small included angle and a cosine value close to 1, the joint matching probability is... Handle matching probability =0.90;

[0063] Substitute into the comprehensive identification probability formula for medical devices Calculate the final probability: where the function weights are... (Holding part) (Working section, highest weight, because the pliers head directly determines the use of the instrument) (Connecting part); When distinguishing between curved and straight hemostatic forceps, the characteristic significance coefficient of the forceps head (working part) (Improving the distinguishing weight of bending features), other parts Finally, the overall probability of obtaining the curved hemostat was obtained. The overall probability of using straight hemostats The system determined that the image in the picture was a curved hemostat. In the end, the AR glasses displayed a curved hemostat with the text label "Curved Hemostat - 13cm (sterilized)" overlaid on it, and a straight hemostat labeled "Straight Hemostat - 11cm (remaining usage time 1.5h)".

[0064] When the surgeon says "hand over the curved hemostat," the nurse doesn't need to look down at the instrument table. She simply speaks into the AR glasses' microphone and says "find the curved hemostat." The voice input module instantly captures the command and converts the speech into text using a medical-specific voice-to-text model. Redundant words like "find" are automatically removed. If the nurse's pronunciation is unclear, the system will use a semantic association model to prioritize matching instruments with consistent core words to prevent the command from becoming invalid.

[0065] System extracts text instructions Curved hemostats, and the instrument text label already identified in the current image. Curved hemostat - 13cm Straight hemostat - 11cm The separation clamp is set to -15cm for matching; this is then substituted into the text matching algorithm formula. Calculate edit distance and For comparison, -13cm needs to be deleted. 1; and In comparison, straight lines need to be replaced with curved lines. However, the semantic differences are significant; calculating text length (4 characters for curved hemostat) (14cm curved hemostat, 8 characters) Assigning semantic correlation coefficients and The core terms are consistent. ; and Although they are both hemostatic forceps, they are different models. Final matching degree calculation:

[0066] ; (Mismatch detected), and verification performed. The procedure is recommended to be performed on the current surgical recommendation list, and a match is confirmed after double verification.

[0067] Once the match is successful, the system immediately triggers the highlight rendering function, drawing a red dashed outline (2px wide) of the curved hemostat in the AR glasses screen, and filling the inside of the outline with a red halo with 50% transparency. At the same time, it sends a prompt to the nurse through the bone conduction headphones that the curved hemostat has been located in the second row on the left side of the instrument table. The nurse can directly see the highlighted curved hemostat through the AR glasses, and can grab and pass it to the surgeon without having to check each one individually.

[0068] Example 2

[0069] This embodiment is for orthopedic hip replacement surgery, using the assistive device nurse of the present invention to quickly complete the identification and transfer of instruments.

[0070] The scrub nurse wears AR glasses adapted for orthopedic surgery. The glasses are equipped with an adjustable headband. After the system is connected to the edge computing box, it automatically calibrates the camera's macro parameters to adapt to the recognition of instruments at close range of 20-50cm, such as screwdrivers with a diameter of 3-5mm. At the same time, it calibrates the hand movement recognition threshold.

[0071] When the circulating nurse selects hip replacement surgery, the system loads a dedicated instrument library for this procedure, including 45 instruments such as acetabular reamers, femoral stem prosthesis implanters, screwdrivers of various sizes, and bone hammers. For the multi-size screwdrivers, including PH2 and PH3 models, which differ only in the groove shape of the screwdriver head, the system specifically stores the partial characteristics of their groove shape, i.e., PH2 groove width is 2.5mm and PH3 groove width is 3.0mm. Each size of screwdriver is assigned a unique color label (PH2 is labeled blue and PH3 is labeled green) to ensure accurate differentiation. In addition, the system will also display an instrument placement diagram on the AR glasses screen to guide the nurse to place the instruments from left to right in the order of use, reducing the time spent searching during the operation.

[0072] When the surgery enters the acetabular reaming stage, the surgeon needs to use an acetabular reamer. After the AR glasses camera captures the acetabular reamer, the computing unit first locates the irregularly shaped instrument through its shape features, and then breaks it down into three parts: the reamer head, the connecting rod, and the handle. The characteristics of each part of the acetabular reamer in the instrument library are compared: reamer head diameter 50mm, connecting rod length 10cm, and abrasive texture density of 3 lines / mm. After confirmation, a 50mm label for the acetabular reamer is superimposed on the AR glasses screen, indicating that it is used for acetabular reaming, and indicating the appropriate pressure.

[0073] When the surgeon needs a PH3 screwdriver for prosthesis fixation, the nurse says "find PH3 screwdriver." After the voice is converted to text, the system compares the text labels already recognized on the instrument table: PH2 screwdriver 15cm (blue), PH3 screwdriver 15cm (green), bone hammer 20cm. By matching the core name PH3 screwdriver with the label PH3 screwdriver 15cm, and confirming that the screwdriver head groove shape is consistent with the requirement, the system then draws a green highlighted outline of the PH3 screwdriver in the AR glasses. The outline flashes 3 times to reinforce the prompt, and an M5 screw is labeled next to the outline. The nurse can accurately grab the screwdriver within 2 seconds without needing a magnifying glass to check the screwdriver head model, thus avoiding the problem of mismatched screwdriver and screw models.

[0074] In the later stages of surgery, instruments need to be changed frequently. The system uses a camera to track the nurse's hand movements and gaze direction in real time. When the nurse puts down the screwdriver and turns her gaze to the bone hammer, the AR glasses screen automatically updates—the highlighted outline of the screwdriver disappears, and the text label "Bone Hammer 20cm" for the bone hammer automatically pops up, and the label position remains in the center of the screen as the nurse's gaze moves. If the nurse needs the bone hammer, she only needs to say "confirm bone hammer," and the system immediately highlights the bone hammer and simultaneously prompts through the bone conduction headphones that the bone hammer has been sterilized and can be used directly. The entire process requires no manual operation, meets the sterility requirements of the operating room, and reduces the burden on nurses to remember the sterilization status.

[0075] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0076] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0077] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0078] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0079] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0080] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0081] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0082] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A real-time intraoperative surgical instrument identification system based on AR devices, used for instrument identification in laparoscopic cholecystectomy, characterized in that, The system includes: an AR glasses terminal, an edge computing unit, a surgical instrument library, a text matching module, and a highlight rendering module; The AR glasses terminal includes a camera module, a built-in display, and a voice input module, and is designed for use by scrub nurses. The edge computing unit receives the video stream and performs frame-by-frame decomposition, and identifies the instruments in the image using a surgical instrument recognition algorithm; The surgical instrument library stores feature information of various surgical instruments to support matching operations of the recognition algorithm; The text matching module receives text instructions converted by the voice input module and pairs them with the instrument text results output by the recognition algorithm; The highlight rendering module performs contour drawing and highlighting on the successfully matched target device, and transmits the processed image to the built-in display of the AR glasses to achieve virtual and real superposition. The application process of the system is as follows: The system automatically connects to the edge computing unit in the operating room to complete the camera focus calibration and display parameter adjustment. At the same time, the system automatically detects the AR glasses' battery level and synchronizes it with the surgical timing system. When laparoscopic cholecystectomy is selected on the system touch screen, the system automatically loads a list of instruments specific to this surgery from the instrument library: containing 32 kinds of surgical forceps instruments, and caches the partial feature data of each instrument to the edge computing box. In addition, the system will also load the instrument usage sequence suggestions for this surgery, and display the current suggested preparation in the corner of the AR glasses screen: dissection forceps, and the assistant nurse should prepare the instrument table in advance. As the gaze sweeps across the instrument table, the AR glasses' camera captures the image synchronously with the viewing angle, transmitting one frame of image to the edge computing box every 33 milliseconds. The computing unit then processes the image. All instruments in the image are located using segmentation technology, and the overall areas of different instruments are distinguished. Each instrument is divided into three parts: the head, the joint, and the handle. The working part of the head is represented by i=2, the joint connecting part by i=3, and the handle gripping part by i=1. A 128-dimensional feature vector is extracted from each part. Where j represents the j-th type of instrument, specifically: a lightweight feature extraction network is adopted, which is adapted to the computing power of edge computing units, and performs layer-by-layer convolution operations on each segmented instrument image; firstly, basic features are extracted through 16 depthwise separable convolutional layers, including texture features, shape features, and structural features; then, key features are enhanced through 3 residual connection layers, and the angle error is calculated to be ≤0.5° through Hough transform; at the same time, attention mechanism is used to filter out features such as intraoperative blood contamination and surgical light reflection interference; finally, the high-dimensional features are compressed to 256 dimensions through global average pooling layers, and then processed by fully connected layers combined with L2 normalization. After normalization, the feature vector magnitude is 1, which facilitates similarity calculation, and finally outputs a 128-dimensional feature vector. This ensures that the feature dimensions are consistent and distinctive. The probability formula for matching single-part features ,in, It is the first Class I medical devices Partial matching probability, It is the first Class I medical devices Partial feature vectors, It is the first in the instrument library Standard feature vectors of similar instruments yes and The included angle is compared with the standard feature vector of the corresponding part in the instrument library. Calculate the matching probability for each part; Substitute into the comprehensive identification probability formula for medical devices Calculate the final probability: where, Functional weights for the gripping part; The working part has the highest weight, as the pliers head directly determines the purpose of the instrument. The weight of the connecting part; The distinguishing weight of the forceps head, when distinguishing between curved and straight hemostats, is determined by the significance coefficient of the characteristic of the working part of the forceps head. That is, the distinguishing weight of the enhanced bending feature, and other parts. The final probability of obtaining the curved hemostat is... The overall probability of using straight hemostats The system determines whether the forceps in the image are curved or straight by taking the larger of the two values. Finally, the AR glasses display showed the curved hemostat with the text label "Curved Hemostat" overlaid on it, and the straight hemostat was labeled "Straight Hemostat - Remaining Usage Time". When the surgeon says "hand over the curved hemostat," the nurse simply speaks into the AR glasses microphone and says "find the curved hemostat." The voice input module instantly captures the command and converts the speech into text using a medical-specific speech-to-text model. It also automatically removes redundant words such as "find" and "search." If the nurse's pronunciation is unclear, the system will use a semantic association model to prioritize matching instruments with consistent core words to avoid command failure. System extracts text instructions Curved hemostats, and the instrument text label already identified in the current image. Perform the matching and substitute it into the text matching algorithm formula. Calculate the edit distance between text X and text Y. Where M is the text matching degree, X is the speech-to-text command, and Y is the device recognition result text. The lengths of texts X and Y are the number of characters. It is the semantic correlation coefficient of the text; when A match is considered successful when the value is ≥0.9, triggering the highlighting of the target device; if 0.7 ≤ <0.9, a list of suspected medical devices is displayed via AR glasses for nurses to select from; if <0.7 indicates no matching device found; Once the match is successful, the system immediately triggers the highlight rendering function, drawing a red dotted outline of the curved hemostat in the AR glasses screen and filling the inside of the outline with a red halo with 50% transparency. At the same time, it sends a prompt to the nurse through the bone conduction headphones that the curved hemostat has been located in the second row on the left side of the instrument table. The nurse can directly see the highlighted curved hemostat through the AR glasses and can grab and pass it to the surgeon without having to check each one individually.

2. The real-time intraoperative surgical instrument identification system based on AR equipment according to claim 1, characterized in that, The AR glasses terminal includes a camera module, a built-in display, and a voice input module. The camera module is relatively fixed in the user's field of vision and synchronously captures real-time video streams of the operating table area as the user's head moves. The built-in display adopts an optical waveguide display structure to present augmented reality images. The voice input module integrates a noise-canceling microphone and bone conduction headphones. The microphone is used to collect voice signals, and the bone conduction headphones are used to output audio signals. It also has a built-in speech-to-text processing component.

3. The real-time intraoperative surgical instrument identification system based on AR equipment according to claim 1, characterized in that, The surgical instrument library stores feature information on various surgical instruments, including partial feature data, complete feature data, and associated data. The partial feature data includes the storage of dimensional parameters, contour features, and texture features for the grip, working part, and connecting part of the instrument, respectively. The complete feature data includes the instrument's three-dimensional model data, standard name information, specification and model information, department information, and applicable surgical type information. The associated data includes the instrument's common combinations, usage frequency information, and sterilization expiration information.

4. A method for real-time intraoperative surgical instrument identification based on an AR device, the method being applicable to the real-time intraoperative surgical instrument identification system based on an AR device as described in any one of claims 1-3, characterized in that, The method includes: S1. The nurse wears AR glasses and connects to the edge computing unit. The nurse uses voice to trigger the calibration of the camera, monitor and voice sensitivity, and loads the current surgical instrument feature data into the local cache. S2 and AR glasses cameras move with the nurse's perspective to capture video streams from the operating table, which are then wirelessly transmitted to the edge computing unit. S3, the edge computing unit decomposes the video stream into single images, calls the recognition algorithm to complete region segmentation, multi-layer segmentation, feature matching and comprehensive judgment, and outputs the recognition results and text; S4. The voice input module collects nurses' instructions on the names of instruments and converts them into text instructions using a pre-trained model. S5. The text matching module compares the text command with the recognition result text and calculates the matching degree. If the threshold is reached, the text matching module outputs a matching success signal to the highlight rendering module. S6. The highlight rendering module generates highlight data of the successfully matched device and transmits it to the AR glasses display to achieve virtual-real overlay and guide the grasping process. S7. If a nurse's voice is misrecognized, the system records the data; postoperative correction data is uploaded to the cloud, and the recognition model is optimized through transfer learning.

5. The method for real-time intraoperative surgical instrument identification based on AR devices according to claim 4, characterized in that, In step S3, the edge computing unit decomposes the video stream frame by frame into individual images, which are then processed using a surgical instrument recognition algorithm. The specific steps are as follows: S31. Perform multi-level segmentation on the equipment in the image, decomposing it into multiple independent parts; S32. Match each segment with the segment feature information in the surgical instrument library to generate the matching probability of each segment; S33. Based on the matching probabilities of all parts, output the device recognition result with the highest probability and the corresponding text label.

Images