Laparoscopic instrument recognition and tracking method with visual perception enhancement and global attention

Abstract

The application discloses a laparoscope instrument recognition and tracking method with visual perception enhancement and global attention. The method collects a surgical scene image in real time through a laparoscope camera, and adopts an innovative visual perception enhancement and global attention module to pre-process the input image, thereby significantly improving the image quality and feature expression capability. Known surgical instruments are recognized and positioned by using an improved high-efficiency target detection WF-YOLO model. For unknown surgical instruments, a FastSAM model is used for interactive segmentation and template extraction, and then ORB feature extraction and a FLANN matcher are used to realize fast matching and positioning. The system generates a control instruction according to the recognition result, drives a laparoscope mechanical arm to adjust the pose in real time, and ensures that the surgical instrument is always in the center of the picture, thereby realizing stable tracking. Through effective fusion of perception enhancement, global attention and a detection model, the application solves the problems of insufficient recognition of unknown instruments and poor real-time performance in the prior art, and improves the precision and safety of surgical operation.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and computer vision technology. More specifically, this invention relates to a method for laparoscopic instrument identification and tracking with enhanced visual perception and global attention. Background Technology

[0002] The widespread adoption of laparoscopic technology has propelled surgical procedures towards minimally invasive and precise techniques, making the need for dynamic instrument tracking and stable visual field during surgery increasingly crucial. Currently, in clinical practice, the adjustment of the laparoscopic lens primarily relies on manual operation by assistants, who must constantly follow the movement of surgical instruments to ensure they remain in the frame. This not only increases the collaborative burden on medical staff but also carries the risk of instruments slipping out of the field of vision due to delayed reactions or operational deviations, affecting surgical precision and safety. The limitations of this manual tracking are particularly pronounced in prolonged surgeries or complex procedures.

[0003] Existing laparoscopic surgical instrument tracking technologies have significant limitations: traditional robotic arm-assisted systems can only move along a preset trajectory and cannot dynamically adjust according to the real-time position of the instrument, resulting in insufficient flexibility; although deep learning-based automatic tracking models can identify known instruments and drive the robotic arm to follow them, they are only applicable to instrument types in the training dataset and are powerless against new or unlabeled instruments. Meanwhile, the increasing use of new surgical instruments in clinical practice limits the versatility of the system.

[0004] In the tracking of unknown instruments, existing technologies often require manual marking of the initial position of the instruments, frequently interrupting the surgical procedure and affecting the operational rhythm. Some algorithms attempt to track unknown instruments using feature matching techniques such as SIFT and SURF, but these algorithms are computationally intensive and have slow single-frame processing times, making it difficult to meet the real-time requirements of laparoscopic surgery. Furthermore, features are easily lost or mismatched in scenarios such as instrument rotation, scaling, and blood occlusion, leading to tracking interruptions. In addition, most algorithms are not optimized for the hardware environment of surgical scenarios, making it difficult to run efficiently on the embedded control unit of the laparoscopic robotic arm, thus hindering clinical translation and application.

[0005] Existing technologies, such as the SH-YOLO network model (which includes a feature extraction network, a feature fusion network, and a detection head connected in sequence), can detect laparoscopic instruments in real time. However, its image preprocessing relies on traditional enhancement techniques (such as simple contrast adjustment), without considering the human visual perception mechanism. This results in insufficient feature expression in complex lighting and noise environments, and the lack of global context modeling makes it susceptible to local interference. Summary of the Invention

[0006] One object of the present invention is to solve at least the above-mentioned problems and / or defects, and to provide at least the advantages described below.

[0007] To achieve these objectives and other advantages of the present invention, a method for laparoscopic instrument recognition with enhanced visual perception and global attention is provided, comprising: S1. Construct a WF-YOLO model for recognizing known instruments and an adaptive recognition module for recognizing unknown instruments on the control unit. The front end of the WF-YOLO model integrates a visual perception enhancement layer based on Weber-Fechner's law and a global attention encoding layer based on the Transformer encoder structure. S2. The control unit acquires raw images of the surgical scene in real time through the laparoscopic camera; The S3 and WF-YOLO models obtain the corresponding instrument features after preprocessing the original images; S4. The WF-YOLO model matches the instrument features with a pre-stored dataset of known instruments. If the match is successful, bounding box I is generated to complete the identification and localization of the known surgical instruments; otherwise, proceed to S5. S5. For unknown instruments, the adaptive recognition module performs corresponding segmentation, feature extraction, and matching operations on the instrument features, and adaptively generates the corresponding bounding box II to complete the on-site identification and positioning of unknown surgical instruments.

[0008] Preferably, in S1, the network architecture of the WF-YOLO model is as follows: a WF layer is integrated as a front-end preprocessing module on the basis of the real-time object detection model YOLOv8, and a Transformer encoder is embedded in the feature fusion network of YOLOv8. The WF Layer is directly connected to the CSPDarknet backbone input layer of YOLOv8 to achieve synchronous parameter updates through gradient backpropagation.

[0009] Preferably, in S1, the adaptive recognition module includes: The FastSAM model combines user interaction boxes with the prompt encoder of SAM for semi-supervised real-time segmentation. The ORB submodule performs feature extraction on the template segmented by the FastSAM model and the original image, respectively. The FLANN submodule performs feature matching on the binary feature descriptors extracted by the ORB module.

[0010] Preferably, the FastSAM model works as follows: In the original image displayed on the interface, the user operator receives interactive instructions through the operation interface, and selects the area where the unknown surgical instrument to be segmented is located through the interactive box in the interactive instructions. The cue encoder in the FastSAM model converts the coordinates of the selected region into an embedding vector that can be fused with the features of the image encoder to generate a segmentation template through the mask decoder.

[0011] Preferably, the ORB submodule works as follows: An improved FAST corner detection algorithm is used to analyze the segmentation template output by the FastSAM model and the key points in the overall image. The key points refer to pixels in the image whose grayscale or gradient changes drastically in local areas and can be applied to subsequent matching and recognition. The orientation-aware BRIEF algorithm generates corresponding 32-dimensional binary feature descriptors for key points.

[0012] Preferably, the FLANN submodule operates as follows: Normalize the feature vector dimension of the binary feature descriptors extracted from the ORB submodule; A multi-level kd-tree index is constructed based on the normalized feature points and the levels are divided. The fast retrieval of near nearest neighbors is achieved by performing multi-tree parallel search within the search radius. The search results are filtered based on a preset distance threshold; The RANSAC algorithm performs geometric verification matching between the screening results and the pre-stored unknown instrument template library. If the match is successful, a bounding box II related to the instrument's location is generated; otherwise, the screening results are marked as new unknown instruments, and the operator is prompted to intervene through the user interface.

[0013] Preferably, in S5, the unknown device is determined to have been successfully identified by the already generated bounding box II, and the corresponding information of the unknown device is added to the known device dataset.

[0014] Preferably, in S3 and S4, the visual perception enhancement layer introduces multiple training parameters based on the Weber-Fechner theorem to perform dynamic nonlinear transformation on the input original image or instrument features to obtain the enhanced feature I; The training parameters include: a first parameter γ for setting the feature response baseline threshold, a second parameter α for controlling the sensitivity of feature changes, and a third parameter β for adjusting the nonlinear compression strength; The global attention encoding layer embeds positional encoding and residual connections, and uses a Transformer encoder structure to model the global context relationship of feature I in order to capture long-range dependency information in the original image or instrument features.

[0015] A laparoscopic instrument tracking method, which employs the aforementioned visual perception enhancement and global attention laparoscopic instrument recognition method, further includes: S6. Calculate the deviation value (Δx, Δy) between the instrument center and the image center based on bounding box I or bounding box II. Transform the deviation value (Δx, Δy) into control commands for a 3-DOF laparoscopic robotic arm. After receiving the control commands, the robotic arm adjusts the position of the laparoscopic lens by driving joint movement through servo motors, moves the identified surgical instruments to the center of the image, and maintains tracking status throughout the subsequent movement of the surgical instruments.

[0016] A laparoscopic instrument identification and tracking system, which adopts the surgical instrument tracking method described above, includes a laparoscopic camera, a laparoscopic robotic arm, and a user operating terminal, and further includes a control unit that is communicatively connected to the laparoscopic camera, the laparoscopic robotic arm, and the user operating terminal. The control unit is configured to use a rk3588 chip with a built-in neural network processing unit (NPU).

[0017] The present invention has at least the following beneficial effects: Firstly, this invention enables accurate identification, dynamic tracking, and automatic robotic arm following of known and unknown instruments during laparoscopic surgery. Through collaborative identification using the WF-YOLO and FastSAM models, feature matching with ORB+FLANN, and full-module NPU acceleration, it meets the high requirements of real-time performance and robustness in surgical scenarios. Specifically, this invention uses the WF-YOLO model to quickly identify known instruments and the FastSAM model to segment unknown instruments, addressing the problem of insufficient coverage of new instruments by traditional algorithms. The dynamic robotic arm following mechanism ensures that critical surgical areas (instrument operation ends) are always clearly visible, avoiding misoperations caused by field of view shifts. Simultaneously, the flexible intervention function of the user interface allows medical staff to fine-tune the field of view according to surgical needs, balancing automation efficiency with the flexibility of manual operation, effectively preventing surgical risks caused by lost field of view, and ensuring the continuity and accuracy of surgical procedures.

[0018] Secondly, the surgical instrument identification and tracking system provided by this invention, through the coordinated operation of the RK3588 chip, the laparoscopic robotic arm, and the user interface, can output instrument position information and robotic arm control commands in real time, avoiding frequent manual adjustments of the camera by medical staff. Furthermore, this invention does not rely on high-performance external computing devices; it achieves full-process acceleration through the NPU of the RK3588 chip, adapting to the embedded hardware environment of laparoscopic surgery and facilitating deployment in clinical scenarios. Simultaneously, the algorithm is highly adaptable to complex situations such as instrument rotation, scaling, and occlusion, covering various types of laparoscopic surgery, reducing algorithm debugging costs in different surgical scenarios, and providing technical support for the standardization and precision development of minimally invasive surgery.

[0019] Other advantages, objectives and features of the present invention will become apparent in part from the following description, and in part from those skilled in the art through study and practice of the invention. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall process of the real-time identification and tracking algorithm for laparoscopic surgical instruments based on enhanced visual perception and global attention in this invention. Figure 2 This is a network structure diagram of WF-YOLO in this invention; Figure 3 This is a schematic diagram of the classification and processing flow of known and unknown instruments in an embodiment of the present invention; Figure 4 This is a schematic diagram of the dynamic following control process of the robotic arm in an embodiment of the present invention; Figure 5 This is a flowchart illustrating the user interface interaction function in an embodiment of the present invention; Figure 6 This is a schematic diagram of the overall architecture of the surgical instrument identification and tracking system in an embodiment of the present invention. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the accompanying drawings, so that those skilled in the art can implement it based on the description.

[0022] A laparoscopic surgical instrument recognition and tracking algorithm based on enhanced visual perception and global attention. The core function of this algorithm is to identify and dynamically locate known and unknown instruments in the surgical scene in real time, and generate hexadecimal control commands to drive the robotic arm to adjust the position of the laparoscopic lens so that the instruments are always kept in the center of the image. The specific execution steps of the algorithm include: S1. Construct a WF-YOLO model for recognizing known instruments and an adaptive recognition module for recognizing unknown instruments on the control unit. The WF-YOLO model integrates a visual perception enhancement layer based on the Weber-Fechner theorem and a global attention encoding layer based on the Transformer encoder structure at its front end. The visual perception enhancement layer is used to enhance contrast and suppress noise in the input image or features based on the Weber-Fechner theorem. It performs dynamic nonlinear transformation on the input image or features through multiple trainable parameters. The parameters include a first parameter for setting the baseline threshold of the feature response, a second parameter for controlling the sensitivity of feature changes, and a third parameter for adjusting the nonlinear compression intensity. This layer achieves contrast enhancement and background noise suppression through the dynamic nonlinear transformation. The global attention encoding layer is used to model the global context relationship of the enhanced features using the Transformer encoder structure to capture long-range dependency information in the image.

[0023] S2. The WF-YOLO model acquires raw images of the surgical scene in real time using a laparoscopic camera, and uses these raw images as input. Building upon the existing single-stage detection architecture, the WF-YOLO model introduces an attention mechanism and a feature fusion module to enhance its detection capabilities for small-sized instruments and partially occluded instruments. Furthermore, a multi-task learning mechanism simultaneously optimizes classification and regression tasks, improving the model's generalization performance and robustness in real surgical scenarios. Simultaneously, the control unit (rk3588 chip) pre-stores a dataset of known surgical instruments. This dataset is obtained by annotating and training a large number of images of common laparoscopic surgical instruments (such as hemostats and needle holders), containing multi-dimensional feature information such as the instrument's appearance and size, to support the WF-YOLO model in achieving high-precision recognition.

[0024] S3. An innovative WF-YOLO model is used to preprocess the input image to significantly improve image quality and enhance its feature representation capabilities; for example... Figure 2As shown, this WF-YOLO network structure innovatively integrates an enhancement layer (WF Layer) based on the Weber-Fechner theorem into the input of the YOLOv8 architecture, forming an end-to-end optimized detection process. This enhancement layer adjusts the contrast of the input image and suppresses high-frequency noise using three learnable parameters (γ, α, β), simulating the human eye's visual perception mechanism, thus providing the subsequent detection network with more vivid and higher-quality image data. The preprocessed data then enters the YOLOv8 backbone network for multi-scale feature extraction, and is further fused through the neck network, fully aggregating semantic and positional information from different levels. This entire structure enables the WF layer and the detection network to be jointly trained, significantly improving the model's generalization ability, robustness, and detail perception in complex scenarios such as medical images.

[0025] In this step, the innovation of WF-YOLO compared to existing network models (such as SH-YOLO) lies in the following: SH-YOLO processes images using only a simple feature extraction network, while WF-YOLO innovatively integrates a WF Layer as a front-end preprocessing module based on YOLOv8. This integration method adopts an end-to-end training mechanism, jointly optimizing the parameters (γ, α, β) of the WF Layer with the parameters of the YOLOv8 backbone network, avoiding feature loss caused by independent steps in traditional preprocessing (such as simple contrast enhancement). Specifically, the WF Layer is directly connected to the input layer of the YOLOv8 CSPDarknet backbone, achieving synchronous parameter updates through gradient backpropagation. Furthermore, WF-YOLO introduces a global attention module, embedding a Transformer encoder in the feature fusion network to capture long-range dependencies, while SH-YOLO's feature fusion is limited to local convolution operations, making it susceptible to noise interference in surgical scenes. This innovation is not a simple fusion of existing technologies, but rather utilizes the Weber-Fechner theorem formula. In this formula, x and y represent the input and mapped CT voxel values, respectively. Simulating the nonlinear response of the human eye, dynamic enhancement is achieved, significantly improving detection accuracy in blood-occluded scenarios. Simultaneously, the formula uses γ as the stimulation threshold to determine the feature response baseline, α to control the sensitivity of feature changes, and β to adjust the nonlinear compression intensity. These three elements together form a dynamic gating mechanism to enhance, suppress, or compress input features. Compared to other traditional enhancement methods, its parameter self-learning and joint optimization with the lower network avoid the static limitations of traditional enhancements. The global attention encoding layer uses a Transformer encoder to model global relationships; compared to standard attention, this layer embeds positional encoding and residual connections to optimize the surgical scene occlusion problem.

[0026] Meanwhile, to further enhance the model's ability to perceive surgical instruments of different sizes, this invention integrates a multi-scale pyramid pooling module (PPM) after the global attention encoding layer. The calculation formula is as follows: Where P is the feature map output by the global attention coding layer; AdaptiveAvgPool 3 D ( ) indicates the implementation of adaptive 3D average pooling operation, pooling window p i Take values ​​of (1×1), (2×2), (4×4), and (8×8) respectively; f_i is the multi-scale feature map of the i-th level output. f i After 1×1 convolution to reduce the dimensionality to 1 / 4 of the original number of channels, it is then upsampled to restore the same spatial resolution as P. Finally, it is concatenated with the original feature P in the channel dimension to form a multi-scale fused feature.

[0027] Furthermore, to address the class imbalance problem in laparoscopic images, the WF-YOLO model employs Tversky Loss as its core formula: Where TP, FN, and FP represent true positive, false negative, and false positive, respectively; α and β are the corresponding weighting parameters (in practical applications, α=0.3 and β=0.7 can be set to emphasize the penalty for false negatives); and ε is the smoothing term (to avoid division by zero).

[0028] Furthermore, a deep supervised loss (DS Loss) is introduced to achieve multi-level optimization, the formula of which is: In the above formula, X_ i For the output of the i-th layer, Y For ground truth, n For the number of floors, α For dynamic decay weights, TverskyLoss ( ) represents the segmentation loss function, which improves model generalization and increases the accuracy of instrument segmentation through deep supervision.

[0029] S3. Input the enhanced image into the RK3588 chip so that the WF-YOLO model on the RK3588 chip can analyze the input image, determine whether the device is of a known type, and select the corresponding branch based on the determination result: Branch 1: If the judgment result is a known instrument, the WF-YOLO model deployed on the rk3588 chip is called to perform inference on the image, extract instrument features and match them with the pre-stored known instrument dataset; if the match is successful, bounding box I is generated to realize the rapid matching and localization of the known instrument, and the bounding box data is sent to the laparoscopic robotic arm control module. Branch 2: If the result indicates an unknown instrument, the process directly enters the adaptive recognition module (which includes the FastSAM model, ORB submodule, and FLANN submodule) for segmentation, feature extraction, and matching. Specifically, the operator selects the unknown surgical instrument in the current image using the user interface. After selection, the FastSAM model interactively segments the selected area, generating a surgical instrument template, which is then added. The ORB submodule extracts feature points from both the template and the overall image, and the FLANN matcher matches these features. If a match is successful, the unknown instrument information is recorded, and a bounding box II is generated to achieve rapid matching and localization of the unknown instrument. (Simultaneously, the generation of bounding box II confirms successful identification of the unknown instrument, and its information is added to the known instrument dataset for subsequent instrument identification, expanding the known instrument dataset.) This information is then sent to the robotic arm control module. If a match fails, the instrument is marked as a new unknown instrument, and the user interface is prompted for manual intervention. By fusing the identification results of known instruments with those of successfully matched unknown instruments, overall instrument positioning information is generated. The issue of manual labeling is addressed through automated FastSAM interactive segmentation, reducing interruptions. The computationally intensive and real-time-efficient SIFT / SURF approach is resolved using ORB+FLANN for efficient matching, reducing single-frame time to milliseconds. Feature loss / mismatch issues are mitigated through WF-YOLO's perceptual enhancement and global attention to improve robustness. Insufficient SH-YOLO preprocessing is addressed through dynamic optimization via WF Layer integration.

[0030] The interactive segmentation of the FastSAM model differs from existing technologies that rely on fully automatic segmentation (such as U-Net). The FastSAM model's interactive segmentation process works as follows: After the user selects an unknown device on the interface, FastSAM's prompt encoder converts the bounding box coordinates into embedding vectors, which are then fused with the image encoder features. A segmentation mask is generated through a mask decoder. The innovation lies in introducing a dynamic prompt optimization mechanism, and this process is unattended, filling the gap in real-time extraction of unknown devices. For the segmented unknown device templates, the user interface management function supports saving and naming them. Later, they can be added to a known device dataset through manual review or an automatic training module, achieving a transfer from unknown to known and reducing repetitive segmentation operations.

[0031] The ORB submodule provides accurate templates during feature extraction, effectively addressing interference between instruments and surrounding tissues or other instruments in surgical scenarios. Specifically, when extracting features from the template and the overall image, the ORB submodule employs an improved FAST corner detection algorithm to extract key points (e.g., pixels in the image with significant and unique local grayscale or gradient changes, suitable for subsequent matching and recognition, are used as key points for feature extraction), and generates 32... The ORB algorithm, using binary descriptors, offers significant advantages over other feature extraction algorithms such as SIFT and SURF. ORB extracts features through keypoint orientation estimation and scale pyramid construction, maintaining a feature matching accuracy above 90% even with 180-degree instrument rotation. It adapts to changes in instrument posture caused by manipulation during surgery. Compared to existing SIFT and SURF algorithms, the ORB submodule used in this branch demonstrates significant efficiency advantages while meeting the demands of surgical scenarios. The ORB submodule's feature extraction speed is over 100 times faster than SIFT and over 50 times faster than SURF, achieving second-level feature extraction for a single frame, and is unrestricted by patents. While SIFT offers superior scale invariance, its high computational complexity makes it difficult to meet the real-time requirements of laparoscopic surgery. Although SURF is faster than SIFT, it still faces patent restrictions and its feature stability is inferior to the ORB submodule in reflective instrument environments. Furthermore, the ORB submodule exhibits stronger anti-interference capabilities compared to other feature extraction algorithms, ensuring accuracy in complex surgical environments.

[0032] The FLANN submodule (also known as the FLANN matcher) employs a hierarchical indexing architecture for feature matching: it constructs multi-level kd-tree indexes from the binary feature descriptors extracted from the ORB, achieving fast near-nearest neighbor retrieval through a search radius strategy. The specific process involves feature vector dimension normalization, followed by kd-tree hierarchical partitioning, multi-tree parallel search, distance threshold filtering, and finally RANSAC geometric verification. Compared to BF (brute-force matching) and KNN (k-nearest neighbor matching), the FLANN submodule's technical advantages are: (1) BF uses traversal matching (time complexity O(n²)), KNN uses linear scanning + sorting (time complexity O(n²logn)), while FLANN reduces the complexity to O(nlogn) through index tree; in surgical scenarios, when the number of feature points in a single frame is 500-1000, FLANN matching takes less time and is significantly improved compared to BF and KNN, meeting the hard requirements of laparoscopic surgery; (2) When the instrument feature points in the surgical image are dense (including redundant features caused by reflection and occlusion), when the number of feature points exceeds 1000, BF and KNN will experience obvious lag, while FLANN controls the increase in matching time within a linear range by dynamically adjusting the index tree depth. (3) The NPU of the rk3588 chip can accelerate the index building process of FLANN, while BF and KNN are difficult to utilize hardware parallel computing resources due to the strong seriality of their algorithms, and the performance gap is further widened in embedded scenarios.

[0033] S4. The system generates control commands based on the bounding boxes of identified known or unknown instruments, driving the laparoscopic robotic arm to adjust its position and angle in real time. This ensures that the identified surgical instruments remain centered in the view, and the robotic arm continuously tracks them during subsequent instrument movement. This provides medical staff with a stable surgical field of view, effectively improving the convenience and safety of surgical procedures, and achieving a high degree of integration between technical practicality, clinical safety, and operational efficiency.

[0034] This step is designed for a robotic arm with at least three degrees of freedom. It uses hexadecimal instructions to control the laparoscopic robotic arm. The position of the robotic arm is adjusted based on the bounding box coordinates output by the RK3588 chip, ensuring that the identified surgical instruments are always centered in the image. Control instructions are continuously updated during subsequent instrument movements to guarantee the accuracy of the surgical procedure. Specifically, this step calculates the deviation (Δx, Δy) between the instrument center and the image center in real time based on the bounding box of known or unknown instruments, and converts it into control instructions for the three-degree-of-freedom laparoscopic robotic arm (laparoscopic left-right yaw motion, laparoscopic up-down yaw motion, and laparoscopic zoom): the horizontal deviation Δx corresponds to the rotation axis compensation angle, and the vertical deviation Δy corresponds to the pitch axis compensation angle. The control parameters are encoded into hexadecimal instructions and sent to the robotic arm controller via a UART interface (9600bps baud rate). After receiving the instructions, the robotic arm uses high-precision servo motors to drive joint movements, adjusting the position of the laparoscopic lens to keep the instruments centered in the image. The tracking response time is fast, and the position accuracy reaches the second level.

[0035] A laparoscopic surgical instrument recognition and tracking system based on enhanced visual perception and global attention, such as Figure 6As shown, the aforementioned laparoscopic surgical instrument recognition and tracking method based on enhanced visual perception and global attention operates on a hardware platform equipped with a control unit (i.e., the RK3588 chip with an integrated neural network processing unit (NPU)). It works in conjunction with a laparoscopic camera, a 3-DOF laparoscopic robotic arm, and a user interface (also known as the user control terminal) to achieve its functions. Specifically, the RK3588 chip connects to the laparoscopic camera and other components via an information interaction module to achieve data exchange. The user interface displays the entire process of instrument recognition and tracking for real-time viewing by medical personnel. Specifically, the RK3588 chip connects to the laparoscopic camera to receive acquired surgical images; it connects to the laparoscopic robotic arm to send control commands; and it connects to the user interface to achieve data interaction such as command transmission and status display. The RK3588 chip serves as the core, with a built-in NPU (Neural Processing Unit) that boasts powerful computing capabilities. It accelerates algorithms such as WF-YOLO model inference, FastSAM model segmentation, ORB feature extraction, and FLANN matching, ensuring the real-time performance and accuracy of device recognition. Simultaneously, it integrates algorithm calculations and data interaction results to coordinate and control the overall system process, including robotic arm dynamic following and user interface interaction, ensuring that all modules work collaboratively.

[0036] Furthermore, when the algorithms involved in each module run on the RK3588 chip, they are accelerated by utilizing the high-performance Neural Processing Unit (NPU) integrated on the RK3588 chip. Specifically, the NPU is used to accelerate the inference operations of neural network models such as WF-YOLO and FastSAM, as well as the operation of graph-related algorithms such as feature extraction in the ORB submodule and keypoint matching in the FLANN submodule. This ensures the real-time performance of the algorithms in laparoscopic surgery scenarios, meeting the basic requirements of laparoscopic surgery. Specifically, the acceleration of each algorithm using the 6 TOPS computing power of the RK3588 chip's NPU mainly includes: (1) Neural network acceleration: The backbone network inference of the WF-YOLO model and the segmentation head operation of the visual basic model are mapped to multiple computing cores of the NPU for parallel processing, which significantly shortens the computation time; (2) Feature extraction and matching acceleration: The corner detection of ORB feature extraction is transformed into NPU binary convolution operation, and the kd-tree construction and distance calculation of FLANN matching are accelerated by NPU vector operation unit, which effectively improves the efficiency of single-frame feature processing; (3) Cooperative scheduling: The task is scheduled using a dynamic priority queue (neural network inference is the highest priority and feature processing is the second highest priority) to maintain the high utilization of the NPU and ensure that the total processing time of a single frame meets the real-time requirements.

[0037] Furthermore, the RK3588 chip establishes a two-way communication mechanism with the user interface through an information interaction module. Its core objective is to provide medical staff with convenient and intuitive operational support, assisting in the efficient conduct of surgeries. Specifically, the RK3588 chip dynamically transmits surgical scene images with overlaid instrument bounding boxes to the interface via the information interaction module, simultaneously displaying the recognition results and completing real-time information display, allowing medical staff to monitor instrument positioning and system operation in real time. In addition, the user interface helps medical staff achieve flexible operation control; the laparoscopic surgical instrument recognition and tracking system supports switching between "WF-YOLO mode" (for rapid recognition of known instruments) and "ORB mode" (for enhanced matching of unknown instruments) to adapt to different surgical scenario requirements, receiving and responding to operation commands input from the interface. like Figure 5 As shown, the user interface built on the RK3588 chip receives operation commands from medical staff, such as switching recognition modes (WF-YOLO model priority, ORB priority), fine-tuning the current position of the robotic arm, and managing unknown instrument templates (saving, deleting), etc. The RK3588 chip processes these commands and feeds back real-time images (overlaid with instrument bounding boxes), recognition results, and robotic arm status to the user interface for display, realizing a flexible combination of algorithmic automation and human intervention, including the following steps: First, the user interface based on the RK3588 chip receives operation instructions from medical staff. The instruction types include recognition mode switching, robotic arm fine-tuning, and unknown instrument template management (saving, deleting), etc.

[0038] Secondly, the RK3588 chip processes the received instructions and feeds back information such as real-time images (overlaid with the instrument bounding box), recognition results, and robotic arm status to the user interface.

[0039] Finally, the system displays the received real-time images, recognition results, robotic arm status, and other information for medical staff to view and reference.

[0040] In the laparoscopic surgical instrument identification and tracking system, medical staff can fine-tune the angle and position of the laparoscopic robotic arm: it provides adjustment buttons for left and right tilting of the endoscope, up and down tilting of the endoscope, and extension and retraction of the endoscope, so that medical staff can manually correct the field of vision according to their operating habits and ensure that key areas are clearly presented. The laparoscopic surgical instrument identification and tracking system has an unknown instrument management function, which supports saving, naming or deleting the segmented unknown instrument templates, so that they can be directly called in subsequent surgeries and reduce repetitive operations.

[0041] Through the above interactive design, the high efficiency of the algorithm's automated tracking is preserved, while giving medical staff the authority to intervene flexibly. This effectively reduces the complexity of operation, allowing medical staff to focus more on the core surgical procedures and improve the smoothness and safety of the surgery.

[0042] The above solution is merely an illustration of a preferred example and is not limited thereto. When implementing this invention, appropriate substitutions and / or modifications can be made according to the user's needs.

[0043] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. It can be applied to various fields suitable for the present invention. Other modifications can be readily made by those skilled in the art. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and examples shown and described herein.

Claims

1. A method for laparoscopic instrument recognition based on enhanced visual perception and global attention, characterized in that, include: S1. Construct a WF-YOLO model for recognizing known instruments and an adaptive recognition module for recognizing unknown instruments on the control unit. The front end of the WF-YOLO model integrates a visual perception enhancement layer based on Weber-Fechner's law and a global attention encoding layer based on the Transformer encoder structure. S2. The control unit acquires raw images of the surgical scene in real time through the laparoscopic camera; The S3 and WF-YOLO models obtain the corresponding instrument features after preprocessing the original images; S4. The WF-YOLO model matches the instrument features with a pre-stored dataset of known instruments. If the match is successful, bounding box I is generated to complete the identification and localization of the known surgical instruments; otherwise, proceed to S5. S5. For unknown instruments, the adaptive recognition module performs corresponding segmentation, feature extraction, and matching operations on the instrument features, and adaptively generates the corresponding bounding box II to complete the on-site identification and positioning of unknown surgical instruments. In S1, the network architecture of the WF-YOLO model is as follows: WFLayer is integrated as a front-end preprocessing module on the basis of the real-time object detection model YOLOv8, and a Transformer encoder is embedded in the feature fusion network of YOLOv8. The WF Layer is directly connected to the CSPDarknet backbone input layer of YOLOv8 to achieve synchronous parameter updates through gradient backpropagation. In S1, the adaptive recognition module includes: The FastSAM model combines user interaction boxes with the prompt encoder of SAM for semi-supervised real-time segmentation. The ORB submodule performs feature extraction on the template segmented by the FastSAM model and the original image, respectively. A FLANN submodule that performs feature matching on the binary feature descriptors extracted by the ORB module; The FastSAM model works as follows: In the original image displayed on the interface, the user operator receives interactive instructions through the operation interface, and selects the area where the unknown surgical instrument to be segmented is located through the interactive box in the interactive instructions. The cue encoder in the FastSAM model converts the coordinates of the selected region into an embedding vector that can be fused with the features of the image encoder to generate a segmentation template through the mask decoder; The ORB submodule works as follows: An improved FAST corner detection algorithm is used to analyze the segmentation model output by the FastSAM model and the key points in the overall image. The key points refer to pixels in the image whose grayscale or gradient changes drastically in local areas and can be applied to subsequent matching and recognition. The orientation-aware BRIEF algorithm generates corresponding 32-dimensional binary feature descriptors for key points; The FLANN submodule works as follows: Normalize the feature vector dimension of the binary feature descriptors extracted from the ORB submodule; A multi-level kd-tree index is constructed based on the normalized feature points and the levels are divided. The fast retrieval of near nearest neighbors is achieved by performing multi-tree parallel search within the search radius. The search results are filtered based on a preset distance threshold; The RANSAC is used to perform geometric verification matching between the screening results and the pre-stored unknown instrument template library. If the match is successful, a bounding box II related to the instrument positioning is generated. Otherwise, the screening results are marked as new unknown instruments, and the operator is prompted to intervene through the user interface. In S5, the unknown device is confirmed to have been successfully identified by the already generated bounding box II, and the corresponding information of the unknown device is added to the known device dataset.

2. The laparoscopic instrument recognition method with enhanced visual perception and global attention as described in claim 1, characterized in that, In S3 and S4, the visual perception enhancement layer introduces multiple training parameters based on the Weber-Fechner theorem to perform dynamic nonlinear transformation on the input original image or instrument features to obtain the enhanced feature I. The training parameters include: a first parameter γ for setting the feature response baseline threshold, a second parameter α for controlling the sensitivity of feature changes, and a third parameter β for adjusting the nonlinear compression strength; The global attention encoding layer embeds positional encoding and residual connections, and uses a Transformer encoder structure to model the global context relationship of feature I in order to capture long-range dependency information in the original image or instrument features.

3. A method for tracking laparoscopic instruments, comprising employing the visual perception enhancement and global attention-based laparoscopic instrument recognition method as described in any one of claims 1-2, characterized in that, Also includes: S6. Calculate the deviation value (Δx, Δy) between the instrument center and the image center based on bounding box I or bounding box II. Transform the deviation value (Δx, Δy) into control commands for a 3-DOF laparoscopic robotic arm. After receiving the control commands, the robotic arm adjusts the position of the laparoscopic lens by driving joint movement through servo motors, moves the identified surgical instruments to the center of the image, and maintains tracking status throughout the subsequent movement of the surgical instruments.

4. A laparoscopic instrument identification and tracking system, employing the laparoscopic instrument tracking method as described in claim 3, comprising a laparoscopic camera, a laparoscopic robotic arm, and a user operating terminal, characterized in that, Also includes: A control unit that communicates with the laparoscopic camera, laparoscopic robotic arm, and user control terminal; The control unit is configured to use a rk3588 chip with a built-in neural network processing unit (NPU).

Images