Urinary calculus laser surgery anti-mishandling method and device based on visual AI model and storage medium

Abstract

This invention discloses a method, device, and storage medium for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model. The method includes: acquiring and preprocessing a video stream captured in real-time by a monocular ureteroscope; inputting the image to be identified into a multi-task neural network model, simultaneously outputting a laser point detection box, a mucosal region segmentation mask, and a monocular depth map; determining the coordinates of the laser point center pixel based on the detection box, determining the coordinates of the nearest mucosal edge pixel based on the segmentation mask, and obtaining the corresponding first and second depth values ​​from the depth map; inputting the coordinates, depth values, and intrinsic parameter matrix into a three-dimensional distance calculation model, reconstructing the three-dimensional spatial point coordinates based on the pinhole imaging formula, and calculating the actual physical distance; comparing the actual physical distance with dynamically adjusted safety and warning thresholds, and determining the risk level through fuzzy logic; generating control commands based on the risk level and sending them to the laser host to achieve graded early warning, power adjustment, or emergency interruption.

Description

Technical Field

[0001] This invention relates to the field of image / video analysis technology, specifically to a medical image processing method, system, and device based on a visual AI model, and particularly to a method, device, and storage medium for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model. Background Technology

[0002] Urinary tract stones are a common disease. With the continuous development of laser lithotripsy technology, ureteroscopic holmium laser lithotripsy has become a major minimally invasive treatment. However, approximately 96% of the laser energy is converted into heat energy during the procedure, which can easily lead to tissue thermal damage. Accidental laser injury to the ureteral mucosa can cause serious complications such as submucosal false passage formation, perforation, ureteral rupture, or even avulsion. Some patients may require ureteral reimplantation or nephrectomy. Furthermore, bleeding caused by mucosal damage during the procedure directly affects the surgical process. Blood absorbs laser energy, reducing lithotripsy efficiency. Blood clots and floating blood particles severely obscure the surgical field, increasing the difficulty of the procedure and even forcing its termination. To maintain a clear surgical field, surgeons often need to increase the perfusion pressure, but this increases the risk of postoperative complications such as systemic inflammatory response syndrome and urosepsis.

[0003] Currently, in clinical practice, safe laser operation mainly relies on the doctor's visual judgment and accumulated experience. However, this manual judgment method has inherent defects: prolonged operation can easily lead to visual fatigue and decreased accuracy of judgment; endoscopic images lack depth information, and two-dimensional images cannot truly reflect the actual physical distance between the laser point and the tissue, making it difficult to assess the risks when the endoscope angle changes or moves forward or backward; factors such as bleeding obstruction, air bubble interference, specular highlights, motion blur, and artifacts caused by turbid irrigation fluid further increase the difficulty and uncertainty of manual interpretation.

[0004] To address these issues, researchers have attempted to introduce artificial intelligence (AI) technology into endoscopic surgery. For example, one team developed a YOLO-based real-time kidney stone detection system capable of identifying the location and size of stones; others explored monocular depth estimation technology for endoscopic images to provide spatial awareness for surgical navigation. However, existing solutions have the following shortcomings: First, most studies focus on a single task, failing to organically integrate target recognition, tissue segmentation, and depth estimation, and thus cannot simultaneously acquire complete information on three key targets: laser point, stone, and mucosa. Second, existing solutions mostly remain at the two-dimensional pixel level, failing to combine two-dimensional coordinates with depth information to calculate actual physical distances, and thus cannot fundamentally solve the problem of misjudgment caused by endoscopic movement. Third, existing systems have simple response mechanisms, mostly relying on simple alarms or manual intervention, lacking real-time communication with the laser host and automatic interruption functions, making it difficult to cut off laser output within milliseconds to ensure surgical safety, and even more difficult to provide early warning before bleeding occurs and avoid the chain reaction of blurred vision. Summary of the Invention

[0005] This invention overcomes the shortcomings of the prior art and provides a method, equipment and storage medium for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: a method for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model, comprising the following steps:

[0007] S1. Acquire the surgical field video stream acquired in real time by the ureteroscope, and preprocess the current video frame in the surgical field video stream to obtain the image to be identified.

[0008] S2. Input the image to be identified into the trained multi-task neural network model, extract image features through the shared backbone network of the multi-task neural network model, and process the image features through parallel target detection head, instance segmentation head and monocular depth estimation head respectively, and output the laser point detection box, mucosal region segmentation mask and monocular depth map in the current video frame.

[0009] S3. Determine the center pixel coordinates of the laser point according to the laser point detection frame, determine the mucosal edge pixel coordinates that are closest to the center pixel coordinates of the laser point according to the mucosal region segmentation mask, and obtain the first depth value corresponding to the center pixel coordinates of the laser point and the second depth value corresponding to the mucosal edge pixel coordinates according to the monocular depth map.

[0010] S4. Input the center pixel coordinates of the laser point, the pixel coordinates of the mucosal edge, the first depth value, the second depth value, and the pre-calibrated intrinsic parameter matrix of the ureteroscope into the three-dimensional distance calculation model to calculate the actual physical distance between the laser point and the ureteral mucosa.

[0011] S5. Compare the actual physical distance with the preset safety threshold and warning threshold to determine the risk level of the current surgical scenario;

[0012] S6. Based on the determined risk level, generate the corresponding control command and send it to the laser host to control the energy output status of the laser host.

[0013] In a preferred embodiment of the present invention, the training process of the multi-task neural network model includes:

[0014] S201. A training sample set is constructed using a specific data augmentation method that simulates an endoscopic environment;

[0015] S202. The multi-stage training of the multi-task neural network model is carried out using the training sample set, including a pre-training stage on a large-scale public dataset, a fine-tuning stage on a self-built in vivo and in vitro endoscopy dataset, and an adaptation stage on a small amount of data from the target hospital.

[0016] The specific data augmentation method includes at least one of simulated bleeding occlusion, simulated bubble highlights, and simulated image blurring.

[0017] In a preferred embodiment of the present invention, the calculation process of the actual physical distance includes:

[0018] S401, According to the pinhole imaging model formula The center pixel coordinates of the laser point Compared with the first depth value Convert to first three-dimensional space point coordinates;

[0019] S402, Set the pixel coordinates of the mucosal edge With the second depth value Convert to point coordinates in the second three-dimensional space;

[0020] S403. Calculate the Euclidean distance between the coordinates of the first three-dimensional space point and the coordinates of the second three-dimensional space point, and use it as the actual physical distance;

[0021] in, The focal length parameter in the intrinsic parameter matrix, The coordinates of the principal point in the intrinsic parameter matrix are given.

[0022] In a preferred embodiment of the present invention, the preset safety threshold and warning threshold are dynamically adjusted, including:

[0023] S501. Obtain the current output power and current pulse frequency of the laser host;

[0024] S502. Calculate the dynamic safety threshold and dynamic warning threshold based on the current output power, current pulse frequency and image clarity evaluation factor of the current video frame.

[0025] The dynamic safety threshold is positively correlated with the current output power and the current pulse frequency, and the dynamic warning threshold is positively correlated with the current output power.

[0026] In a preferred embodiment of the present invention, the determination of the risk level includes:

[0027] S601. Calculate the membership function value of the actual physical distance relative to the dynamic safety threshold and the dynamic warning threshold, wherein the membership function value represents the first membership degree of the actual physical distance belonging to the safe state, the second membership degree of the actual physical distance belonging to the warning state, and the third membership degree of the actual physical distance belonging to the dangerous state, respectively.

[0028] S602. Determine the risk level of the current surgical scenario based on the maximum value among the first membership degree, the second membership degree, and the third membership degree.

[0029] In a preferred embodiment of the present invention, the control command includes:

[0030] S611. When the state is determined to be safe, generate a first control command that allows full power laser output.

[0031] S612. When a warning state is determined, a second control command is generated to trigger an audible and visual warning signal and reduce the laser output power.

[0032] S613. When a dangerous situation is determined, a third control command is generated to urgently interrupt the laser energy output.

[0033] In a preferred embodiment of the present invention, the second control command includes: reducing the laser output power in stages according to the numerical relationship between the actual physical distance and the dynamic warning threshold; including:

[0034] When the actual physical distance is greater than or equal to the first distance threshold and less than the dynamic safety threshold, the laser output power is reduced to the first power value;

[0035] When the actual physical distance is greater than or equal to the dynamic warning threshold and less than the first distance threshold, the laser output power is reduced to the second power value;

[0036] Wherein, the first distance threshold is between the dynamic warning threshold and the dynamic safety threshold, and the second power value is lower than the first power value.

[0037] In a preferred embodiment of the present invention, the method further includes the following steps:

[0038] S7. The current video frame, laser point detection box, mucosal region segmentation mask, monocular depth map and actual physical distance are associated and stored in the local database.

[0039] S8. When the amount of data accumulated in the local database reaches a preset threshold, the data in the local database is used to perform online incremental learning on the multi-task neural network model and update the model parameters.

[0040] S9. Upload the updated model parameters to the cloud server via wireless network, and receive the latest basic model obtained from global data training based on multiple devices from the cloud server for offline OTA update.

[0041] In a preferred embodiment of the present invention, a device for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model is provided, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the aforementioned method for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model.

[0042] In a preferred embodiment of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the above-described method for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model.

[0043] In a preferred embodiment of the present invention, a system for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model is provided, comprising: a video preprocessing module, a multi-task neural network inference module, a three-dimensional distance calculation module, a dynamic risk assessment module, a laser host interactive control module, and adapter interfaces that are respectively connected to a monocular ureteroscope and a laser host for communication.

[0044] The video preprocessing module is configured to acquire the surgical field video stream acquired in real time by a monocular ureteroscope and preprocess the current video frame to obtain the image to be identified.

[0045] The multi-task neural network inference module has a built-in trained multi-task neural network model. The multi-task neural network model is equipped with a shared backbone network and parallel target detection head, instance segmentation head, and monocular depth estimation head connected to the output of the shared backbone network. The multi-task neural network inference module is configured to input the image to be identified into the multi-task neural network model, extract image features through the shared backbone network, process them synchronously through the three parallel task heads, and simultaneously output the laser point detection box, mucosal region segmentation mask, and monocular depth map in the current video frame.

[0046] The three-dimensional distance calculation module is configured to determine the center pixel coordinates of the laser point based on the laser point detection box, determine the mucosal edge pixel coordinates that are closest to the center pixel coordinates of the laser point based on the mucosal region segmentation mask, obtain the first depth value corresponding to the center pixel coordinates of the laser point and the second depth value corresponding to the mucosal edge pixel coordinates from the monocular depth map, and combine the pre-calibrated ureteroscope intrinsic parameter matrix to convert the two sets of pixel coordinates and corresponding depth values ​​into three-dimensional spatial point coordinates based on the pinhole imaging model, and calculate the actual physical distance between the laser point and the ureteral mucosa.

[0047] The dynamic risk assessment module is configured to acquire the current output power, current pulse frequency, and image clarity evaluation factor of the current video frame of the laser host, calculate the dynamic safety threshold and dynamic warning threshold, and calculate the membership function value of the actual physical distance relative to the dynamic safety threshold and dynamic warning threshold based on fuzzy logic, and determine the risk level of the current surgical scenario based on the maximum membership value.

[0048] The laser host interactive control module is configured to generate corresponding control commands based on the determined risk level and send them to the laser host through an adapter interface to control the energy output status of the laser host.

[0049] In a preferred embodiment of the present invention, a simulation training method for a urinary tract stone laser surgery anti-injury system based on a visual AI model is also provided. This method trains the multi-task neural network model built into the aforementioned urinary tract stone laser surgery anti-injury system, and includes the following steps:

[0050] T1. Construct a training sample set simulating an endoscopic surgical environment. A specific data augmentation method for urinary endoscopic scenarios is used to amplify the original samples. The specific data augmentation method includes at least one of simulated bleeding occlusion, simulated bubble highlights, and simulated image blur. Each sample in the training sample set is assigned three types of supervision signals: ground truth values ​​of laser point and tissue target detection boxes, ground truth values ​​of ureteral mucosal region segmentation masks, and ground truth values ​​of absolute depth maps.

[0051] T2. Perform multi-stage training on the multi-task neural network model. The multi-stage training includes, in sequence: a pre-training stage on a large-scale public medical image dataset, a fine-tuning stage on a self-built in vivo and in vitro urinary endoscopic dataset, and an adaptation stage on surgical scenario data from the target hospital.

[0052] In the pre-training stage, a self-supervised learning approach is adopted, with photometric consistency loss and depth smoothness loss as loss functions, and the model outputs a relative depth map.

[0053] During the fine-tuning phase, samples with ground truth absolute depth values ​​are introduced for hybrid supervised training. The model parameters are optimized by combining self-supervised loss and absolute depth supervised loss, so that the model outputs an absolute depth map in millimeters. At the same time, for the instance segmentation head, a joint loss function of Dice loss and cross-entropy loss is used to optimize the mucosal region segmentation accuracy, and for the target detection head, bounding box regression loss and classification loss are used to optimize the target detection accuracy.

[0054] In the adaptation phase, an incremental learning approach is adopted, freezing the shared backbone network parameters of the model and only fine-tuning the top-level parameters of the three parallel task heads to complete the adaptation of the model to the target surgical scenario.

[0055] T3. Deploy the multi-task neural network model that has completed multi-stage training to the anti-collateral damage system to complete the system's simulation training and deployment.

[0056] In a preferred embodiment of the present invention, an iterative method for a laser surgery prevention system for urinary tract stones based on a visual AI model is provided. This method iteratively optimizes the multi-task neural network model within the laser surgery prevention system for urinary tract stones, and includes the following steps:

[0057] U1. A single anti-miscarriage system will associate the current video frame, laser point detection frame, mucosal region segmentation mask, monocular depth map, actual physical distance and risk level judgment results during the operation in real time, and store them in a local database with encryption.

[0058] U2. When the amount of data accumulated in the local database reaches a preset threshold, a single anti-mishandling system uses the locally stored data to perform online incremental learning on the built-in multi-task neural network model, freezes the shared backbone network parameters of the model, updates only the top-level parameters of the parallel task head, and generates locally updated model parameters.

[0059] U3. A single anti-collateral damage system uploads the locally updated model parameters to the cloud server via an encrypted wireless network. The cloud server receives the locally updated model parameters uploaded by multiple anti-collateral damage systems and uses a federated averaging algorithm to fuse multiple sets of locally updated model parameters to generate globally optimized model parameters.

[0060] U4 and the cloud server distribute the globally optimized model parameters to each anti-collateral damage system. Each anti-collateral damage system receives the globally optimized model parameters during idle periods and completes the offline OTA update of the multi-task neural network model.

[0061] This invention addresses the shortcomings of the prior art and has the following beneficial effects:

[0062] (1) By constructing a multi-task neural network model, and setting up a target detection head, an instance segmentation head, and a monocular depth estimation head in parallel on the same shared backbone network, the synchronous identification and spatial information extraction of three key targets in the surgical field of view—laser points, stones, and ureteral mucosa—were achieved. This multi-task joint learning mechanism enables the model to share underlying visual information during the feature extraction stage. The detection task provides target region priors for the segmentation task, the segmentation task provides edge constraints for the depth estimation task, and the depth estimation task provides spatial context for detection and segmentation. The three tasks promote each other and optimize synergistically, significantly improving the recognition accuracy and robustness in complex endoscopic environments.

[0063] (2) By combining monocular depth estimation technology with a three-dimensional spatial distance calculation model, a leapfrog transformation from two-dimensional pixel distance to three-dimensional physical distance is achieved without increasing any hardware costs. By substituting the center pixel coordinates of the laser point, the pixel coordinates of the nearest mucosal edge, and the corresponding depth value into the pinhole imaging model, the three-dimensional spatial coordinates of the laser point and the mucosal edge are accurately reconstructed, and then the Euclidean distance between them is calculated as the actual physical distance. This fundamentally overcomes the inherent defects of traditional two-dimensional pixel distance, which cannot reflect the real spatial relationship and is easily affected by the movement and rotation of the endoscope, so that the distance measurement results truly correspond to the actual intervals between human tissues, providing an accurate and reliable quantitative basis for subsequent risk assessment. Even if the endoscope lens moves or rotates, the three-dimensional distance remains stable, effectively eliminating false alarms and missed alarms caused by lens movement.

[0064] (3) This invention establishes a two-layer decision-making mechanism of dynamic threshold generation and fuzzy logic judgment, realizing a refined assessment of surgical risks. The dynamic threshold module obtains the current output power and current pulse frequency of the laser host in real time, and dynamically adjusts the values ​​of the safety threshold and warning threshold in combination with the image clarity evaluation factor, so that the system can adaptively adjust the safety margin according to the actual surgical parameters and image quality, avoiding the problem of insufficient adaptability of fixed thresholds under different surgical conditions. The fuzzy logic judgment module calculates the membership function value of the actual physical distance relative to the dynamic threshold, transforming the discrete risk level judgment into continuous risk membership calculation. This not only smooths the state jump caused by small fluctuations in distance measurement, but also provides richer control information for subsequent graded responses, making the risk judgment more in line with the continuous change characteristics of clinical practice.

[0065] (4) The graded response collaborative control strategy achieves progressive safety protection from early warning to interruption. When the system determines that it is in a warning state, it not only triggers graded audio-visual warning signals (using different colors and frequencies according to the proximity), but more importantly, it simultaneously reduces the laser output power, and the power reduction is greater the closer the distance. This provides doctors with buffer time and adjustment opportunities, avoiding the impact of sudden interruption on the smoothness of the operation; when it is determined to be in a dangerous state, the system ensures that the laser energy output is cut off in milliseconds through a dual mechanism of software instructions and hardware interruption, ensuring patient safety even in extreme cases such as software freeze. Compared with the single alarm or complete reliance on manual intervention in the prior art, the graded collaborative control strategy of this invention takes into account both safety and surgical efficiency, which is more in line with actual clinical needs. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a flowchart of a preferred embodiment of the present invention. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein. Therefore, the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0070] like Figure 1 As shown, a method for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model is applied to a surgical safety control system connected to a monocular ureteroscope and a laser host, and includes the following steps:

[0071] S1. Acquire the surgical field video stream captured in real time by the ureteroscope, and preprocess the current video frame in the surgical field video stream to obtain the image to be identified.

[0072] In practice, a high-definition camera built into a monocular ureteroscope is used to capture a real-time video stream within the surgical field of view at a frame rate of 30fps or higher. For each captured video frame... ,in Representing pixel coordinates, the system performs the following preprocessing operations:

[0073] The first step is to correct the distortion of the original image. Using pre-calibrated monocular ureteroscope distortion coefficients k1, k2, p1, and p2, the OpenCV library is employed. The function corrects the image to obtain the corrected image. The calibration process adopted the Zhang Zhengyou calibration method, which was completed by photographing the checkerboard pattern from different angles.

[0074] The second step is to perform brightness normalization on the corrected image. The average brightness of the image is then calculated. and standard deviation Mapping image pixel values ​​to Interval: ,in, Use a small constant to prevent division by zero. The normalized image is used as the image to be identified, denoted as . Their size was uniformly scaled down to 640×640 pixels to accommodate the input requirements of subsequent neural networks.

[0075] S2. Input the image to be identified into the trained multi-task neural network model, extract image features through the shared backbone network of the multi-task neural network model, and process the image features through parallel target detection head, instance segmentation head and monocular depth estimation head respectively, and output the laser point detection box, mucosal region segmentation mask and monocular depth map in the current video frame.

[0076] Furthermore, the training process of a multi-task neural network model includes:

[0077] S201. A training sample set is constructed using a specific data augmentation method that simulates an endoscopic environment;

[0078] S202. Use the training sample set to perform multi-stage training on the multi-task neural network model, including a pre-training stage on a large-scale public dataset, a fine-tuning stage on a self-built in vivo and in vitro endoscopy dataset, and an adaptation stage on a small amount of data from the target hospital.

[0079] Specific data augmentation methods include at least one of simulating bleeding occlusion, simulating bubble highlights, and simulating image blur.

[0080] In one embodiment, training a multi-task neural network model requires three types of supervision signals: ground truth of object detection boxes, ground truth of mucosal segmentation mask, and ground truth of absolute depth.

[0081] The ground truth of the mucosal segmentation mask was obtained using a semi-automatic annotation + expert correction strategy. First, the pre-trained model MedSAM, widely used in medical image segmentation, was used to perform initial segmentation on the original endoscopic image, generating a preliminary mucosal region mask. Then, at least three urologists with more than five years of experience in ureteroscopy used professional medical image annotation tools to correct the preliminary segmentation results frame by frame. Annotation specifications required that the mucosal region be defined as the visible mucosal tissue on the inner surface of the ureteral lumen, excluding stones, laser fibers, bleeding areas, and obvious air bubbles. A majority voting mechanism was used to integrate the annotation results of multiple experts; for areas with significant disagreements, higher-level experts arbitrated the results. A quality control process was established throughout the annotation process to ensure that the accuracy of the annotated mucosal edges reached the pixel level. During training, the segmentation mask output by the model... With truth mask Calculate Dice loss and cross-entropy loss: .

[0082] The absolute depth map ground truth is obtained using a hybrid supervised training strategy. During the pre-training phase, the model only uses self-supervised loss (photometric consistency loss). and depth smoothing loss Output relative depth map During the fine-tuning phase, a small number of images with absolute depth ground truth were introduced for supervision. The absolute depth ground truth was obtained in the following ways: (1) In vitro simulation environment: using an imaging system with the same optical parameters as a ureteroscope, targets at known distances (5mm, 10mm, 15mm, 20mm) were photographed to construct an in vitro depth dataset; (2) In vivo sparse depth points: in keyframes of some surgical videos, using optical fibers of known diameter as references, the absolute depth was calculated: ;in, The actual diameter of the optical fiber (e.g.) ), The pixel diameter of the optical fiber in the image. Let be the focal length. This allows for the sparse acquisition of absolute depth ground truth points in the image. The loss function during the fine-tuning stage is... ;in, The L2 loss is calculated only for pixels with true absolute depth values. After training, the model directly outputs an absolute depth map in millimeters. .

[0083] In practical implementation, the image to be identified The input is fed into the trained multi-task neural network model. In this embodiment, the multi-task neural network model adopts the improved YOLO-2026-SE architecture and includes the following components:

[0084] Shared backbone network: EfficientNet-Lite is used as the feature extraction backbone. The input image first passes through a 7×7 convolutional layer with a stride of 2, outputting a feature map F0 of size 320×320. Subsequently, features are extracted step-by-step through the MBConv module and the SE attention module, resulting in three feature maps of different scales: F1 (160×160), F2 (80×80), and F3 (40×40). Each MBConv module contains depthwise separable convolutions, batch normalization, and the Swish activation function.

[0085] Feature Pyramid Network: A bidirectional feature pyramid network (BiFPN) is adopted. F3 is used as the top-level feature, and it is upsampled and fused with F2 to obtain P2; similarly, P2 is upsampled and fused with F1 to obtain P1. Simultaneously, P1 is downsampled and fused with P2, and P2 is downsampled and fused with P3, forming a bidirectional feature fusion path from top to bottom and bottom to top. The fusion method uses weighted fast fusion. ,in These are learnable weights.

[0086] Three parallel task heads:

[0087] (1) Target Detection Head: Detection heads are set up at three scales: P1, P2, and P3. Each detection head contains three bounding boxes. For each bounding box, the detection head outputs a (4+1+C) dimensional vector, where 4 represents the bounding box coordinate offset. 1 represents the target confidence level, and C represents the number of categories (in this example, C=3, corresponding to laser point, stone, and ureter). The coordinate decoding formula is:

[0088] ;

[0089] ;

[0090] in, The coordinates of the top left corner of the grid cell. The width and height of the anchor frame are defined. After non-maximum suppression (NMS), the output laser point detection frame is determined. , represented as .

[0091] (2) Instance Segmentation Head: A parallel segmentation branch is added to the detection head. For each detected target, the segmentation head outputs a fixed-size (28×28) mask coefficient through a fully convolutional network, which is then combined with the prototype mask to generate the final mask. Specifically, the model first generates k prototype masks through a segmentation branch. The size is 28×28; simultaneously, k mask coefficients are predicted for each detected target. The final target mask is obtained by linearly combining the prototype masks and activating them with a Sigmoid function.

[0092] ;

[0093] For the ureteral mucosa, since it is a background area rather than a specific target, this embodiment uses an alternative approach: the image is divided into a grid, and for each grid cell, it is predicted whether it belongs to the mucosal region, thereby generating a mucosal region segmentation mask. Its size is the same as the input image, and each pixel value is 0 or 1, where 1 indicates that it belongs to the mucosa.

[0094] (3) Monocular Depth Estimation Header: A depth estimation branch follows the P3 feature map. This branch consists of three upsampling modules, each containing bilinear upsampling, 3×3 convolution, and ReLU activation, ultimately outputting a depth map of the same size as the input image. The depth value represents the actual physical distance from the lens to the tissue surface, measured in millimeters. The depth estimation head is trained using a self-supervised method, and the loss function includes:

[0095] ;

[0096] in, To compensate for photometric consistency loss, motion estimation between consecutive frames is used; For depth smoothing loss, the continuity of depth values ​​between adjacent pixels is encouraged, and the calculation formula is as follows:

[0097] ;

[0098] The three task heads mentioned above are executed in parallel, sharing the computation results of the same backbone network. In one forward propagation, the model simultaneously outputs:

[0099] Laser point detection box set: .

[0100] Mucosal region segmentation mask:

[0101] Monocular depth map: , in millimeters.

[0102] S3. Determine the center pixel coordinates of the laser point based on the laser point detection box, determine the mucosal edge pixel coordinates that are closest to the center pixel coordinates of the laser point based on the mucosal region segmentation mask, and obtain the first depth value corresponding to the center pixel coordinates of the laser point and the second depth value corresponding to the mucosal edge pixel coordinates based on the monocular depth map.

[0103] In practical implementation, the key data required for calculation is extracted from the identification results:

[0104] Laser point center coordinate calculation: For the detected laser point detection box Its center pixel coordinates The calculation formula is:

[0105] ;

[0106] ;

[0107] Since laser points typically appear as bright areas in images and may occupy multiple pixels, this embodiment further performs weighted centroid calculation on the pixels within the detection box to improve accuracy.

[0108] ;in, Iterate through all pixels within the detection box. This is the grayscale value of the pixel.

[0109] Recent mucosal edge point search: Obtain mucosal region segmentation mask Next, the contour of the mucosal region is extracted. N represents the total number of outline pixels. Then, the center of the laser point is calculated. Squared Euclidean distance to each contour pixel:

[0110] ;

[0111] The choice makes The smallest contour pixel is taken as the nearest mucosal edge point, and its coordinates are: :

[0112] ;

[0113] Depth value extraction: from depth map In the process, obtain the depth value at the center coordinates of the laser point. And the depth value at the coordinates of the nearest mucosal edge point. :

[0114] ;

[0115] ;

[0116] To prevent the influence of single-point outliers, the median filtering result within a 3×3 neighborhood is actually used as the final depth value:

[0117] ;

[0118] The same calculation applies.

[0119] S4. Input the center pixel coordinates of the laser point, the pixel coordinates of the mucosal edge, the first depth value, the second depth value, and the pre-calibrated intrinsic parameter matrix of the ureteroscope into the three-dimensional distance calculation model to calculate the actual physical distance between the laser point and the ureteral mucosa.

[0120] Furthermore, the calculation process for the actual physical distance includes:

[0121] S401, According to the pinhole imaging model formula The center pixel coordinates of the laser point Compared with the first depth value Convert to first three-dimensional space point coordinates;

[0122] S402, Set the pixel coordinates of the mucosal edge With the second depth value Convert to point coordinates in the second three-dimensional space;

[0123] S403. Calculate the Euclidean distance between the coordinates of the first three-dimensional space point and the coordinates of the second three-dimensional space point, and use it as the actual physical distance.

[0124] in, The focal length parameter is in the intrinsic parameter matrix. These are the coordinates of the principal points in the intrinsic parameter matrix.

[0125] In practical implementation, the system converts two-dimensional pixel coordinates into three-dimensional spatial coordinates based on the pinhole imaging model. Let the pre-calibrated intrinsic parameter matrix K be:

[0126] .

[0127] in, This is the focal length parameter (in pixels). The coordinates of the main point (in pixels).

[0128] For the center of the laser point, its three-dimensional spatial coordinates The calculation is as follows:

[0129] ;

[0130] Methods, equipment, and storage media for preventing accidental injury during laser surgery for urinary tract stones based on visual AI models. ;

[0131] ;

[0132] For the nearest mucosal edge point, its three-dimensional spatial coordinates The calculation is as follows:

[0133] ;

[0134] ;

[0135] ;

[0136] The actual physical distance between the laser spot and the ureteral mucosa This is the Euclidean distance between the two points mentioned above:

[0137] ;

[0138] Considering the potential errors in monocular depth estimation, this embodiment introduces a confidence-weighted mechanism. The confidence level for depth estimation is defined as follows: Based on local texture complexity calculation:

[0139] ;

[0140] in, The gray-level variance of the region near the laser point. This is a preset threshold. When When the value is high, the above three-dimensional distance is used directly; when... At lower levels, use two-dimensional pixel distance. Distance is estimated using empirical formulas as compensation:

[0141] ;

[0142] ;

[0143] Where f is the average of fx and fy, The two-dimensional pixel distance between the center of the laser spot and the edge of the mucosa:

[0144] .

[0145] S5. Compare the actual physical distance with the preset safety threshold and warning threshold to determine the risk level of the current surgical scenario.

[0146] Furthermore, the preset safety thresholds and warning thresholds are dynamically adjusted, including:

[0147] S501. Obtain the current output power and current pulse frequency of the laser host;

[0148] S502. Calculate the dynamic safety threshold and dynamic warning threshold based on the current output power, current pulse frequency and image clarity evaluation factor of the current video frame.

[0149] Among them, the dynamic safety threshold is positively correlated with the current output power and the current pulse frequency, and the dynamic warning threshold is positively correlated with the current output power.

[0150] In practice, the current output power P (in watts) and pulse frequency F (in Hertz) of the laser host are acquired in real time. Simultaneously, the image sharpness evaluation factor Q of the current video frame is calculated using the Laplacian variance method.

[0151] ;

[0152] in, For grayscale images, the Laplacian operator is a 3×3 Laplacian kernel; the larger the variance, the clearer the image.

[0153] Dynamic security threshold and dynamic warning threshold The calculation formula is as follows:

[0154] ;

[0155] .

[0156] in:

[0157] The basic safety threshold is set to 2.0 mm. The basic warning threshold is set to 0.5mm; P0 is the reference power, set to 10W; F0 is the reference frequency, set to 10Hz; Q0 is the reference sharpness, obtained by statistically analyzing normal images. The adjustment coefficient, determined experimentally, is taken in this embodiment. .

[0158] The above formula ensures that the safety threshold increases accordingly when the laser power increases, the frequency increases, or the image becomes blurred, thus providing a higher safety margin.

[0159] S6. Based on the determined risk level, generate the corresponding control command and send it to the laser host to control the energy output status of the laser host.

[0160] Furthermore, the determination of risk level includes:

[0161] S601. Calculate the membership function value of the actual physical distance relative to the dynamic safety threshold and the dynamic warning threshold. The membership function value represents the first membership degree of the actual physical distance belonging to the safe state, the second membership degree belonging to the warning state, and the third membership degree belonging to the dangerous state, respectively.

[0162] S602. Determine the risk level of the current surgical scenario based on the maximum value among the first membership degree, the second membership degree, and the third membership degree.

[0163] In practical implementation, to avoid frequent state jumps caused by fluctuations in distance measurement, this embodiment uses fuzzy logic for risk level determination. Three membership functions are defined:

[0164] Security status membership :

[0165] ;

[0166] ;

[0167] ,when .

[0168] Dangerous state membership degree :

[0169] ;

[0170] ;

[0171] .

[0172] Warning state membership :

[0173] ;

[0174] For actual physical distance Calculate the three membership values ​​respectively. Then, take the state corresponding to the maximum value as the current risk level:

[0175] ;

[0176] For example, if , ,but , , The maximum value of 0.533 corresponds to a dangerous state, therefore it is determined to be a dangerous state.

[0177] Furthermore, the control commands include:

[0178] S611. When the state is determined to be safe, generate a first control command that allows full power laser output.

[0179] S612. When a warning state is determined, a second control command is generated to trigger an audible and visual warning signal and reduce the laser output power.

[0180] S613. When a dangerous situation is determined, a third control command is generated to urgently interrupt the laser energy output.

[0181] The second control command includes: reducing the laser output power in stages based on the numerical relationship between the actual physical distance and the dynamic warning threshold; including:

[0182] When the actual physical distance is greater than or equal to the first distance threshold and less than the dynamic safety threshold, the laser output power is reduced to the first power value;

[0183] When the actual physical distance is greater than or equal to the dynamic warning threshold and less than the first distance threshold, the laser output power will be reduced to the second power value.

[0184] The first distance threshold is between the dynamic warning threshold and the dynamic safety threshold, and the second power value is lower than the first power value.

[0185] In practice, control commands are sent to the laser host via a communication interface based on the determined risk level. The communication protocol uses an RS232 serial port with a baud rate of 115200, and the data format is: start bit (0xAA) + command code + parameter + checksum.

[0186] when At this time, the first control command is generated, command code 0x01, parameter 0x00 indicates that full power output is allowed. The laser host maintains the currently set power value.

[0187] when At that time, a second control command is generated. First, an audible and visual warning is triggered: the LED strip is controlled via GPIO to flash yellow (frequency 2Hz), while the audio module plays an intermittent alert tone (frequency 1Hz, duty cycle 50%). Then, based on the actual physical distance... With dynamic warning threshold The numerical relationship allows for a graded reduction in laser output power:

[0188] Set the first distance threshold That is, the arithmetic mean of the dynamic safety threshold and the dynamic warning threshold.

[0189] when When the actual physical distance is in the upper half of the warning range (relatively safe), the laser output power is reduced to the first power value. ;in, The initial power setting for the doctor. The control command code is 0x02, and the parameter is P1 (coded in units of 0.1W).

[0190] when When the actual physical distance is in the lower half of the warning range (more dangerous), the laser output power is reduced to the second power value. The control command code is 0x03, and the parameter is P2.

[0191] when At that time, a third control command is generated. A red warning is triggered (LED stays red on + continuous rapid alarm sound), and an emergency interrupt command is immediately sent. To ensure reliability, this embodiment employs a dual mechanism of hardware interrupt and software command:

[0192] At the software level: interrupt instruction code 0xFF and parameter 0x00 are sent via RS232.

[0193] Hardware level: The system design includes a hardware comparator independent of the main control chip, which will... Analog quantity and preset danger threshold The comparison is performed, and once the value falls below the threshold, a TTL high-level interrupt signal is sent directly to the laser host through the high-speed digital I / O port, with a response time of less than 5ms.

[0194] It also includes the following steps:

[0195] S7. Associate and store the current video frame, laser point detection box, mucosal region segmentation mask, monocular depth map, and actual physical distance to the local database.

[0196] S8. When the amount of data accumulated in the local database reaches a preset threshold, the data in the local database is used to perform online incremental learning on the multi-task neural network model and update the model parameters.

[0197] S9. Upload the updated model parameters to the cloud server via wireless network, and receive the latest basic model obtained from global data training based on multiple devices from the cloud server for offline OTA update.

[0198] In practice, all key data during the surgical procedure is recorded in real time, and a local database is built. Each record includes:

[0199] Timestamp, current video frame (Compressed to JPEG format), laser point detection frame Mucosal region segmentation mask (Run-length encoding compression), monocular depth map (16-bit integer format), actual physical distance Risk level (state), control instructions, and response status.

[0200] When the amount of data accumulated in the local database reaches 1000 records, online incremental learning is triggered. The system uses the newly accumulated data to fine-tune the multi-task neural network model. Incremental learning employs mini-batch stochastic gradient descent with a learning rate set to 0.1 times the initial learning rate, updating only the parameters of the last few layers of the network to ensure rapid convergence without destroying existing knowledge.

[0201] Weekly, the system uploads updated model parameters (excluding complete training data) to the cloud server via an encrypted wireless network. The cloud server aggregates model parameters from multiple devices and uses a federated learning algorithm to perform global model fusion, generating the latest base model. During idle periods, each device downloads the latest base model from the cloud for offline OTA updates, enabling continuous system evolution.

[0202] By storing video streams, recognition results, distance data, and control commands of the entire surgical process in a local database, online incremental learning is automatically triggered when the data volume accumulates to a preset threshold, enabling the model to continuously adapt to the current operating habits of the hospital or doctors. Simultaneously, cloud-based federated learning enables the collective intelligent evolution of multiple devices, forming a complete closed loop from data collection and model optimization to deployment and updates. This gives the system self-evolution capabilities, allowing it to learn from each surgery, continuously improving its generalization ability and robustness, providing a continuous technical guarantee for long-term clinical application.

[0203] Based on the above technical solution, a device for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model is provided, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the above-mentioned method for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model.

[0204] In practice, the processor uses the NVIDIA Jetson AGX Orin embedded GPU module, which integrates 2048 CUDA cores and 64 Tensor cores, and supports INT8 quantization acceleration.

[0205] The system uses 32GB of LPDDR5 memory and a 256GB NVMe solid-state drive for data storage.

[0206] The interfaces include:

[0207] Video input interface: HDMI or SDI interface, for connecting a monocular ureteroscope camera.

[0208] Laser control interface: RS232 / 485 / CAN bus interface, compatible with mainstream laser host.

[0209] Display output interface: HDMI interface, for connecting to the operating room monitor.

[0210] GPIO interface: used to control the audible and visual alarm unit.

[0211] Audible and visual alarm unit interface: includes a three-color LED indicator (red / yellow / green) and a piezoelectric buzzer.

[0212] A computer-readable storage medium is also provided, which includes a stored computer program, wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the above-described method for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model.

[0213] In practice, the storage medium can be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0214] In one embodiment, a system for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model is provided, including: a video preprocessing module, a multi-task neural network inference module, a three-dimensional distance calculation module, a dynamic risk assessment module, a laser host interactive control module, and adapter interfaces that communicate with a monocular ureteroscope and a laser host respectively.

[0215] The video preprocessing module is configured to acquire the surgical field video stream acquired in real time by a monocular ureteroscope and preprocess the current video frame to obtain the image to be identified.

[0216] The multi-task neural network inference module has a built-in trained multi-task neural network model. The multi-task neural network model is equipped with a shared backbone network and parallel target detection head, instance segmentation head, and monocular depth estimation head connected to the output of the shared backbone network. The multi-task neural network inference module is configured to input the image to be recognized into the multi-task neural network model, extract image features through the shared backbone network, process them synchronously through the three parallel task heads, and simultaneously output the laser point detection box, mucosal region segmentation mask, and monocular depth map in the current video frame.

[0217] The three-dimensional distance calculation module is configured to determine the center pixel coordinates of the laser point based on the laser point detection box, determine the mucosal edge pixel coordinates that are closest to the center pixel coordinates of the laser point based on the mucosal region segmentation mask, obtain the first depth value corresponding to the center pixel coordinates of the laser point and the second depth value corresponding to the mucosal edge pixel coordinates from the monocular depth map, and combine it with the pre-calibrated ureteroscope intrinsic parameter matrix to convert the two sets of pixel coordinates and corresponding depth values ​​into three-dimensional spatial point coordinates based on the pinhole imaging model, and calculate the actual physical distance between the laser point and the ureteral mucosa.

[0218] The dynamic risk assessment module is configured to acquire the current output power, current pulse frequency, and image clarity evaluation factor of the current video frame of the laser host, calculate the dynamic safety threshold and dynamic warning threshold, and calculate the membership function value of the actual physical distance relative to the dynamic safety threshold and dynamic warning threshold based on fuzzy logic, and determine the risk level of the current surgical scenario based on the maximum membership value.

[0219] The laser host interactive control module is configured to generate corresponding control commands based on the determined risk level and send them to the laser host through an adapter interface to control the energy output status of the laser host.

[0220] In practical implementation, the core of the system is a surgical safety control host equipped with an NVIDIA Jetson AGX Orin embedded GPU module. The host features 32GB of LPDDR5 memory and a 256GB NVMe solid-state drive. It integrates multiple standardized adapter interfaces, including HDMI / SDI video input interfaces, RS232 / 485 laser control interfaces, GPIO audible and visual alarm interfaces, HDMI display output interfaces, and an encrypted wireless network module. During system setup, the video input interfaces communicate in real-time with the high-definition camera of the monocular ureteroscope in the operating room; the laser control interfaces are directly connected to the control port of the clinically commonly used holmium laser host; the display output interfaces are connected to the intraoperative monitor in the operating room; and the audible and visual alarm interfaces are connected to external red, yellow, and green LED indicators and a piezoelectric buzzer, achieving seamless compatibility with existing surgical equipment without requiring modifications to the existing operating room equipment.

[0221] Internally, the system integrates a video preprocessing module, a multi-task neural network inference module, a 3D distance calculation module, a dynamic risk assessment module, and a laser host interactive control module as software functional units. Each module operates collaboratively based on real-time data streams, meeting the millisecond-level response requirements throughout the procedure. During ureteroscopic holmium laser lithotripsy, once the ureteroscope enters the patient's ureter, the video preprocessing module acquires the surgical field video stream transmitted by the ureteroscope in real time at a frame rate of 30fps. Each frame of the original image is first corrected for distortion using a pre-calibrated ureteroscope distortion coefficient to eliminate image distortion caused by the wide-angle lens. Then, the corrected image undergoes brightness normalization and is uniformly scaled to a 640×640 pixel resolution, generating an image to be recognized that meets the model input requirements, which is then synchronously transmitted to the multi-task neural network inference module.

[0222] The multi-task neural network inference module incorporates a multi-task neural network model that completes training and quantization. The model employs an improved YOLO-2026-SE architecture, sharing an EfficientNet-Lite backbone network. The backend connects to a bidirectional feature pyramid network for multi-scale feature fusion, and then connects to parallel target detection, instance segmentation, and monocular depth estimation heads. The forward inference time for a single frame is less than 30ms, fully meeting intraoperative real-time requirements. After the image to be recognized is input, the model extracts low-level visual features through the shared backbone network. After bidirectional fusion via the feature pyramid, the three parallel task heads simultaneously complete the calculations. In one forward propagation, the model simultaneously outputs the laser point detection box, the ureteral mucosal region segmentation mask, and the monocular absolute depth map for the current frame. These three results are synchronously transmitted to the 3D distance calculation module, achieving simultaneous acquisition of laser point, mucosal tissue, and spatial depth information, avoiding the delay and error accumulation caused by serial processing of multiple models.

[0223] After receiving the inference results, the 3D distance calculation module first calculates the weighted centroid based on the coordinate range of the laser point detection box and the grayscale values ​​of the pixels within the box, obtaining the sub-pixel-level coordinates of the laser point center pixel. Then, it extracts the contour of the mucosal segmentation mask, traverses all mucosal edge pixels, and searches for the mucosal edge pixel coordinates closest to the laser point center pixel. Simultaneously, it extracts the median depth value of the 3×3 neighborhood corresponding to the two coordinate points from the monocular depth map, using them as the first depth value of the laser point and the second depth value of the mucosal edge point, respectively, eliminating calculation errors caused by single-point depth anomalies. Subsequently, the module, combined with a pre-calibrated ureteroscope internal parameter matrix, converts the two sets of pixel coordinates and corresponding depth values ​​into 3D spatial point coordinates in the camera coordinate system based on the pinhole imaging model, calculates the Euclidean distance between the two points, and finally obtains the actual physical distance between the laser point and the ureteral mucosa. The calculation results are synchronously transmitted to the dynamic risk assessment module, fundamentally solving the shortcomings of 2D pixel distance in reflecting the true spatial relationship and being easily affected by the endoscope's movement.

[0224] The dynamic risk assessment module obtains the current output power and pulse frequency of the holmium laser host in real time through the laser control interface. Simultaneously, it calculates the image clarity evaluation factor of the current video frame using the Laplace variance method and dynamically adjusts the safety and warning thresholds according to a preset formula. When the laser power increases, the pulse frequency accelerates, or the image becomes blurred due to bleeding or bubbles, the safety threshold increases accordingly to reserve a higher safety margin, avoiding the insufficient adaptability of fixed thresholds under different surgical conditions. Subsequently, based on fuzzy logic rules, the module calculates the membership degree of the actual physical distance relative to the dynamic safety threshold, dynamic warning threshold, and dynamic warning threshold for the safe, warning, and dangerous states. The state corresponding to the maximum membership degree is taken as the risk level of the current surgical scenario, smoothing out state jumps caused by minor fluctuations in distance measurement. The assessment result is synchronously transmitted to the laser host interactive control module.

[0225] The laser host interactive control module generates corresponding control commands based on the final risk level and sends them to the laser host via an adapter interface, simultaneously displaying visual prompts on the intraoperative monitor. When the condition is determined to be safe, a command is generated allowing full power output, the laser host maintains the doctor's preset output parameters, and the intraoperative monitor displays a green safety indicator. When the condition is determined to be warning, a yellow LED is first triggered to flash at a frequency of 2Hz via the GPIO interface, accompanied by an intermittent warning sound. Then, the laser output power is reduced in stages according to the relative relationship between the actual physical distance and the threshold, with a larger reduction in power as the laser approaches the mucosa. At the same time, the position and actual distance between the laser point and the nearest mucosa are marked on the intraoperative monitor, providing the doctor with an opportunity for operation buffer and adjustment. When the condition is determined to be dangerous, a red LED is immediately triggered to remain constantly lit, accompanied by a continuous rapid alarm sound. Simultaneously, an emergency interrupt command is sent to the laser host within 5ms through a dual mechanism of software commands and hardware interrupts to cut off the laser energy output. Even in the extreme case of software freeze, the patient's safety can be ensured through hardware interruption, achieving fully automated safety control from warning to protection.

[0226] In this embodiment, a simulation training method for a urinary tract stone laser surgery accident prevention system based on a visual AI model is also provided. The multi-task neural network model built into the urinary tract stone laser surgery accident prevention system is used for training, and includes the following steps:

[0227] T1. Construct a training sample set simulating an endoscopic surgical environment. Use specific data augmentation methods for urinary endoscopic scenarios to augment the original samples. The specific data augmentation methods include at least one of simulated bleeding occlusion, simulated bubble highlights, and simulated image blur. Each sample in the training sample set is assigned three types of supervision signals: ground truth values ​​of laser point and tissue target detection boxes, ground truth values ​​of ureteral mucosal region segmentation masks, and ground truth values ​​of absolute depth maps.

[0228] T2. Perform multi-stage training on the multi-task neural network model. The multi-stage training includes: a pre-training stage on a large-scale public medical image dataset, a fine-tuning stage on a self-built in vivo and in vitro urinary endoscopic dataset, and an adaptation stage on surgical scenario data from the target hospital.

[0229] In the pre-training stage, a self-supervised learning approach is adopted, with photometric consistency loss and depth smoothness loss as loss functions, and the model outputs a relative depth map.

[0230] During the fine-tuning phase, samples with ground truth absolute depth values ​​are introduced for hybrid supervised training. The model parameters are optimized by combining self-supervised loss and absolute depth supervised loss, so that the model outputs an absolute depth map in millimeters. At the same time, for the instance segmentation head, a joint loss function of Dice loss and cross-entropy loss is used to optimize the mucosal region segmentation accuracy, and for the target detection head, bounding box regression loss and classification loss are used to optimize the target detection accuracy.

[0231] In the adaptation phase, an incremental learning approach is adopted, freezing the shared backbone network parameters of the model and only fine-tuning the top-level parameters of the three parallel task heads to complete the adaptation of the model to the target surgical scenario.

[0232] T3. Deploy the multi-task neural network model that has completed multi-stage training to the anti-collateral damage system to complete the system's simulation training and deployment.

[0233] In a specific implementation, the entire training process is completed on a training server equipped with an NVIDIA A100 GPU, closely mirroring the real-world scenario of urinary endoscopic surgery to ensure that the trained model can adapt to complex clinical environments.

[0234] The first step in training is to construct the training sample set. Basic sample data is collected, including 12,000 frames of endoscopic images acquired in an in vitro simulated ureteral environment and 8,000 key frames of de-identified ureteroscopic surgery videos from the urology departments of several top-tier hospitals. The use of all samples has been approved by the ethics committee and complies with medical data privacy protection regulations. For each frame of sample, three types of supervisory signals were constructed. The ground truth of the target detection box was annotated by two experts with the title of associate chief physician or above in urology. The annotation categories included three core targets: laser points, stones, and ureteral lumen. The ground truth of the mucosal region segmentation mask was generated by a semi-automatic annotation plus expert correction method. Initial segmentation was first completed by a pre-trained MedSAM model, and then frame-by-frame correction was performed by three doctors with more than 5 years of experience in ureteroscopic surgery. A majority voting mechanism was used to integrate the annotation results of multiple experts. For areas with large discrepancies, the chief physician made the final arbitration to ensure that the annotation accuracy of the mucosal edge reached the pixel level. The ground truth of the absolute depth map was obtained in a hybrid way. For in vitro samples, continuous absolute depth ground truth was obtained by collecting data from a standard target at a known distance. For in vivo samples, sparse absolute depth ground truth points were calculated by using a laser fiber of known diameter as a size reference, thus completing the construction of the basic sample set.

[0235] Subsequently, to address common interference scenarios in endoscopic surgery such as bleeding, air bubbles, and blurring, specific data augmentation methods were employed to expand the basic sample set. These methods included simulating different degrees of bleeding occlusion by overlaying irregular red regions of varying transparency onto the image to recreate the visual obstruction effect caused by bleeding during surgery; simulating air bubbles and specular highlights by randomly adding circular highlight regions of different sizes and transparency to the image to recreate the interference effects of air bubbles and specular reflections during surgery; and simulating image blurring caused by motion blur and irrigation fluid turbidity by reducing image sharpness through Gaussian blur and random noise overlay to recreate the visual effects of lens movement and irrigation fluid turbidity during surgery. Ultimately, a training sample set of over 50,000 frames was obtained, which was divided into training, validation, and test sets in an 8:1:1 ratio, significantly improving the model's robustness to complex surgical scenarios.

[0236] After completing the sample set construction, multi-stage training was performed on the multi-task neural network model. The first stage was pre-training on a large-scale public dataset. ImageNet and the public medical endoscopy dataset CAD-PE were used to pre-train the shared backbone network of the model. The pre-training process adopted a self-supervised learning approach, using photometric consistency loss and depth smoothing loss between consecutive frames as the core loss functions. The training period was 30 epochs, the initial learning rate was set to 1e-4, and the AdamW optimizer was used to enable the model to learn the general endoscopy image feature extraction capabilities and stably output relative depth maps. The second stage involves fine-tuning on a self-built in vivo and external endoscopy dataset. The pre-trained backbone network parameters are unfrozen, and the entire network is fine-tuned using the constructed training sample set. A hybrid supervision strategy is employed during training. For the monocular depth estimation head, parameters are optimized using a combination of self-supervised loss and L2 supervised loss based on the absolute depth ground truth, enabling the model to output absolute depth maps in millimeters. For the instance segmentation head, a joint loss function of Dice loss and cross-entropy loss is used to optimize the segmentation accuracy of mucosal regions. For the target detection head, CIoU bounding box regression loss and cross-entropy classification loss are used to optimize the detection accuracy of targets such as laser points. The training cycle for the fine-tuning stage is 50 epochs, with an initial learning rate set to 1e-5. A cosine annealing learning rate scheduling strategy with preheating is used. Training is terminated early when the validation set loss no longer decreases for 10 consecutive epochs, and the model weights with the highest validation set accuracy are saved. The third stage is the adaptation stage on a small amount of data from the target hospital. 1000 de-identified surgical video frames from the target hospital are collected. An incremental learning approach is adopted, freezing all parameters of the shared backbone network of the model and only fine-tuning the parameters of the top fully connected layer and convolutional layer of the three parallel task heads. The training period is 10 epochs, and the initial learning rate is set to 5e-6. This avoids destroying the general features that the model has already learned and can also complete the adaptation to the surgical scene and equipment parameters of the target hospital, further improving the stability of the model in the target usage scenario.

[0237] After training, the model obtained from multi-stage training was compressed using INT8 quantization to enable real-time inference on an embedded GPU. It was then deployed to the control unit of the anti-injury system, and its accuracy and real-time performance were verified using a test set and an in vitro simulated surgical environment. Test results showed that the model achieved a 99.2% accuracy rate in detecting laser points, a Dice coefficient of 0.96 for mucosal segmentation, an average relative error of less than 5% in depth estimation, and a single-frame inference time of no more than 30ms, fully meeting the requirements for real-time intraoperative protection. This completed the simulation training and deployment of the entire system.

[0238] Furthermore, in this embodiment, an iterative method for a urinary tract stone laser surgery anti-injury system based on a visual AI model is also provided. This method is used to iteratively optimize the multi-task neural network model in the urinary tract stone laser surgery anti-injury system, and includes the following steps:

[0239] U1. A single anti-miscarriage system will associate the current video frame, laser point detection frame, mucosal region segmentation mask, monocular depth map, actual physical distance and risk level judgment results during the operation in real time, and store them in a local database with encryption.

[0240] U2. When the amount of data accumulated in the local database reaches a preset threshold, a single anti-mishandling system uses the locally stored data to perform online incremental learning on the built-in multi-task neural network model, freezes the shared backbone network parameters of the model, updates only the top-level parameters of the parallel task head, and generates locally updated model parameters.

[0241] U3. A single anti-collateral damage system uploads the locally updated model parameters to the cloud server via an encrypted wireless network. The cloud server receives the locally updated model parameters uploaded by multiple anti-collateral damage systems and uses a federated averaging algorithm to fuse multiple sets of locally updated model parameters to generate globally optimized model parameters.

[0242] U4 and the cloud server distribute the globally optimized model parameters to each anti-collateral damage system. Each anti-collateral damage system receives the globally optimized model parameters during idle periods and completes the offline OTA update of the multi-task neural network model.

[0243] In practice, the entire iterative process adheres to medical data privacy protection standards. The original surgical data is stored locally on a single device and no patient-related information is uploaded to the cloud, achieving a compliant iteration where data is available but not visible.

[0244] The entire iterative process begins with the clinical use of a single device. Deployed in various hospitals, each individual anti-miscarriage system, during each surgery, real-time association and binding of the current surgical video frame, laser point detection frame, mucosal region segmentation mask, monocular depth map, actual physical distance between the laser point and the mucosa, risk level assessment result, control commands, and laser host response status are performed according to timestamps. The video frames are compressed using JPEG, and the segmentation mask is compressed using run-length encoding, then encrypted using the AES256 encryption algorithm and stored in the device's local NVMe solid-state drive database. Simultaneously, the system performs compliance verification and cleaning of the stored data, removing invalid, ambiguous, and redundant data to ensure that every piece of stored data has value for model training. The system continuously counts the effective data volume in the local database. When the accumulated effective data volume reaches a preset threshold of 1000 records, an online incremental learning process is automatically triggered during the idle period after surgery.

[0245] When a single device initiates an incremental learning task, to avoid catastrophic forgetting of the model, a fixed backbone and fine-tuned head incremental fine-tuning strategy is adopted. All parameters of the shared backbone network of the multi-task neural network model are frozen, and only the top-level parameters of the three parallel task heads are updated. The learning rate is set to 0.1 times the initial training learning rate. A mini-batch stochastic gradient descent optimizer is used, with a batch size of 8 and a training period of 5 epochs. Locally stored surgical data is used as training samples, and the corresponding labeled ground truth values ​​are used as supervision signals to fine-tune the model parameters and generate locally updated model parameters. After incremental learning is completed, the system verifies the accuracy of the updated model using a locally reserved test set. Only when the model's detection accuracy, segmentation accuracy, and depth estimation accuracy are all no lower than the original model will the updated model parameters be used as valid locally updated parameters; otherwise, the system immediately rolls back to the original model parameters to ensure that the clinical performance of the device does not decline.

[0246] Every weekend, all online anti-miscarriage systems upload locally validated incremental model parameters to the cloud-based federated learning server via an encrypted wireless network. The uploaded content contains only the incremental model parameters and does not include any original surgical videos or patient-related data. End-to-end encryption is used throughout the process to ensure data transmission security. After receiving the locally updated model parameters from multiple devices, the cloud server uses a federated averaging algorithm to fuse all parameters. During fusion, weights are assigned to the corresponding model parameters based on the number of training samples uploaded by each device; devices with larger sample sizes receive higher parameter weights. Simultaneously, outlier parameters that cause a decrease in accuracy are automatically removed to avoid impacting the overall model performance. After fusion, globally optimized model parameters are generated. The global model is then comprehensively validated using multi-center, multi-scenario test sets. Only when the global model's generalization ability and robustness are superior to the previous version's base model across different hospitals, devices, and surgical scenarios is it released as the official update.

[0247] The cloud server distributes the verified globally optimized model parameters to all connected anti-misoperation systems. Each device automatically receives the global model parameters from the cloud during nighttime or idle periods between surgeries, completing an offline OTA update. During the update process, the system retains the stable model parameters of the previous version as a backup. If power outages, network outages, or other abnormalities occur during the update process, the system can immediately roll back to the stable version without affecting the normal clinical use of the devices. After the update is complete, the system will use the new version of the global model in subsequent surgeries, while continuing to collect effective data during the surgical process to initiate the next round of iterative optimization, forming a complete closed loop of "data collection - local incremental learning - cloud federated fusion - global update deployment". This allows the model to continuously learn from each clinical surgery, constantly improving its protective accuracy and adaptability in complex surgical scenarios.

[0248] In summary, by deeply optimizing image / video analysis algorithms, an intelligent anti-miscarriage system integrating multi-task recognition, three-dimensional distance calculation, dynamic threshold determination, hierarchical response control, and data closed-loop optimization was constructed. This system effectively solves the problems of strong subjectivity due to reliance on manual judgment, the inability of two-dimensional distance to reflect real spatial relationships, insufficient recognition accuracy in complex scenarios, and a single response mechanism in existing technologies. It significantly improves the safety, reliability, and intelligence level of laser surgery for urinary tract stones, and has good clinical application prospects and promotional value.

[0249] Based on the preferred embodiments of the present invention described above, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A method for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model, characterized in that, Includes the following steps: S1. Acquire the surgical field video stream acquired in real time by the ureteroscope, and preprocess the current video frame in the surgical field video stream to obtain the image to be identified. S2. Input the image to be identified into the trained multi-task neural network model, extract image features through the shared backbone network of the multi-task neural network model, and process the image features through parallel target detection head, instance segmentation head and monocular depth estimation head respectively, and output the laser point detection box, mucosal region segmentation mask and monocular depth map in the current video frame. S3. Determine the center pixel coordinates of the laser point according to the laser point detection frame, determine the mucosal edge pixel coordinates that are closest to the center pixel coordinates of the laser point according to the mucosal region segmentation mask, and obtain the first depth value corresponding to the center pixel coordinates of the laser point and the second depth value corresponding to the mucosal edge pixel coordinates according to the monocular depth map. S4. Input the center pixel coordinates of the laser point, the pixel coordinates of the mucosal edge, the first depth value, the second depth value, and the pre-calibrated intrinsic parameter matrix of the ureteroscope into the three-dimensional distance calculation model to calculate the actual physical distance between the laser point and the ureteral mucosa. S5. Compare the actual physical distance with the preset safety threshold and warning threshold to determine the risk level of the current surgical scenario; S6. Based on the determined risk level, generate the corresponding control command and send it to the laser host to control the energy output status of the laser host.

2. The method for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model according to claim 1, characterized in that: The training process of the multi-task neural network model includes: S201. A training sample set is constructed using a specific data augmentation method that simulates an endoscopic environment; S202. The multi-stage training of the multi-task neural network model is carried out using the training sample set, including a pre-training stage on a large-scale public dataset, a fine-tuning stage on a self-built in vivo and in vitro endoscopy dataset, and an adaptation stage on a small amount of data from the target hospital. The specific data augmentation method includes at least one of simulated bleeding occlusion, simulated bubble highlights, and simulated image blurring.

3. The method for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model according to claim 1, characterized in that: The calculation process for the actual physical distance includes: S401, According to the pinhole imaging model formula The center pixel coordinates of the laser point Compared with the first depth value Convert to first three-dimensional space point coordinates; S402, Set the pixel coordinates of the mucosal edge With the second depth value Convert to point coordinates in the second three-dimensional space; S403. Calculate the Euclidean distance between the coordinates of the first three-dimensional space point and the coordinates of the second three-dimensional space point, and use it as the actual physical distance; in, The focal length parameter in the intrinsic parameter matrix, The coordinates of the principal point in the intrinsic parameter matrix are given.

4. The method for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model according to claim 1, characterized in that: The preset safety thresholds and warning thresholds are dynamically adjusted, including: S501. Obtain the current output power and current pulse frequency of the laser host; S502. Calculate the dynamic safety threshold and dynamic warning threshold based on the current output power, current pulse frequency and image clarity evaluation factor of the current video frame. The dynamic safety threshold is positively correlated with the current output power and the current pulse frequency, and the dynamic warning threshold is positively correlated with the current output power.

5. The method for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model according to claim 1 or 4, characterized in that: The determination of the risk level includes: S601. Calculate the membership function value of the actual physical distance relative to the dynamic safety threshold and the dynamic warning threshold, wherein the membership function value represents the first membership degree of the actual physical distance belonging to the safe state, the second membership degree of the actual physical distance belonging to the warning state, and the third membership degree of the actual physical distance belonging to the dangerous state, respectively. S602. Determine the risk level of the current surgical scenario based on the maximum value among the first membership degree, the second membership degree, and the third membership degree.

6. The method for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model according to claim 1, characterized in that: The control commands include: S611. When the state is determined to be safe, generate a first control command that allows full power laser output. S612. When a warning state is determined, a second control command is generated to trigger an audible and visual warning signal and reduce the laser output power. S613. When a dangerous situation is determined, a third control command is generated to urgently interrupt the laser energy output.

7. The method for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model according to claim 6, characterized in that: The second control command includes: reducing the laser output power in stages according to the numerical relationship between the actual physical distance and the dynamic warning threshold; including: When the actual physical distance is greater than or equal to the first distance threshold and less than the dynamic safety threshold, the laser output power is reduced to the first power value; When the actual physical distance is greater than or equal to the dynamic warning threshold and less than the first distance threshold, the laser output power is reduced to the second power value; Wherein, the first distance threshold is between the dynamic warning threshold and the dynamic safety threshold, and the second power value is lower than the first power value.

8. The method for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model according to claim 1, characterized in that: It also includes the following steps: S7. The current video frame, laser point detection box, mucosal region segmentation mask, monocular depth map and actual physical distance are associated and stored in the local database. S8. When the amount of data accumulated in the local database reaches a preset threshold, the data in the local database is used to perform online incremental learning on the multi-task neural network model and update the model parameters. S9. Upload the updated model parameters to the cloud server via wireless network, and receive the latest basic model obtained from global data training based on multiple devices from the cloud server for offline OTA update.

9. A device for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the method for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform the method for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model as described in any one of claims 1 to 8.

11. A system for preventing accidental injury during laser surgery for urinary tract stones based on a visual AI model, characterized in that, include: The system includes a video preprocessing module, a multi-task neural network inference module, a three-dimensional distance calculation module, a dynamic risk assessment module, a laser host interactive control module, and adapter interfaces for communication with the monocular ureteroscope and the laser host, respectively. The video preprocessing module is configured to acquire the surgical field video stream acquired in real time by a monocular ureteroscope and preprocess the current video frame to obtain the image to be identified. The multi-task neural network inference module has a built-in trained multi-task neural network model. The multi-task neural network model is equipped with a shared backbone network and parallel target detection head, instance segmentation head, and monocular depth estimation head connected to the output of the shared backbone network. The multi-task neural network inference module is configured to input the image to be identified into the multi-task neural network model, extract image features through the shared backbone network, process them synchronously through the three parallel task heads, and simultaneously output the laser point detection box, mucosal region segmentation mask, and monocular depth map in the current video frame. The three-dimensional distance calculation module is configured to determine the center pixel coordinates of the laser point based on the laser point detection box, determine the mucosal edge pixel coordinates that are closest to the center pixel coordinates of the laser point based on the mucosal region segmentation mask, obtain the first depth value corresponding to the center pixel coordinates of the laser point and the second depth value corresponding to the mucosal edge pixel coordinates from the monocular depth map, and combine the pre-calibrated ureteroscope intrinsic parameter matrix to convert the two sets of pixel coordinates and corresponding depth values ​​into three-dimensional spatial point coordinates based on the pinhole imaging model, and calculate the actual physical distance between the laser point and the ureteral mucosa. The dynamic risk assessment module is configured to acquire the current output power, current pulse frequency, and image clarity evaluation factor of the current video frame of the laser host, calculate the dynamic safety threshold and dynamic warning threshold, and calculate the membership function value of the actual physical distance relative to the dynamic safety threshold and dynamic warning threshold based on fuzzy logic, and determine the risk level of the current surgical scenario based on the maximum membership value. The laser host interactive control module is configured to generate corresponding control commands based on the determined risk level and send them to the laser host through an adapter interface to control the energy output status of the laser host.

12. A simulation training method for a laser surgery anti-injury system for urinary tract stones based on a visual AI model, characterized in that, Training the multi-task neural network model built into the laser surgery anti-injury system for urinary tract stones as described in claim 11 includes the following steps: T1. Construct a training sample set simulating an endoscopic surgical environment. A specific data augmentation method for urinary endoscopic scenarios is used to amplify the original samples. The specific data augmentation method includes at least one of simulated bleeding occlusion, simulated bubble highlights, and simulated image blur. Each sample in the training sample set is assigned three types of supervision signals: ground truth values ​​of laser point and tissue target detection boxes, ground truth values ​​of ureteral mucosal region segmentation masks, and ground truth values ​​of absolute depth maps. T2. Perform multi-stage training on the multi-task neural network model. The multi-stage training includes, in sequence: a pre-training stage on a large-scale public medical image dataset, a fine-tuning stage on a self-built in vivo and in vitro urinary endoscopic dataset, and an adaptation stage on surgical scenario data from the target hospital. In the pre-training stage, a self-supervised learning approach is adopted, with photometric consistency loss and depth smoothness loss as loss functions, and the model outputs a relative depth map. During the fine-tuning phase, samples with ground truth absolute depth values ​​are introduced for hybrid supervised training. The model parameters are optimized by combining self-supervised loss and absolute depth supervised loss, so that the model outputs an absolute depth map in millimeters. At the same time, for the instance segmentation head, a joint loss function of Dice loss and cross-entropy loss is used to optimize the mucosal region segmentation accuracy, and for the target detection head, bounding box regression loss and classification loss are used to optimize the target detection accuracy. In the adaptation phase, an incremental learning approach is adopted, freezing the shared backbone network parameters of the model and only fine-tuning the top-level parameters of the three parallel task heads to complete the adaptation of the model to the target surgical scenario. T3. Deploy the multi-task neural network model that has completed multi-stage training to the anti-collateral damage system to complete the system's simulation training and deployment.

13. An iterative method for a laser surgery system for urinary tract stones based on a visual AI model to prevent accidental injury, characterized in that, Iterative optimization of the multi-task neural network model in the laser surgery for urinary tract stones prevention system as described in claim 11 includes the following steps: U1. A single anti-miscarriage system will associate the current video frame, laser point detection frame, mucosal region segmentation mask, monocular depth map, actual physical distance and risk level judgment results during the operation in real time, and store them in a local database with encryption. U2. When the amount of data accumulated in the local database reaches a preset threshold, a single anti-mishandling system uses the locally stored data to perform online incremental learning on the built-in multi-task neural network model, freezes the shared backbone network parameters of the model, updates only the top-level parameters of the parallel task head, and generates locally updated model parameters. U3. A single anti-collateral damage system uploads the locally updated model parameters to the cloud server via an encrypted wireless network. The cloud server receives the locally updated model parameters uploaded by multiple anti-collateral damage systems and uses a federated averaging algorithm to fuse multiple sets of locally updated model parameters to generate globally optimized model parameters. U4 and the cloud server distribute the globally optimized model parameters to each anti-collateral damage system. Each anti-collateral damage system receives the globally optimized model parameters during idle periods and completes the offline OTA update of the multi-task neural network model.

Images