A laser triggering control method, system, device, medium, and product

Abstract

This application discloses a laser triggering control method, system, device, medium, and product, relating to the field of medical device technology. This application employs an open-source, promptable concept segmentation model as a zero-sample model, relying on general visual knowledge and semantic understanding capabilities (prompt words) to identify optical fibers, stones, and tissues. Simultaneously, this application separately identifies stones, human tissue, and laser optical fibers. On one hand, it generates laser triggering control commands by combining stone and tissue masks, avoiding accidental damage to body tissues and improving lithotripsy accuracy. On the other hand, it further determines the laser application point (i.e., the laser application location or lithotripsy location) based on the fiber mask and smooths the laser application point. Combining the smoothed application point with the relative positions of the stone and tissue masks generates laser triggering control commands, further improving lithotripsy accuracy. In summary, this application achieves automated, high-precision lithotripsy control of medical lasers.

Description

Technical Field

[0001] This application relates to the field of medical device technology, and in particular to a laser triggering control method, system, device, medium and product. Background Technology

[0002] Stones generally refer to solid masses that form within the cavities of certain organs in the body. The most common are urinary system stones (kidney stones, ureteral stones, etc.), followed by gallstones. Laser lithotripsy (holmium, thulium, green laser, etc.) has gradually become the mainstream treatment for stones due to its advantages such as being minimally invasive, having good hemostasis, and not causing electrical interference.

[0003] In laser lithotripsy, controlling the medical laser is a key challenge. There are two existing control methods: one involves the doctor visually identifying the stones frame by frame and manually operating the laser; the other utilizes neural network models such as Convolutional Neural Networks (CNNs) and U-Nets for stone identification and laser control. Both methods have their limitations.

[0004] The method of doctors visually identifying stones frame by frame and manually operating a medical laser is prone to missing small stones or multiple stones due to the complex internal environment of the human body, numerous mucosal folds, and susceptibility to interference from bleeding, air bubbles, and reflections. Furthermore, frequent switching of the field of view during the procedure makes target tracking difficult, and the laser aiming response is delayed, resulting in low stone fragmentation efficiency and long operation time. The accuracy of this method heavily relies on the doctor's experience, and due to limited equipment and technical capabilities at the grassroots level, poor accessibility to complex stone surgeries, and inconsistent manual operation, laser deviation and damage to normal mucosa are likely to occur.

[0005] The method of using neural network models such as CNN and U-Net for stone identification and medical laser control has poor generalization due to the need for a large amount of labeled data for training, and cannot be applied to diverse scenarios. Summary of the Invention

[0006] The purpose of this application is to provide a laser triggering control method, system, device, medium and product to achieve automated and high-precision lithotripsy control of medical lasers, without the need for extensive training with labeled samples or reliance on physician experience.

[0007] To achieve the above objectives, this application provides the following solution.

[0008] In a first aspect, this application provides a laser triggering control method, comprising: Obtain the current frame image from the endoscope video stream; The open-source, prompt-based concept segmentation method segments the current frame image based on segmentation prompts to obtain fiber optic masks, stone masks, and tissue masks. The laser application point at the current moment is calculated based on the fiber optic mask; the laser application point is the point on the stone or tissue where the laser emitted from the fiber optic cable irradiates. The laser impact point at the current moment is smoothed to obtain the smoothed impact point at the current moment; Based on the relative positions of the smoothing point and the tissue mask at the current moment, and the relative positions of the smoothing point and the stone mask at the current moment, a laser triggering control command is generated.

[0009] Secondly, a laser triggering control system is provided, including: a processing module and a laser triggering module; The processing module is connected to the laser triggering module. The processing module is used to execute the laser triggering control method described above to obtain a laser triggering control command and send the laser triggering control command to the laser triggering module. The laser triggering module is connected to the medical laser and is used to control the medical laser according to the laser triggering control command.

[0010] Thirdly, a computer device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the laser triggering control method described above.

[0011] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the laser triggering control method described above.

[0012] Fifthly, a computer program product is provided, including a computer program that, when executed by a processor, implements the laser triggering control method described above.

[0013] According to the specific embodiments provided in this application, this application has the following technical effects.

[0014] This application provides a laser triggering control method, system, device, medium, and product. It employs an open-source, promptable concept segmentation model as a zero-shot model, relying on general visual knowledge and semantic understanding (prompt words) to identify optical fibers, stones, and tissues. This model is pre-trained using a vast amount of publicly available images, eliminating the need for separate training on large datasets of labeled endoscopic images. Simultaneously, this application separately identifies stones, human tissue, and laser fibers. On one hand, it generates laser triggering control commands by combining stone and tissue masks, avoiding accidental damage to body tissues and improving lithotripsy accuracy. On the other hand, it further determines the laser action point (i.e., the laser action location or lithotripsy location) based on the fiber mask and smooths the laser action point to obtain a smoothed action point. Combining the relative position of the smoothed action point with the stone and tissue masks generates laser triggering control commands, further improving lithotripsy accuracy. In summary, this application achieves automated, high-precision lithotripsy control of medical lasers without requiring extensive labeled sample training or relying on physician experience. Attached Figure Description

[0015] Figure 1 This is a schematic flowchart of a laser triggering control method provided in an embodiment of this application.

[0016] Figure 2 This is a schematic flowchart illustrating another execution process of the laser triggering control method provided in one embodiment of this application.

[0017] Figure 3 This is a schematic diagram of the fitting result of the upper and lower edge lines provided in an embodiment of this application.

[0018] Figure 4 This is a schematic diagram of the laser triggering control process provided in one embodiment of this application.

[0019] Figure 5 This is a schematic diagram of the structure of a laser triggering module provided in one embodiment of this application.

[0020] Figure 6 This is a schematic diagram illustrating the working principle of a laser triggering control system provided in an embodiment of this application.

[0021] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0022] The present application will now be described in conjunction with the accompanying drawings and specific embodiments.

[0023] In one exemplary embodiment, a laser triggering control method is provided, such as... Figure 1 As shown, it includes the following steps 101-105.

[0024] Step 101: Obtain the current frame image from the endoscope video stream; Step 102: Using an open-source promptable concept segmentation model, the current frame image is segmented according to segmentation prompts to obtain fiber optic mask, stone mask, and tissue mask. Step 103: Calculate the laser application point at the current moment based on the above fiber optic mask; the laser application point is the irradiation point of the laser emitted by the fiber optic on the stone or tissue. Step 104: Smooth the laser action point at the current moment to obtain the smoothed action point at the current moment; Step 105: Generate laser trigger control commands based on the relative positions of the smoothing point and the tissue mask at the current moment, and the relative positions of the smoothing point and the stone mask at the current moment.

[0025] Implementing steps 101-105 above enables automated, high-precision lithotripsy control of medical lasers, without requiring extensive training with labeled samples or relying on physician experience.

[0026] In another exemplary embodiment, in order to reduce the amount of computation and improve the recognition speed, the original image needs to be preprocessed in step 101 above. Step 101 above can be implemented by the following steps 201-203.

[0027] Step 201: Crop the original image of the current frame in the endoscope video stream to obtain a first image of a preset size; Step 202: Perform format conversion processing on the first image to obtain a second image that is adapted to the input format of the open-source prompting concept segmentation model; Step 203: Normalize the brightness and contrast of the second image to obtain the normalized second image, which is used as the current frame image.

[0028] Step 201 above involves cropping the original image, removing black borders and the interface around the edges, and retaining only the central, effective endoscopic observation area, processing it to a fixed size (360×360 or 400×400 pixels). Step 202 above converts the image from the camera's BGR format to RGB format to meet the input requirements of the open-source, promptable concept segmentation model. Step 203 above normalizes the image's brightness and contrast to reduce interference from bleed, bubbles, and reflections on the recognition results, ensuring stable subsequent segmentation.

[0029] After preprocessing, the delay in obtaining the current frame image should be controlled within 50ms to meet the real-time requirements of the surgery.

[0030] In another exemplary embodiment, an open-source cue-based concept segmentation model is used to perform zero-sample segmentation of the current frame image. No additional training is required. Optical fibers, stones, and tissues are directly identified by segmentation cue words in different text forms. The segmentation cue words in step 102 above include: a first cue word matching optical fiber attributes, a second cue word matching stone attributes, and a third cue word matching tissue attributes.

[0031] In another exemplary embodiment, step 102 described above may be implemented using steps 301-303.

[0032] Step 301: Input the first cue word and the current frame image into the open-source cue-enabled concept segmentation model to obtain the fiber mask output by the open-source cue-enabled concept segmentation model; Step 302: Input the second cue word and the current frame image into the open-source cue-enabled concept segmentation model to obtain the stone mask output by the open-source cue-enabled concept segmentation model; Step 303: Input the aforementioned third cue word and the current frame image into the open-source cue-enabled concept segmentation model to obtain the tissue mask output by the open-source cue-enabled concept segmentation model.

[0033] In another exemplary embodiment, the first prompt word is set to "blue equipment", and the specific execution process of step 301 above is as follows: Input the text prompt "blue equipment" into the open-source promptable concept segmentation model; The open-source concept segmentation model outputs a binary mask for the fiber region, namely the fiber mask mask_fiber, where a pixel value of 1 indicates that the position belongs to the fiber, and a pixel value of 0 indicates that the position does not belong to the fiber.

[0034] In another exemplary embodiment, the second prompt word is set to "stone", and the specific execution process of step 302 above is as follows: Input the text prompt "stone" into the open-source promptable concept segmentation model; The open-source concept segmentation model can directly output a binary mask of the stone region, namely the stone mask mask_stone.

[0035] The mask with the highest confidence level is retained for subsequent trigger judgment.

[0036] Masking rules: mask_stone(y,x)=1 → This point is a stone; mask_stone(y,x)=0 → This point is not a stone.

[0037] In another exemplary embodiment, to avoid the laser hitting normal tissue (e.g., the bladder wall), step 303 of this application further identifies the tissue. The third prompt word is set to "skin," and the specific execution process of step 303 is as follows: Input the text suggestion "skin" into the open-source suggestive concept segmentation model; The open-source suggestive concept segmentation model directly outputs a binary mask of the tissue region, namely the tissue mask mask_skin.

[0038] During the subsequent generation of laser-triggered control commands, this tissue area is designated as a no-fire zone.

[0039] In another exemplary embodiment, after obtaining the fiber mask, the following steps 401-403 are also set to determine the validity of the fiber mask.

[0040] Step 401: Determine whether the ratio of the area of ​​the fiber mask to the total area of ​​the current frame image is greater than a ratio threshold, or whether the total number of pixels contained in the fiber mask is less than a preset pixel threshold, and obtain a third determination result. Step 402: If the third judgment result is yes, it indicates that the fiber mask is invalid, and return to the step of "acquire current frame image"; Step 403: If the third judgment result is negative, it indicates that the fiber mask is effective; The step of calculating the laser point of action at the current moment based on the fiber mask is executed after the fiber mask becomes effective.

[0041] In another exemplary embodiment, step 401 above is specifically configured as follows.

[0042] If the ratio is too large (for example, the ratio > 25%) or the total number of pixels is too small (for example, the total number of pixels < 2000), it is determined to be an invalid mask and no further calculation is performed.

[0043] In another exemplary embodiment, a method is provided for calculating the laser point of action at the current moment based on the fiber mask, including the following steps 501-504.

[0044] Step 501: Obtain at least two upper edge points and at least two lower edge points of the optical fiber based on the optical fiber mask; Step 502: Construct the equation of the upper edge line of the optical fiber based on at least two upper edge points; Step 503: Construct the equation of the lower edge line of the optical fiber based on at least two lower edge points; Step 504: Calculate the intersection point of the upper edge line equation and the lower edge line equation, and use it as the laser action point at the current moment.

[0045] In another exemplary embodiment, the specific implementation process of step 501 above is as follows: Extract the pixel coordinates (x, y) of all pixels belonging to the fiber from the fiber mask; For each x-coordinate, extract all corresponding y-values: the pixel with the smallest y-value is recorded as the top edge point, and the pixel with the largest y-value is recorded as the bottom edge point; This yields the set of upper edge points and the set of lower edge points of the optical fiber.

[0046] In another exemplary embodiment, in steps 502 and 503 above, the Random Sample Consensus (RANSAC) algorithm is used to fit the straight line of the fiber edge. Its core function is to automatically eliminate erroneous points from the noisy and interfering fiber edge points and fit the straight line that best fits the actual direction of the fiber. This solves the problem that ordinary straight line fitting is easily biased by abnormal points caused by image reflections, bubbles, and mucous membrane wrinkles.

[0047] like Figure 3 As shown, the RANSAC algorithm is used to fit lines to the upper and lower edge points respectively, resulting in two line equations: Equation of the upper edge line: ; Equation of the lower edge line: ; in: , The slope of the line. , It is the line intercept.

[0048] In this embodiment, the RANSAC algorithm can automatically eliminate noise points, making the straight line more closely match the actual direction of the optical fiber.

[0049] In another exemplary embodiment, in step 503 above, the extension direction of the optical fiber can be regarded as the intersection of two edge lines, which is the laser application point. .

[0050] Simultaneous equations: ; Solving for: ; ; Based on the fiber's position in the image (upper half / lower half / center), determine whether the point of action is within a reasonable range to avoid outliers.

[0051] In another exemplary embodiment, an alternative method is provided for calculating the laser point of action at the current moment based on the fiber mask, including steps 601-603 as follows.

[0052] Step 601: Obtain the center points of at least two cross-sections of the optical fiber based on the optical fiber mask; Step 602: Construct the centerline equation of the optical fiber based on the obtained center points; Step 603: Calculate the intersection of the centerline equation with the stone mask or tissue mask, and use it as the laser application point at the current moment.

[0053] In another exemplary embodiment, due to image jitter and fiber optic sway, the directly calculated laser impact point will fluctuate. Therefore, the laser impact point is smoothed in step 103 above. For example, in this embodiment, a two-dimensional Kalman filter is used to smooth the laser impact point at the current moment to obtain the smoothed impact point at the current moment, specifically including the following steps 701-703.

[0054] Step 701: Based on the state vector of the smoothing point of action at the previous time step, predict the state vector of the prior prediction point of action at the current time step using the following formula. ; in, For the previous moment Smooth the state vector at the point of application. For the current moment The state vector of the point of application is predicted prior to the prior knowledge. This is the state transition matrix; Step 702: If the laser action point at the current moment is invalid, then output the prior predicted action point at the current moment as the smooth action point at the current moment. Step 703: If the laser action point at the current moment is valid, then the prior predicted action point at the current moment is corrected using the observed state vector of the laser action point at the current moment, and the posterior predicted action point at the current moment is obtained as the smooth action point at the current moment.

[0055] Step 703 above can be implemented using steps 801-805 as follows.

[0056] Step 801: Based on the state vector of the prior prediction point of action at the current time, calculate the prior covariance at the current time using the following formula; ; in, For the current moment The prior covariance, For the previous moment The posterior covariance, i.e., the reliability, This is process noise, generated by system jitter; Step 802: Calculate the Kalman gain at the current time using the following formula based on the prior covariance at the current time. ; in, For the current moment Kalman gain, For the measurement matrix, To observe noise; Step 803: Based on the Kalman gain at the current time and the observed state vector of the laser point of action at the current time, calculate the state vector and posterior covariance of the predicted point of action at the current time. ; ; in, For the current moment The posterior prediction of the state vector at the point of action. For the current moment The observed state vector at the laser point of action. It is the identity matrix. For the current moment The posterior covariance; Step 804: Output the posterior prediction action point at the current time, as the smoothing action point at the current time.

[0057] The state vector in the above embodiment includes: position x, y, and velocity. , : ; in, Let be the state vector at any point. , These are the x-axis and y-axis coordinates of the point. , Let x be the velocity of this point in the x-axis direction and y-axis direction.

[0058] The state transition matrix described above is expressed as follows: ; in, This represents the inter-frame time interval.

[0059] The measurement matrix described above is represented as follows: .

[0060] Steps 701-703 and 801-804 above can be summarized as follows: Determine whether the laser's point of action at the current moment is valid (within a reasonable area of ​​the image and with valid coordinates; a reasonable area of ​​the image means that the minimum distance between each point on the edge and the edge of the image is greater than a distance threshold; valid coordinates mean that the coordinates are within a reasonable area of ​​the image and the coordinate values ​​are greater than or equal to 0). If effective: Perform the update step, correct with the observed values, and output the posterior prediction action point as the smoothing action point; If invalid: do not perform update, directly use the predicted value (i.e., the prior prediction point of action) as the output to ensure that the trajectory is continuous and uninterrupted.

[0061] In another exemplary embodiment, such as Figure 4 As shown, the judgment criteria in step 105 above during the generation of laser trigger control commands are as follows: If the smoothing point falls within the tissue mask, triggering is disabled; If the smoothing point is not within the tissue mask but falls within the stone mask, triggering is allowed; It will not be triggered in other situations.

[0062] In another exemplary embodiment, step 105 above may be implemented using steps 901-905.

[0063] Step 901: Determine whether the laser application point at the current moment is located on the tissue mask, and obtain the first determination result; Step 902: If the first judgment result is yes, then a low-level laser trigger control command is generated; Step 903: If the first judgment result is negative, then determine whether the laser action point at the current moment is located on the stone mask to obtain the second judgment result; Step 904: If the second judgment result is yes, then a high-level laser trigger control command is generated; Step 905: If the second judgment result is negative, a low-level laser trigger control command is generated. A high-level laser trigger control command is used to control a medical laser to emit a single pulse. Low-level laser trigger control commands are used to control the latching pulse output of medical lasers.

[0064] In another exemplary embodiment, the laser triggering control method described above further includes: When the excitation completion feedback signal is received from the laser trigger module, the feedback image after excitation is obtained from the endoscope video stream; the laser trigger module is used to receive and control the medical laser according to the laser trigger control command, and output the excitation completion feedback signal when the excitation is completed. An open-source cue-based concept segmentation model is used to segment the feedback image based on the second cue word matching the stone attribute to obtain a feedback stone mask; The feedback stone mask is compared with the target stone mask to determine the stone fragmentation result; the target stone mask is the stone mask used when generating the laser trigger control command. If the lithotripsy result characterization fails to hit the target, the process returns to the step of "acquiring the current frame image from the endoscopic video stream." After "generating a laser trigger control command based on the relative position of the smoothing point and the tissue mask at the current moment, and the relative position of the smoothing point and the stone mask at the current moment," a manual confirmation step is added. This added manual confirmation step is... Figure 2 The process of determining whether to "confirm activation" can be implemented using a pop-up window or other methods, which are not limited here.

[0065] The manual confirmation step is cancelled upon receiving a cancellation command from a user with the target permissions. This user can be a physician of a certain level or a staff member responsible for software development and operation; no specific limitation is made here.

[0066] In this embodiment, the laser triggering position is determined based on the wall tube extension point (i.e., the smoothing action point). First, it is determined whether the laser hits the tissue mask. If the wall tube extension point exists in the tissue mask, it is considered that the laser may hit the inner wall of the tissue or organ. Then, the current position is discarded and the process continues. If the extension point of the wall tube hits the stone mask, it is considered to have hit the stone part, and a signal that can trigger the laser is generated, namely a high-level laser trigger control command. This embodiment integrates the smoothing action point, stone mask, and tissue mask to determine whether the medical laser can be fired at the current moment, which can improve the accuracy of lithotripsy and avoid including tissue.

[0067] In another exemplary embodiment, the above-mentioned open-source suggestible concept segmentation model may be the open-source SAM3 (Segment Anything Model 3) model, but is not limited to the SAM3 model.

[0068] Based on the same inventive concept, this application also provides a laser triggering control device for implementing the laser triggering control method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more laser triggering control device embodiments provided below can be found in the limitations of the laser triggering control method described above, and will not be repeated here.

[0069] In an exemplary embodiment, a laser trigger control device is provided, including: a processing module and a laser trigger module; the processing module is connected to the laser trigger module, and the processing module is configured to execute the laser trigger control method in the above embodiment to obtain a laser trigger control instruction and send the laser trigger control instruction to the laser trigger module; the laser trigger module is connected to a medical laser, and the laser trigger module is configured to control the medical laser according to the laser trigger control instruction.

[0070] In another exemplary embodiment, the above laser trigger control system further includes an endoscope image acquisition system; the endoscope image acquisition system is connected to the processing module, and the endoscope image acquisition system is configured to acquire an endoscope video stream and send the endoscope video stream to the processing module.

[0071] The endoscope image acquisition system acquires real-time intraoperative images (i.e., the endoscope video stream) and transmits the images to the processing module. The processing module performs stone detection and surgical trigger condition judgment based on an open-source promptable concept segmentation model. When the excitable condition is met, the processing module sends an instruction to the laser trigger module through the communication module, causing the laser trigger module to drive the medical laser for laser irradiation to achieve the effect of hitting the stone.

[0072] The endoscope image acquisition system includes an image acquisition module and an image processing module. The image acquisition module includes an endoscope, and the image processing module is an image processing system配套 with the endoscope. Data acquisition does not require an additional image processing protocol. The endoscope video stream is acquired by the endoscope image acquisition system and output to the processing module to obtain the current frame image. After testing, the delay is below 50 ms, meeting the low-delay usage scenario. The specific process of the test is as follows: Test whether the delay time of the current frame image is less than a preset time threshold (exemplarily 50 ms); If not, it indicates that the current frame image does not meet the delay requirement. Wait to acquire the next current frame image and return to the step of "testing whether the delay time of the current frame image is less than the preset time threshold"; If so, it indicates that the current frame image meets the delay requirement, and the step of "segmenting the current frame image according to the segmentation prompt words using an open-source promptable concept segmentation model to obtain a fiber mask, a stone mask, and a tissue mask" is executed after determining that the current frame image meets the delay requirement.

[0073] In another exemplary embodiment, the above laser trigger module has communication and medical laser excitation functions.

[0074] The processing module requires relatively high computing power, so a computer with image processing and control functions is selected. The laser trigger module only needs to receive signals and trigger signal output, and the selected model is the STM32F103C8T6 processor of ST company, such as Figure 5 As shown, the STM32F103C8T6 processor comes with a Universal Asynchronous Receiver / Transmitter (UART) and a General-Purpose Input / Output (GPIO) port, which can perform simple signal processing and signal triggering functions.

[0075] In another exemplary embodiment, the STM32F103C8T6 processor described above serves as the lower-level control core, and the computer described above serves as the upper-level computer.

[0076] The STM32F103C8T6 processor is responsible for receiving instructions from the host computer and executing laser output control.

[0077] In another exemplary embodiment, such as Figure 6 As shown, the working principle of the laser triggering control system described above is as follows: After the host computer identifies and locates the stone region in the tissue endoscopy image (i.e., the current frame image) using an open-source, cue-based concept segmentation model, it determines whether the laser emission conditions are met based on the stone location and safe distance. When emission is permitted, the host computer sends the character command "1" to the slave computer via serial communication; if the identification result indicates a non-stone region, excessive distance, or a safety lock state, other commands are sent or no valid trigger signal is sent.

[0078] The STM32F103C8T6 controller monitors serial port data in real time and only responds to valid "1" (i.e., high level) commands received from the serial port. When the signal is detected, a control level is immediately output. Depending on the driving method of the medical laser, if the medical laser has a level drive control unit, it will directly drive the medical laser to emit laser light. If the medical laser does not have a level drive control unit, it will drive the medical laser through a relay to turn on the medium-low intensity medical laser and start the internal timer for 100ms timing control. After the timing ends, the laser output will be automatically turned off, completing one safe pulse emission.

[0079] If the serial port receives a signal other than "1" or invalid data, the STM32F103C8T6 controller will not execute the laser trigger action and will remain in the original standby state to avoid accidental triggering or irradiation of normal bladder mucosa tissue.

[0080] The laser triggering control system provided in this application achieves rapid response, precise time and width control, and high safety of laser triggering through dual hardware timing and software interlocking. At the same time, it simplifies the system structure, reduces costs, and facilitates integration with endoscope systems.

[0081] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 7 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a laser triggering control method.

[0082] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0083] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0084] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0085] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0086] In another exemplary embodiment, the following experiment is provided to verify the technical effect of the technical solution provided in this application.

[0087] The data in this experiment were obtained from multiple sets of simulated videos and clinical videos of cystoscopic surgery.

[0088] The simulation video used a realistic silicone kidney model and a mixture of sand and flour to simulate kidney stones in different states. A total of 10 effective videos were captured, covering typical clinical scenarios such as different lighting conditions, air bubble interference, mucosal folds, and changes in stone size and location. The total duration was approximately 30 minutes, and the number of effective image frames for analysis exceeded 3000 frames.

[0089] Running in a unified hardware environment with fixed parameter configurations: image resolution 360×360, open-source cue-enabled concept segmentation model using zero-shot segmentation, cue words "stone," "blue equipment," and "skin," laser triggering module using an ESP32 (Espressif Systems 32-bit Series) controller, serial communication baud rate of 9600, and laser pulse width of 100ms. All data was automatically recorded and statistically analyzed.

[0090] 1. Accuracy of stone identification Judgment criteria: The manually labeled stone area is taken as the real result. If the overlap rate between the output stone mask and the labeled area is ≥50%, it is judged as successful recognition; otherwise, it is judged as recognition failure.

[0091] Calculation formula: Stone identification accuracy = Number of successfully identified images ÷ Total number of test images × 100%.

[0092] 2. Fiber optic positioning success rate Judgment criteria: If the optical fiber appears in the frame and can output a valid optical fiber mask, the positioning is considered successful; if no optical fiber is detected or the mask is invalid (too small / too large area), the positioning is considered a failure.

[0093] Calculation formula: Fiber optic positioning success rate = Number of effective positioning images by fiber optic cable ÷ Total number of images appearing by fiber optic cable × 100%.

[0094] 3. Average tracking error of the laser application point Judgment criteria: Using the manually marked intersection of fiber optic extensions as the standard action point, the pixel distance between the aforementioned smoothing action point and the standard action point is calculated as the single-frame error. The average error of all valid frames is then calculated.

[0095] Calculation formula: Single frame error = ;in, and The x-axis and y-axis coordinates of the smoothing application point. , The x-axis and y-axis coordinates of the standard point of application.

[0096] Average tracking error = Sum of errors in all frames ÷ Number of valid frames 4. System false trigger rate Judgment criteria: If the smoothing point is located in the bladder mucosa or normal tissue area, but still emits a trigger signal, it is judged as a false trigger.

[0097] Calculation formula: False trigger rate = Number of false triggers ÷ Total number of judgments × 100%.

[0098] Based on multiple sets of clinical video tests, the key indicators of the technical solution in this application are as follows: Stone identification accuracy: 87.2%; Fiber optic positioning success rate: 98.5%; Average tracking error of laser action point: 2.4 pixels; System false trigger rate: 0.3%; Average processing time per frame: 76ms.

[0099] Experiments show that the technical solution of this application can complete the segmentation and identification of stones, optical fibers, and mucosal tissue under cystoscopy in real time under zero-sample and no-training conditions. The optical fiber positioning is stable, the laser action point tracking accuracy is high, the false trigger rate is low, and the overall processing speed meets the real-time requirements of the operation. The ESP32 hardware trigger response is fast and the timing is accurate, enabling safe and reliable automatic laser lithotripsy control.

[0100] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0101] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0102] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchain. The processors involved in the embodiments provided in this application may be, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc.

[0103] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0104] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A laser triggering control method, characterized in that, include: Obtain the current frame image from the endoscope video stream; An open-source, promptable concept segmentation model is used to segment the current frame image based on segmentation prompts to obtain fiber optic masks, stone masks, and tissue masks. The laser application point at the current moment is calculated based on the fiber optic mask; the laser application point is the point where the laser emitted from the fiber optic cable irradiates the stone or tissue. The laser impact point at the current moment is smoothed to obtain the smoothed impact point at the current moment; Based on the relative positions of the smoothing point and the tissue mask at the current moment, and the relative positions of the smoothing point and the stone mask at the current moment, a laser triggering control command is generated.

2. The laser triggering control method according to claim 1, characterized in that, The segmentation prompts include: a first prompt matching fiber optic properties, a second prompt matching stone properties, and a third prompt matching body tissue properties; The method employs an open-source, prompt-based concept segmentation model to segment the current frame image based on segmentation prompts, obtaining fiber optic masks, stone masks, and tissue masks. Specifically, this includes: The first cue word and the current frame image are input into the open-source cue-enabled concept segmentation model to obtain the fiber mask output by the open-source cue-enabled concept segmentation model. The second cue word and the current frame image are input into the open-source cue-enabled concept segmentation model to obtain the stone mask output by the open-source cue-enabled concept segmentation model; The third cue word and the current frame image are input into an open-source cue-enabled concept segmentation model to obtain the tissue mask output by the open-source cue-enabled concept segmentation model.

3. The laser triggering control method according to claim 1, characterized in that, The laser point of action at the current moment is calculated based on the fiber mask, specifically including: At least two upper edge points and at least two lower edge points of the optical fiber are obtained based on the optical fiber mask; Construct the equation of the upper edge line of the optical fiber based on at least two upper edge points; Construct the equation of the lower edge line of the optical fiber based on at least two lower edge points; The intersection point of the equations of the upper edge line and the lower edge line is calculated and used as the laser application point at the current moment.

4. The laser triggering control method according to claim 1, characterized in that, The laser point of action at the current moment is calculated based on the fiber mask, specifically including: The center points of at least two cross-sections of the optical fiber are obtained based on the optical fiber mask; Construct the centerline equation of the optical fiber based on the obtained center points; The intersection of the centerline equation and the stone mask or the tissue mask is calculated and used as the laser application point at the current moment.

5. The laser triggering control method according to claim 1, characterized in that, Based on the relative positions of the current smoothing point and the tissue mask, and the current smoothing point and the stone mask, a laser triggering control command is generated, specifically including: Determine whether the laser's point of action is located on the tissue mask at the current moment, and obtain a first determination result; If the first judgment result is yes, then a low-level laser trigger control command is generated; If the first judgment result is negative, then determine whether the laser action point at the current moment is located on the stone mask to obtain the second judgment result; If the second judgment result is yes, then a high-level laser trigger control command is generated; If the second judgment result is negative, a low-level laser trigger control command is generated; The high-level laser trigger control command is used to control the medical laser to emit a single pulse. The low-level laser trigger control command is used to control the lockout pulse output of the medical laser.

6. The laser triggering control method according to claim 1, characterized in that, Obtain the current frame image from the endoscopic video stream, specifically including: The original image of the current frame in the endoscope video stream is cropped to obtain a first image of a preset size; The first image is converted to obtain a second image that is adapted to the input format of the open-source prompting concept segmentation model; The second image is normalized in terms of brightness and contrast to obtain a normalized second image, which is then used as the current frame image.

7. The laser triggering control method according to claim 1, characterized in that, The laser point of application at the current moment is calculated based on the fiber mask, and prior to this, the following steps are also included: Determine whether the ratio of the area of ​​the fiber optic mask to the total area of ​​the current frame image is greater than a ratio threshold, or whether the total number of pixels contained in the fiber optic mask is less than a preset pixel threshold, and obtain a third determination result. If the third judgment result is yes, it indicates that the fiber mask is invalid, and the process returns to the step of "obtaining the current frame image from the endoscope video stream"; If the third judgment result is negative, it indicates that the fiber mask is effective; The step of calculating the laser point of action at the current moment based on the fiber mask is executed after the fiber mask becomes effective.

8. The laser triggering control method according to claim 1, characterized in that, The laser's point of impact at the current moment is smoothed to obtain a smoothed point of impact at the current moment. Specifically, this includes: The laser impact point at the current moment is smoothed using a two-dimensional Kalman filter to obtain the smoothed impact point at the current moment.

9. The laser triggering control method according to claim 8, characterized in that, The laser impact point at the current moment is smoothed using a two-dimensional Kalman filter to obtain the smoothed impact point at the current moment. Specifically, this includes: Based on the state vector of the smoothing point of action at the previous time step, the state vector of the prior prediction point of action at the current time step is predicted using the following formula. ； in, For the previous moment Smooth the state vector at the point of application. For the current moment The state vector of the point of application is predicted prior to the prior knowledge. This is the state transition matrix; If the laser action point at the current moment is invalid, then the prior predicted action point at the current moment is output as the smooth action point at the current moment. If the laser impact point at the current moment is valid, the prior predicted impact point at the current moment is corrected using the observed state vector of the laser impact point at the current moment, and the posterior predicted impact point at the current moment is obtained as the smoothed impact point at the current moment.

10. The laser triggering control method according to claim 9, characterized in that, The prior predicted point of action at the current moment is corrected using the observed state vector of the laser's point of action at the current moment, resulting in the posterior predicted point of action at the current moment, which is then used as the smoothed point of action at the current moment. Specifically, this includes: Based on the state vector of the predicted point of action at the current time, the prior covariance at the current time is calculated using the following formula; ； in, For the current moment The prior covariance, For the previous moment The posterior covariance, This is process noise; Based on the prior covariance at the current moment, the Kalman gain at the current moment is calculated using the following formula; ； in, For the current moment Kalman gain, For the measurement matrix, To observe noise; Based on the Kalman gain at the current moment and the observed state vector of the laser point of action at the current moment, calculate the state vector and posterior covariance of the predicted point of action at the current moment. ； ； in, For the current moment The posterior prediction of the state vector at the point of action. For the current moment The observed state vector at the laser point of action. It is the identity matrix. For the current moment The posterior covariance; Output the posterior prediction action point at the current time step, which will be used as the smoothing action point at the current time step.

11. The laser triggering control method according to claim 1, characterized in that, Also includes: When the excitation completion feedback signal is received from the laser trigger module, the feedback image after excitation is obtained from the endoscope video stream; the laser trigger module is used to receive and control the medical laser according to the laser trigger control command, and output the excitation completion feedback signal when the excitation is completed. An open-source cue-based concept segmentation model is used to segment the feedback image based on the second cue word matching the stone attribute to obtain a feedback stone mask; The feedback stone mask is compared with the target stone mask to determine the stone fragmentation result; the target stone mask is the stone mask used when generating the laser trigger control command. If the lithotripsy result characterization fails to hit, the process returns to the step of "obtaining the current frame image from the endoscopic video stream" and adds a manual confirmation step after "generating a laser trigger control command based on the relative position of the smoothing point at the current moment and the relative position of the smoothing point at the current moment and the stone mask".

12. The laser triggering control method according to claim 11, characterized in that, The manual confirmation step is canceled upon receiving a cancellation command from a user with the target permissions.

13. A laser triggering control system, characterized in that, include: Processing module and laser triggering module; The processing module is connected to the laser triggering module, and the processing module is used to execute the laser triggering control method according to any one of claims 1-12 to obtain a laser triggering control command and send the laser triggering control command to the laser triggering module; The laser triggering module is connected to the medical laser and is used to control the medical laser according to the laser triggering control command.

14. The laser triggering control system according to claim 13, characterized in that, The laser triggering control system also includes an endoscope image acquisition system; The endoscopic image acquisition system is connected to the processing module. The endoscopic image acquisition system is used to acquire endoscopic video streams and send the endoscopic video streams to the processing module.

15. The laser triggering control system according to claim 13, characterized in that, Before segmenting the current frame image using an open-source cue-based concept segmentation model based on segmentation cues to obtain fiber optic masks, stone masks, and tissue masks, the processing module is further configured to: Test whether the delay time of the current frame image is less than a preset time threshold; If not, it indicates that the current frame image does not meet the delay requirement, wait to obtain the next current frame image, and return to the step of "testing whether the delay time of the current frame image is less than the preset time threshold"; If so, it indicates that the current frame image meets the delay requirement. The step of "using an open-source cue-based conceptual segmentation model to segment the current frame image according to segmentation cue words to obtain fiber optic mask, stone mask and tissue mask" is executed after determining that the current frame image meets the delay requirement.

16. The laser triggering control system according to claim 13 or 15, characterized in that, The processing module uses a computer with image processing and control functions.

17. The laser triggering control system according to claim 13, characterized in that, The laser triggering module includes an STM32F103C8T6 processor. The UART interface of the STM32F103C8T6 processor is connected to the processing module, and the GPIO interface of the STM32F103C8T6 processor is connected to the medical laser.

18. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the laser triggering control method according to any one of claims 1-12.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the laser triggering control method according to any one of claims 1-12.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the laser triggering control method according to any one of claims 1-12.

Images