Laser point tracking method based on endoscope sequence image, tracker and lithotripsy system

Abstract

The present application relates to the technical field of laser point tracking, in particular to a laser point tracking method based on endoscope sequence diagram, a tracker and a lithotripsy system, which solves the technical problem of the prior art that single frame detector is easy to cause medical accidents when detecting stones, and the confidence of the single frame detector in detecting the laser point will drop instantly when the derivative drifts, damaging the laser generator source. The laser point tracking method based on endoscope sequence diagram provided by the present application adopts two-stage matching, can find back the target laser point lost due to the shielding of the shielding object, improves the accuracy of tracking the target laser point, can accurately crush the stones, greatly reduces the probability of injury, and improves the safety of the operation. And due to the accuracy of tracking the target laser point, the control signal output by the control system of the laser control is stable, reducing the damage probability of the laser generator.

Description

Technical Field

[0001] This invention relates to the field of laser spot tracking technology, and more specifically, to a laser spot tracking method, tracker, and lithotripsy system based on endoscopic sequence maps. Background Technology

[0002] Laser lithotripsy is one of the mainstream surgical methods for treating urinary tract stones such as kidney stones, ureteral stones, and bladder stones. In this procedure, the doctor first inserts a thin optical fiber into the location of the stone using an endoscope, and then, under endoscopic monitoring, uses the thermal effect of a laser to directly break up the stone.

[0003] However, at the moment of laser excitation, the surface of the stone undergoes thermal ablation and mechanical disintegration, generating a large amount of dust, micro-debris, and cavitation bubbles. These byproducts can obstruct the endoscope's camera field of view or cover the target area for an extremely short time (milliseconds).

[0004] During the procedure, a sheath or other foreign object is introduced to remove stones or fragments. These objects can also interfere with the images captured by the endoscope.

[0005] Under certain pathological conditions (such as ureteral stricture, bleeding, or polyps), the color and texture of the stone surface can be easily confused with the surrounding soft tissues (such as the ureteral wall and renal calyx mucosa), making it difficult to instantly distinguish whether the bright reflective spots come from the stone or the moist mucosa.

[0006] In summary, when a single-frame detector is used to detect stones in images acquired by an endoscope, the stones are directly broken up using the thermal effect of a laser upon detection. However, if the images acquired by the endoscope are interfered with, they may be misidentified as stones. If the system identifies a stone as such, it may instantly fire a laser to break it up, potentially causing a medical accident.

[0007] In addition, the patient's respiratory movements, the impact of the irrigation fluid, and the slight hand tremors caused by prolonged operation by the doctor can all cause high-frequency displacement of the relative position between the laser fiber tip and the stone. The laser point may slip from the stone surface to the ureteral wall within tens of milliseconds, causing a medical accident.

[0008] At the moment of laser excitation, the surface of the stone undergoes thermal ablation and mechanical disintegration, generating a large amount of dust, tiny debris, and cavitation bubbles. When these derivatives drift by, the confidence level of the single-frame detector when detecting the laser point drops instantly, causing the laser control system to output a high-frequency pulse signal of "on-off-on-off". This not only damages the laser generator source, but may also cause medical accidents by firing the laser at a moment when it is mistakenly judged to be safe. Summary of the Invention

[0009] The purpose of this invention is to provide a laser spot tracking method, tracker, and lithotripsy system based on endoscopic sequence maps, which solves the technical problems in the prior art where false detection when using a single-frame detector to detect stones can easily lead to medical accidents; in addition, when derivatives drift by, the confidence of the single-frame detector when detecting laser spots will drop instantly, damaging the laser generator source.

[0010] As a first aspect of the present invention, the present invention provides a laser point tracking method based on endoscopic sequence maps, comprising: Based on the Kalman filter, the predicted position of the target laser point in the current frame image is predicted according to multiple sequence frame images prior to the current moment; Based on a single-target tracking algorithm, target laser points in the current frame image are tracked and detected to obtain a first detection box and a second detection box. The confidence of the first detection box is greater than that of the second detection box. Match the first detection box with the predicted position; When the matching degree between the first detection box and the predicted position is less than or equal to the first preset value, the second detection box is matched with the predicted position. When the matching degree between the second detection box and the predicted position is greater than the second preset value, the motion trajectory of the target laser point is updated according to the second detection box.

[0011] In one embodiment of the present invention, predicting the predicted position of the target laser point in the current frame image based on a Kalman filter and multiple sequence frames prior to the current time includes: Construct a state-space model, wherein the state-space model defines the state vector of the target laser point; Based on the uniform motion model, the current state vector and current error covariance matrix of the target laser point in the current frame image are predicted according to the state vector of the target laser point in the previous frame image. The predicted position of the target laser point in the current frame image is determined based on the current state vector of the target laser point in the current frame image. The state vector of the target laser point can be denoted as:

[0012] In the formula, Let be the center coordinates of the target laser point, and α be the aspect ratio of the frame image. h The height of the frame image. The velocity component of the target laser point; The formula for calculating the current state vector is:

[0013] Among them, FK Here is the state transition matrix. The current state vector of the target laser point; The formula for calculating the current error covariance matrix is: ; in, Let F be the current error covariance matrix. K Here is the state transition matrix. Q K This is the current noise covariance matrix.

[0014] In one embodiment of the present invention, predicting the predicted position of the target laser point in the current frame image based on a Kalman filter and multiple sequence frame images prior to the current time further includes: The current velocity of the target laser point is determined based on the current state vector of the target laser point in the current frame image. When the current velocity of the target laser point is greater than a preset velocity threshold, the weight corresponding to the velocity term in the current noise covariance matrix is ​​increased to update the current noise covariance matrix; The current error covariance matrix is ​​updated based on the updated current noise covariance matrix.

[0015] In one embodiment of the present invention, predicting the predicted position of the target laser point in the current frame image based on a Kalman filter and multiple sequence frame images prior to the current time further includes: Obtain the current observation value of the uniform motion model when predicting the current state vector of the target laser point in the current frame image; Calculate the current Kalman gain based on the current observations; The current state vector of the target laser point in the current frame image is updated based on the current Kalman gain; The formula for calculating the current Kalman gain is as follows:

[0016] Where, in the formula, K k For the current Kalman gain, Given the current error covariance matrix, R k H is the current noise matrix, and H is the observation matrix. It is the transpose of H; The update formula for updating the current state vector based on the current Kalman gain is as follows:

[0017] Where, in the formula, This is the updated current state vector. The current observation value is the current state vector. The current state vector before the update. K k This represents the current Kalman gain.

[0018] In one embodiment of the present invention, the laser point tracking method further includes: Obtain the confidence level corresponding to the detection box that matches the predicted position; The confidence level is then back-mapped to the current noise matrix.

[0019] In one embodiment of the present invention, the confidence level of the first detection box is greater than a first preset threshold, and the confidence level of the second detection box is less than or equal to the first preset threshold and greater than a second preset threshold.

[0020] In one embodiment of the present invention, the confidence level of the first detection box is greater than a first preset threshold, and the confidence level of the second detection box is within a preset range. The preset range is based on the confidence level distribution range of the detection boxes corresponding to the target laser point being occluded by an obstacle in the historical frame image.

[0021] In one embodiment of the present invention, the laser point tracking method further includes: When the matching degree between the first detection box and the predicted position is greater than a first preset value, the motion trajectory of the target laser point is updated according to the first detection box.

[0022] As a second aspect of the present invention, the present invention also provides a laser point tracker based on an endoscopic sequence map, comprising: A Kalman filter is used to predict the predicted position of the target laser point in the current frame image based on multiple sequence frames of images prior to the current moment. A neural network model is used to track and detect target laser points in the current frame image based on a single target tracking algorithm, and obtain a first detection box and a second detection box, wherein the confidence of the first detection box is greater than the confidence of the second detection box. A matching unit is configured to match the first detection box with the predicted position; The trajectory determination unit is used to match the second detection box with the predicted position when the matching degree between the first detection box and the predicted position is less than the first preset value, and to update the motion trajectory of the target laser point according to the second detection box when the matching degree between the second detection box and the predicted position is greater than or equal to the second preset value.

[0023] As a second aspect of the invention, the invention also provides an endoscopic lithotripsy system, comprising: A laser generator is used to emit a laser beam to the target detection area to form a laser spot; An endoscope is used to acquire images of the target detection area; and The laser point tracker described above.

[0024] This invention provides a laser point tracking method based on endoscopic sequence images. First, a Kalman filter is used to predict the predicted position of the target laser point in the current frame image based on multiple sequence frames from previous times. Then, a single-target tracking algorithm is used to determine the high-resolution and low-resolution detection boxes in the current frame image. Next, a two-stage matching process is employed: first, the high-resolution detection box is matched with the predicted position; if the matching fails, the low-resolution detection box is then matched with the predicted position. This method can recover target laser points lost due to obstruction and re-attach them to the target laser point's trajectory, improving the accuracy of target laser point tracking and ensuring the continuity of target laser point tracking. Thus, regardless of whether the stone is obstructed or resembles other tissues, the accuracy of target laser point tracking allows for accurate stone fragmentation, significantly reducing the probability of accidental injury and improving surgical safety. Furthermore, the accuracy of target laser point tracking ensures stable control signals output by the laser control system, reducing the probability of laser generator damage. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0026] Figure 1 The diagram shown is a flowchart of a laser point tracking method based on an endoscope sequence map according to an embodiment of the present invention.

[0027] Figure 2 The diagram shown is a flowchart of a laser point tracking method based on an endoscope sequence map provided in the second embodiment of the present invention.

[0028] Figure 3 The diagram shown is a flowchart of a laser point tracking method based on an endoscope sequence map provided in the third embodiment of the present invention.

[0029] Figure 4The diagram shown is a flowchart of a laser point tracking method based on an endoscope sequence map provided in the fourth embodiment of the present invention.

[0030] Figure 5 The diagram shown is a working block diagram of a laser point tracker based on an endoscope sequence map according to an embodiment of the present invention.

[0031] Figure 6 The diagram shown is a working block diagram of an endoscopic lithotripsy system provided in an embodiment of the present invention. Detailed Implementation

[0032] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] In the description of this invention, it should be noted that the terms "upper", "lower", "front", "horizontal", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0034] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the term "installation" should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral connection; it can refer to a mechanical connection or an electrical connection; it can refer to a direct connection or an indirect connection through an intermediate medium; it can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0035] Exemplary methods As a first aspect of the present invention, the present invention provides a laser point tracking method based on endoscopic sequence maps. Figure 1 The diagram shown is a flowchart illustrating a laser point tracking method based on an endoscopic sequence map according to an embodiment of the present invention. Figure 1 As shown, the laser point tracking method based on endoscopic sequence maps provided by the present invention includes the following steps S100-S800: S100: Based on the Kalman filter, predict the predicted position of the target laser point in the current frame image according to multiple sequence frame images before the current moment; Specifically, during the process of breaking up stones using an endoscope, images of the target detection area are acquired in real time to form a sequence of frame images.

[0036] Multiple sequential frame images are input into a Kalman filter. The Kalman filter can detect the target laser point in each frame image within a preset time period, and form the historical motion trajectory of the target laser point within the preset time period based on the state of the target laser point in each frame image. Based on the historical motion trajectory, the predicted position of the target laser point in the current frame image detected at the current moment can be predicted.

[0037] S200: Based on a single-target tracking algorithm, target laser points in the current frame image are tracked and detected to obtain a first detection box and a second detection box. The confidence of the first detection box is greater than that of the second detection box. Specifically, the single-target tracking algorithm can track target laser points in a continuous sequence of image frames. Multiple candidate detection boxes are selected in the current frame image based on the target detection box. The confidence score of each candidate detection box is calculated, and a first or second detection box is determined from among the multiple candidate detection boxes based on the confidence score, where the confidence score of the first detection box is greater than that of the second detection box.

[0038] Optionally, the first and second detection boxes are determined as follows: (1) Determine candidate detection boxes with a confidence level greater than the first preset threshold as first detection boxes, and determine candidate detection boxes with a confidence level less than or equal to the first preset threshold and greater than the second preset threshold as second detection boxes: Candidate detection boxes with a confidence level greater than a first preset threshold are identified as the first detection box, i.e., the first detection box is a high-scoring detection box. Candidate detection boxes with a confidence level less than or equal to the first preset threshold and greater than a second preset threshold are identified as the second detection box, i.e., the second detection box is a low-scoring detection box.

[0039] Specifically, confidence level refers to the degree to which a candidate detection box is a target laser point. The first preset threshold can be 0.9, meaning that if the first detection box is a high-scoring detection box, the confidence level of it being a target laser point is relatively high. Similarly, the second preset threshold can be 0.3, meaning that if the second detection box is a low-scoring detection box, the confidence level of it being a target laser point is relatively low.

[0040] At the moment of laser excitation, the surface of the stone undergoes thermal ablation and mechanical disintegration, generating a large amount of dust, micro-debris, and cavitation bubbles. When these derivatives drift past the target laser point, the confidence level of the candidate detection box at that location is low, indicating that the target laser point is obscured. In other words, the target laser point is obscured by dust, debris, smoke, and other substances, resulting in low confidence and ultimately leading to its loss.

[0041] Therefore, the second preset threshold can be based on the actual confidence level of candidate detection boxes in historical frame images where the target laser point is obscured by dust, debris, smoke, or other objects. For example, if the actual confidence level of a candidate detection box where the target laser point is obscured by dust, debris, smoke, or other objects is 0.3, then the second preset threshold can be set to a value less than 0.3. In this way, the candidate detection box (i.e., the second detection box) will not be missed.

[0042] (2) Determine candidate detection boxes with a confidence level greater than a first preset threshold as first detection boxes, and determine candidate detection boxes with a confidence level within a preset range as second detection boxes: To reduce the matching workload, candidate detection boxes with confidence levels within a preset range can be selected as the second detection boxes. The preset range can be based on the distribution of the actual confidence levels of candidate detection boxes in historical image frames where the target laser point is obscured by dust, debris, smoke, or other objects. This reduces the number of subsequent second detection boxes that need to be matched, improving efficiency.

[0043] S300: Match the first detection box with the predicted location; Specifically, the high-resolution detection bounding boxes are first matched with the predicted locations to calculate the matching degree between the high-resolution detection bounding boxes and the predicted locations.

[0044] Specifically, the matching degree can be represented by the intersection-union ratio (IU), which is the IU between the first detection box (i.e., the high-resolution detection box) and the predicted location.

[0045] S400: Determine whether the matching degree between the first detection box and the predicted position is greater than the first preset value; When the judgment result of S400 is yes, that is, the matching degree between the first detection box and the predicted position is greater than the first preset value, it means that the target laser point is detected in the first detection box, that is, the position of the target laser point in the current frame image is determined. At this time, the motion trajectory of the target laser point is updated according to the first detection box (the motion trajectory is obtained from the motion trajectory of the target laser point based on multiple sequence frame images before the current time), that is, S800 is executed.

[0046] When the judgment result of S400 is negative, that is, the matching degree between the first detection box and the predicted position is less than the first preset value, it means that no target laser point is detected in the first detection box. At this time, the detection of whether there is a target laser point continues in the low-resolution detection box, that is, S500 is executed.

[0047] S500: Match the second detection box with the predicted position to calculate the matching degree between the second detection box and the predicted position; Similarly, the matching degree between the second detection box and the predicted position can be calculated by the cross-union ratio between the second detection box and the predicted position.

[0048] S600: Determine whether the matching degree between the second detection box and the predicted position is greater than the second preset value; When the judgment result of S600 is yes, it means that the matching degree between the second detection box and the predicted position is greater than the second preset value, indicating that the target laser point is included in the second detection box. That is, the target laser point that was lost due to low confidence has been found. At this time, the motion trajectory of the target laser point is updated according to the second detection box, that is, S700 is executed.

[0049] When the judgment result of S600 is negative, it means that there is no target laser point in the second detection box, which means that there is no target laser point in the current frame image.

[0050] It should be noted that the first preset value and the second preset value can be the same or different. For example, the second preset value may be greater than the first preset value, or the second preset value may be less than the first preset value. Specifically, the first preset value can be determined based on the confidence level corresponding to the first detection box. For example, there is a certain mapping relationship between the confidence level greater than the first preset threshold and the first preset value, and the first preset value can be determined based on the confidence level of the first detection box within this mapping relationship.

[0051] S700: Update the motion trajectory of the target laser point based on the second detection frame.

[0052] Specifically, the motion trajectory of the target laser point refers to the motion trajectory of the target laser point obtained from multiple sequence frames of images prior to the current moment.

[0053] When the second detection box of the target laser point is determined based on the current frame image, the motion trajectory of the target laser point is updated based on the second detection box to make the motion trajectory of the target laser point continuous.

[0054] S800: Update the motion trajectory of the target laser point based on the first detection frame.

[0055] Similarly, the motion trajectory of the target laser point refers to the motion trajectory of the target laser point obtained from multiple sequence frames of images prior to the current moment.

[0056] When the first detection box of the target laser point is determined based on the current frame image, the motion trajectory of the target laser point is updated based on the first detection box to make the motion trajectory of the target laser point continuous.

[0057] This invention provides a laser point tracking method based on endoscopic sequence images. First, a Kalman filter is used to predict the predicted position of the target laser point in the current frame image based on multiple sequence frames from previous times. Then, a single-target tracking algorithm is used to determine the high-resolution and low-resolution detection boxes in the current frame image. Next, a two-stage matching process is employed: first, the high-resolution detection box is matched with the predicted position; if the matching fails, the low-resolution detection box is then matched with the predicted position. This method can recover target laser points lost due to obstruction and re-attach them to the target laser point's trajectory, improving the accuracy of target laser point tracking and ensuring the continuity of target laser point tracking. Thus, regardless of whether the stone is obstructed or resembles other tissues, the accuracy of target laser point tracking allows for accurate stone fragmentation, significantly reducing the probability of accidental injury and improving surgical safety. Furthermore, the accuracy of target laser point tracking ensures stable control signals output by the laser control system, reducing the probability of laser generator damage.

[0058] In one embodiment of the present invention, such as Figure 2 As shown, the specific prediction method for predicting the predicted position of the target laser point using a Kalman filter, namely S100 (predicting the predicted position of the target laser point in the current frame image based on multiple sequence frame images prior to the current time using a Kalman filter), specifically includes the following steps S110-S130: S110: Construct a state-space model, which defines the state vector of the target laser point; Specifically, the state vector X of the target laser point k It can be defined as an 8-dimensional vector, specifically as follows:

[0059] In the formula, Let be the center coordinates of the target laser point, and α be the aspect ratio of the frame image. h The height of the frame image. The velocity component of the target laser point.

[0060] S120: Based on the uniform motion model, predict the current state vector and current error covariance matrix of the target laser point in the current frame image according to the state vector of the target laser point in the previous frame image. The formula for calculating the current state vector is:

[0061] Among them, F K Here is the state transition matrix. The current state vector of the target laser point; The formula for calculating the current error covariance matrix is: ; in, Let F be the current error covariance matrix. K Here is the state transition matrix. Q K This is the current noise covariance matrix.

[0062] S130: Determine the predicted position of the target laser point in the current frame image based on the current state vector of the target laser point in the current frame image.

[0063] In another embodiment of the present invention, such as Figure 3 As shown, S100 (based on the Kalman filter, predicting the predicted position of the target laser point in the current frame image according to multiple sequence frames of images prior to the current time) specifically includes the following steps S140-S160: S140: Determine the current velocity of the target laser point based on its current state vector in the current frame image; S150: When the current velocity of the target laser point is greater than the preset velocity threshold, increase the weight of the velocity term in the current noise covariance matrix to update the current noise covariance matrix; S160: Update the current error covariance matrix based on the updated current noise covariance matrix.

[0064] When an operator manipulates a laser fiber, hand tremors or rapid swinging may cause the target laser point's movement speed to change instantaneously. Therefore, this invention sets a preset speed threshold. When the detected speed of the target laser point exceeds the preset speed threshold, the weight corresponding to the speed term in the noise covariance matrix is ​​increased to dynamically change the weight corresponding to the speed term in the noise covariance matrix, enabling the Kalman filter to respond quickly to sudden changes in movement.

[0065] In another embodiment of the invention, such as Figure 4 As shown, S100 (based on the Kalman filter, predicting the predicted position of the target laser point in the current frame image according to multiple sequence frames of images prior to the current moment) specifically includes the following steps S170-S190: S170: Obtain the current observation value of the uniform motion model when predicting the current state vector of the target laser point in the current frame image; S180: Calculate the current Kalman gain based on the current observations; The formula for calculating the current Kalman gain is as follows:

[0066] Where, in the formula, K k For the current Kalman gain, The current error covariance matrix, R k H is the current noise matrix, and H is the observation matrix. It is the transpose of H.

[0067] S190: Update the current state vector of the target laser point in the current frame image based on the current Kalman gain; The update formula for updating the current state vector based on the current Kalman gain is as follows:

[0068] Where, in the formula, This is the updated current state vector. The current observation value is the current state vector. The current state vector before the update. K k This represents the current Kalman gain.

[0069] Optionally, the confidence level of the detection box matching the predicted position is obtained, and then back-mapped to the current noise matrix based on the confidence level. This invention back-maps the confidence level of the detection box corresponding to the target laser detection point to the measurement noise matrix. R k The lower the confidence level, R k The larger the value, the smoother the trajectory can be achieved by utilizing historical motion inertia when the laser point is severely obstructed and the detection frame position is inaccurate, thus preventing abrupt changes in the control signal.

[0070] As a second aspect of the invention, the invention also provides a laser point tracker based on endoscopic sequence maps. Figure 5 The diagram shown is a working block diagram of a laser point tracker based on an endoscope sequence map according to an embodiment of the present invention. Figure 5 As shown, the present invention provides a laser point tracker 1 based on an endoscope sequence map, comprising: Kalman filter 10 is used to predict the position of the target laser point in the current frame image based on multiple sequence frames of images prior to the current time. That is, the Kalman filter 10 is used to perform S100 in the laser point tracking method based on the endoscopic sequence map described above.

[0071] Neural network model 20 is used to track and detect target laser points in the current frame image based on a single target tracking algorithm, and obtain a first detection box and a second detection box. The confidence of the first detection box is greater than the confidence of the second detection box. That is, neural network model 20 is used to execute S200 in the laser point tracking method based on endoscopic sequence maps described above.

[0072] Matching unit 30, the matching unit is used to match the first detection box with the predicted position; That is, the matching unit 30 is used to execute S300 in the laser point tracking method based on the endoscopic sequence map described above.

[0073] The trajectory determination unit 40 is used to match the second detection box with the predicted position when the matching degree between the first detection box and the predicted position is less than the first preset value, and to update the motion trajectory of the target laser point according to the second detection box when the matching degree between the second detection box and the predicted position is greater than or equal to the second preset value.

[0074] That is, the trajectory determination unit 40 is used to execute S400-S700 in the laser point tracking method based on the endoscopic sequence map described above.

[0075] This invention provides a laser point tracker based on endoscopic sequence images. First, a Kalman filter is used to predict the predicted position of the target laser point in the current frame image based on multiple sequence frames from previous times. Then, a single-target tracking algorithm is used to determine the high-resolution and low-resolution detection boxes in the current frame image. Next, a two-stage matching process is employed: first, the high-resolution detection box is matched with the predicted position; if the matching fails, the low-resolution detection box is then matched with the predicted position. This process can recover target laser points lost due to obstruction and re-associate them with the target laser point's trajectory, improving the accuracy of target laser point tracking and ensuring the continuity of target laser point tracking. Thus, regardless of whether the stone is obstructed or resembles other tissues, the accuracy of target laser point tracking allows for accurate stone fragmentation, significantly reducing the probability of accidental injury and improving surgical safety. Furthermore, the accuracy of target laser point tracking ensures stable control signals output by the laser control system, reducing the probability of laser generator damage.

[0076] As a third aspect of the invention, the invention also provides an endoscopic lithotripsy system, such as... Figure 6 As shown, the endoscopic lithotripsy system includes: a laser generator 60 for emitting a laser beam to the target detection area to form a laser spot; an endoscope 50 for acquiring an image of the target detection area; and the laser spot tracker described above.

[0077] The endoscope transmits the image of the target detection area to the CMOS / CCD sensor of the camera via optical fiber and outputs a standard digital video signal (such as HDMI 2.0). The laser point tracker then decodes the digital video signal to obtain multiple sequence frames.

[0078] Optionally, this invention also employs a professional-grade image acquisition card based on a PCIe interface. This image acquisition card supports Direct Memory Access (DMA) and Zero-Copy technologies. Video frame data can be directly written to the GPU memory via the PCIe bus.

[0079] The laser generator communicates with the laser point tracker in the following two ways: (1) Digital I / O control interface: used for millisecond-level real-time excitation control. The IPC outputs TTL level signals through a dedicated IO control card.

[0080] (2) Serial Communication Interface: Used for parameter adjustment and status monitoring. The control port of the laser spot tracker can be connected to the laser generator via RS-232 or RS-485 interface.

[0081] Exemplary electronic devices As a fourth aspect of the present invention, the present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, it executes the laser point tracking method based on endoscopic sequence maps described above.

[0082] Specifically, the electronic device includes a processor, memory, network interface, and input device connected via a device bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores operating devices and computer programs. The internal memory provides an environment for the operation of the operating devices and computer programs in the non-volatile storage medium. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it follows the steps of a laser point tracking method based on an endoscopic sequence map according to various embodiments of this specification, as described in the above embodiments.

[0083] The processor may include the main processor, as well as baseband chips, modems, etc.

[0084] The memory stores a program that executes the technical solution of this invention, and may also store operating devices and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0085] The processor can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0086] The control handle may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0087] Output devices may include devices that allow information to be output to a user, such as displays, printers, speakers, etc.

[0088] The communication interface may include any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0089] The processor executes the program stored in the memory and calls other devices, which can be used to implement the various steps of the laser point tracking method based on endoscopic sequence maps provided in the above embodiments of this specification.

[0090] The electronic device may also include a display component and a voice component. The display component may be a liquid crystal display or an e-ink display. The input device of the controller may be a touch layer covering the display component, or a button, trackball, or touchpad set on the controller housing, or an external keyboard, touchpad, or mouse, etc.

[0091] Exemplary computer program products and storage media In addition to the methods and devices described above, the laser spot tracking method based on endoscopic sequence maps provided in the embodiments of this specification can also be a computer program product, which includes computer program instructions that, when executed by a processor, cause the processor to perform the steps in the laser spot tracking method based on endoscopic sequence maps according to various embodiments of this specification as described in the "Exemplary Methods" section above.

[0092] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0093] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this specification. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages.

[0094] Furthermore, embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor of the steps in a laser point tracking method based on an endoscopic sequence map according to various embodiments of this specification as described in the "Exemplary Methods" section above.

[0095] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this specification can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0096] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0097] The embodiments described above are merely illustrative of several implementation methods outlined in this specification. While the descriptions are specific and detailed, they should not be construed as limiting the scope of the solutions provided in this specification. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this specification, and these all fall within the scope of protection of this specification. Therefore, the scope of protection for this patent should be determined by the appended claims.

Claims

1. A laser point tracking method based on endoscopic sequence maps, characterized in that, include: Based on the Kalman filter, the predicted position of the target laser point in the current frame image is predicted based on multiple sequence frame images prior to the current moment. Based on a single-target tracking algorithm, target laser points in the current frame image are tracked and detected to obtain a first detection box and a second detection box. The confidence of the first detection box is greater than that of the second detection box. Match the first detection box with the predicted position; When the matching degree between the first detection box and the predicted position is less than or equal to a first preset value, the second detection box is matched with the predicted position. When the matching degree between the second detection box and the predicted position is greater than a second preset value, the motion trajectory of the target laser point is updated according to the second detection box.

2. The laser point tracking method according to claim 1, characterized in that, Based on a Kalman filter, the predicted position of the target laser point in the current frame image is predicted according to multiple sequence frames of images prior to the current moment, including: Construct a state-space model, wherein the state-space model defines the state vector of the target laser point; Based on the uniform motion model, the current state vector and current error covariance matrix of the target laser point in the current frame image are predicted according to the state vector of the target laser point in the previous frame image. The predicted position of the target laser point in the current frame image is determined based on the current state vector of the target laser point in the current frame image. The state vector of the target laser point can be denoted as: In the formula, Let be the center coordinates of the target laser point, and α be the aspect ratio of the frame image. h The height of the frame image. The velocity component of the target laser point; The formula for calculating the current state vector is: Among them, F K Here is the state transition matrix. The current state vector of the target laser point; The formula for calculating the current error covariance matrix is: ； in, Let F be the current error covariance matrix. K Here is the state transition matrix. Q K This is the current noise covariance matrix.

3. The laser point tracking method according to claim 2, characterized in that, Based on the Kalman filter, predicting the predicted position of the target laser point in the current frame image according to multiple sequence frames of images prior to the current time also includes: The current velocity of the target laser point is determined based on the current state vector of the target laser point in the current frame image. When the current velocity of the target laser point is greater than a preset velocity threshold, the weight corresponding to the velocity term in the current noise covariance matrix is ​​increased to update the current noise covariance matrix; The current error covariance matrix is ​​updated based on the updated current noise covariance matrix.

4. The laser point tracking method according to claim 2, characterized in that, Based on the Kalman filter, predicting the predicted position of the target laser point in the current frame image according to multiple sequence frames of images prior to the current time also includes: Obtain the current observation value of the uniform motion model when predicting the current state vector of the target laser point in the current frame image; Calculate the current Kalman gain based on the current observations; The current state vector of the target laser point in the current frame image is updated based on the current Kalman gain; The formula for calculating the current Kalman gain is as follows: Where, in the formula, K k For the current Kalman gain, Given the current error covariance matrix, R k H is the current noise matrix, and H is the observation matrix. It is the transpose of H; The update formula for updating the current state vector based on the current Kalman gain is as follows: Where, in the formula, This is the updated current state vector. The current observation value is the current state vector. The current state vector before the update. K k This represents the current Kalman gain.

5. The laser point tracking method according to claim 4, characterized in that, Also includes: Obtain the confidence level corresponding to the detection box that matches the predicted position; The confidence level is then back-mapped to the current noise matrix.

6. The laser point tracking method according to claim 1, characterized in that, The confidence level of the first detection box is greater than the first preset threshold, and the confidence level of the second detection box is less than or equal to the first preset threshold and greater than the second preset threshold.

7. The laser point tracking method according to claim 1, characterized in that, The confidence level of the first detection box is greater than a first preset threshold, and the confidence level of the second detection box is within a preset range. The preset range is based on the confidence level distribution range of the detection boxes corresponding to the target laser point being occluded by an obstacle in the historical frame image.

8. The laser point tracking method according to claim 1, characterized in that, Also includes: When the matching degree between the first detection box and the predicted position is greater than a first preset value, the motion trajectory of the target laser point is updated according to the first detection box.

9. A laser point tracker based on endoscopic sequence maps, characterized in that, include: Kalman filter, the Kalman filter is used to predict the predicted position of the target laser point in the current frame image based on multiple sequence frame images prior to the current time; A neural network model is used to track and detect target laser points in the current frame image based on a single target tracking algorithm, and obtain a first detection box and a second detection box, wherein the confidence of the first detection box is greater than the confidence of the second detection box. A matching unit is configured to match the first detection box with the predicted position; The trajectory determination unit is used to match the second detection box with the predicted position when the matching degree between the first detection box and the predicted position is less than a first preset value, and to update the motion trajectory of the target laser point according to the second detection box when the matching degree between the second detection box and the predicted position is greater than or equal to a second preset value.

10. An endoscopic lithotripsy system, characterized in that, include: A laser generator is used to emit a laser beam to the target detection area to form a laser spot; An endoscope is used to acquire images of the target detection area; as well as The laser point tracker as described in claim 9.

Images