Stone tracking system, computer equipment and storage medium

Abstract

The invention relates to a stone tracking system, computer equipment and a storage medium. The system comprises a video processing module used for acquiring a video stream and framing the video stream; wherein the video stream is a video stream comprising stones; the calculus detection module is used for identifying a to-be-tracked calculus from the image of the current frame and obtaining current detection result information of the to-be-tracked calculus by using the target detection model; the data backtracking module is used for acquiring historical detection result information of the tracked stone in the image of the previous frame of the current frame; and the calculus tracking module is used for calculating the similarity between the to-be-tracked calculus and the tracked calculus according to the current detection result information and the historical detection result information, and configuring the unique identification information corresponding to the tracked calculus to the to-be-tracked calculus in response to the fact that the similarity is greater than a preset threshold value. By adopting the method, the stone identification and tracking accuracy in the laser lithotripsy operation can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing, in particular to a stone tracking system, a computer device and a storage medium. BACKGROUND

[0002] With the development of medical image processing technology, in-vivo information acquisition devices such as endoscopes have emerged. In laser lithotripsy technology, a clinician can observe by acquiring intraoperative videos through an endoscope, and then introduce laser energy to break up stones.

[0003] However, in the traditional technology, the analysis of the video images collected by the endoscope under laser lithotripsy surgery, such as the analysis and judgment of the position, category, etc. of the stones in the video, all depend on the clinician. Since the judgment time in surgery is limited and the judgment of the stones depends on the experience of the doctor, the identification and tracking of the stones in laser lithotripsy surgery are not accurate, resulting in high surgical risk and low efficiency. SUMMARY

[0004] Therefore, it is necessary to provide a stone tracking system, a computer device and a storage medium capable of improving the accuracy of identification and tracking of stones in laser lithotripsy surgery in view of the above technical problems.

[0005] In a first aspect, a stone tracking system is provided, the system comprising: a video processing module configured to acquire a video stream and frame the video stream; wherein the video stream is a video stream comprising stones; a stone detection module configured to identify a to-be-tracked stone from an image of a current frame and obtain a current detection result information of the to-be-tracked stone by using a target detection model; a data backtracking module configured to acquire a historical detection result information of a tracked stone in an image of a previous frame of the current frame; and a stone tracking module configured to calculate a similarity between the to-be-tracked stone and the tracked stone according to the current detection result information and the historical detection result information, and in response to the similarity being greater than a preset threshold, configure unique identification information corresponding to the tracked stone to the to-be-tracked stone.

[0006] In some embodiments, the stone tracking module is further configured to: in response to the similarity being not greater than the preset threshold, generate corresponding unique identification information for the to-be-tracked stone.

[0007] In some embodiments, the current detection result information comprises at least one of current position information, current pixel size information, current stone category information, current stone composition information, and current stone category confidence.

[0008] In some embodiments, the stone detection module is further configured to: acquire a historical video stream, frame the historical video stream at a preset fixed interval to obtain stone image samples; wherein the historical video stream is a video stream collected by an in-vivo image collection device during a laser lithotripsy surgery; pre-process the stone image samples to obtain processed stone image samples; identify annotation information in the processed stone image samples; wherein the annotation information includes position information, pixel size information, stone category information, and stone composition information; train a target detection model according to the annotation information of the processed stone image samples.

[0009] In some embodiments, the current detection result information includes current position information, current pixel size information, and current stone category information, and the historical detection result information includes historical position information, historical pixel size information, and historical stone category information; and the stone tracking module is specifically configured to: determine predicted position information of a tracked stone in a current frame of images according to the historical position information, and calculate spatial overlap and center distance difference between the stone to be tracked and the tracked stone according to the current position information and the predicted position information; calculate size consistency between the stone to be tracked and the tracked stone according to the current pixel size information and the historical pixel size information; calculate category consistency between the stone to be tracked and the tracked stone according to the current stone category information and the historical stone category information; and calculate similarity between the stone to be tracked and the tracked stone according to the spatial overlap, the center distance difference, the size consistency, and the category consistency.

[0010] In some embodiments, the current detection result information includes current pixel size information, and the system further includes a stone statistics module configured to: calculate size information of the stone to be tracked according to the current pixel size information and the current conversion rate; and classify and count the number of stones in different size intervals according to the size information and the unique identification information.

[0011] In some embodiments, the system further includes a size measurement module configured to: determine a stone to be measured in response to a selection operation of a user; obtain starting coordinates and ending coordinates of a drawn line in response to a line drawing operation of the user on the stone to be measured; calculate size information of the stone to be measured according to the starting coordinates, the ending coordinates, and the current conversion rate.

[0012] In some embodiments, the system further comprises a conversion rate calibration module configured to: obtain a calibration reference target selected by a user and size information of the calibration reference target; in response to a user's line drawing operation, obtain a line segment corresponding to the calibration reference target, and read pixel size information of the line segment; and calibrate the current conversion rate according to the pixel size information of the line segment and the size information of the calibration reference target.

[0013] In a second aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the following when executing the computer program: obtaining a video stream, and frame dividing the video stream; wherein the video stream is a video stream comprising stones; identifying a to-be-tracked stone from an image of a current frame, and obtaining current detection result information of the to-be-tracked stone by using a target detection model; obtaining historical detection result information of a tracked stone in an image of a previous frame of the current frame; and calculating a similarity between the to-be-tracked stone and the tracked stone according to the current detection result information and the historical detection result information, and in response to the similarity being greater than a preset threshold, configuring unique identification information corresponding to the tracked stone to the to-be-tracked stone.

[0014] In a third aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and the computer program is executable on a processor to implement the following: obtaining a video stream, and frame dividing the video stream; wherein the video stream is a video stream comprising stones; identifying a to-be-tracked stone from an image of a current frame, and obtaining current detection result information of the to-be-tracked stone by using a target detection model; obtaining historical detection result information of a tracked stone in an image of a previous frame of the current frame; and calculating a similarity between the to-be-tracked stone and the tracked stone according to the current detection result information and the historical detection result information, and in response to the similarity being greater than a preset threshold, configuring unique identification information corresponding to the tracked stone to the to-be-tracked stone.

[0015] The stone tracking system, the computer device, and the storage medium described above, by frame dividing a continuous video stream comprising stones, and combining a target detection technology and a tracking technology, i.e., by using a target detection model to detect each image respectively, and by using current detection result information of a to-be-tracked stone in a current frame and historical detection result information of a tracked stone in a previous frame, calculating a matching relationship between the to-be-tracked stone in the current frame and the tracked stone in the previous frame, the stone can be efficiently and accurately identified and the same stone can be stably tracked in a continuous video stream in time and space, thereby improving the efficiency and accuracy of stone identification and tracking in a laser lithotripsy operation. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1These are schematic diagrams of the stone tracking system in some embodiments; Figure 2 This is a schematic diagram of the stone tracking system in some other embodiments; Figure 3 This is a schematic diagram of the software interface of the stone tracking system in some embodiments; Figure 4 This is a schematic diagram of the software interface of the stone tracking system in manual measurement mode in some embodiments; Figure 5 This is a schematic diagram of the software interface of the stone tracking system in size calibration mode in some embodiments; Figure 6 This is a diagram showing the internal structure of a computer device in some embodiments. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] In some embodiments, reference may be made to Figure 1 As shown, a kidney stone tracking system 100 is provided, which may include: a video processing module 110, a kidney stone detection module 120, a data backtracking module 130, and a kidney stone tracking module 140; wherein, The video processing module 110 is used to acquire a video stream and divide the video stream into frames; wherein the video stream includes the stones.

[0019] The video stream can be acquired through an in-vivo image acquisition device, which may include, but is not limited to, flexible endoscopes, rigid endoscopes, ureteroscopes, cystoscopes, etc. The video stream may originate from real-time video streams acquired during laser lithotripsy surgery using endoscopes, converted video streams obtained by connecting a camera to an endoscope, or video streams retrieved from memory or a database.

[0020] Specifically, after the video processing module 110 receives the video stream containing stones acquired by the image acquisition device in real time or periodically, it can divide the video stream into frames, for example, by extracting images from the video stream frame by frame in chronological order or according to a preset time interval through a video decoder.

[0021] The stone detection module 120 is used to identify the stones to be tracked from the image of the current frame and to obtain the current detection result information of the stones to be tracked using the target detection model.

[0022] Specifically, the stone detection module 120 uses the images of each frame extracted in chronological order as the image of the current frame, and calls the target detection model to process the image of the current frame in order to identify and detect stones.

[0023] The object detection model can be an object detection model based on a deep convolutional neural network. The object detection model can include, but is not limited to, YOLO (You Only Look Once) version 8-13 series models, R-CNN (Region-based Convolutional Neural Network) series models, Fast R-CNN (Fast Region-based Convolutional Neural Network) series models, and DETR (Detection Transformer) series models. DETR series models can include, but are not limited to, RT-DETR (Real-Time Detection Transformer) and DEIM (Detection Transformer with Improved Matching).

[0024] In some embodiments, the target detection model may be pre-trained, and the stone detection module 120 may also be configured to perform the training steps of the target detection model.

[0025] More specifically, the stone detection module 120 can be used to: acquire historical video streams, extract frames from the historical video streams at preset fixed intervals to obtain stone image samples; wherein the historical video streams are video streams acquired by an in vivo image acquisition device during laser lithotripsy; preprocess the stone image samples to obtain processed stone image samples; identify the annotation information in the processed stone image samples; wherein the annotation information includes location information, pixel size information, stone category information, and stone composition information; and train a target detection model based on the annotation information of the processed stone image samples.

[0026] In this embodiment, the specific training process of the object detection model can be referred to as follows: First, video footage captured by endoscopes (flexible or rigid) during real laser lithotripsy surgeries can be obtained from a hospital's urology database as a historical video stream for training. Then, frames can be extracted from the historical video stream at preset intervals to obtain stone image samples. Furthermore, while extracting a large number of stone image samples, a preliminary stone image sample library can be established to store the samples for subsequent training purposes, ensuring their diversity and scale.

[0027] Next, the stone image samples can be preprocessed. Preprocessing methods include, but are not limited to, data cleaning, deduplication, and noise reduction. After initial video frame extraction and the establishment of a preliminary stone image sample library, there may be some images without stones or with poor pixel quality in the preliminary stone image sample library. Therefore, inaccurate, erroneous, or duplicate image data can be removed through cleaning, deduplication, and noise reduction, thereby improving the quality of training samples and thus improving the accuracy of target detection model training.

[0028] Furthermore, a scoring mechanism based on the professional knowledge of clinicians can be introduced to evaluate the blurriness and clarity of stone image samples. Stone image samples that do not meet the score requirements can be subject to quality control, i.e., inaccurate, erroneous, or duplicate stone image samples can be deleted, which is beneficial to the convergence of the target detection model during training.

[0029] Then, based on the clinical experience of doctors, the main and important stones appearing in the stone image samples can be labeled. The labeling information may include, but is not limited to, the location information, pixel size information, stone category information, and stone composition information of the stone. Specifically, the pixel size information and location information of the target stone can be labeled using the smallest rectangle. The stone category information is determined according to the stone composition report. The stone category information may include, but is not limited to, calcium oxalate stones, calcium phosphate stones, uric acid stones, and magnesium ammonium phosphate stones. Stone composition information can also be attached to each labeled smallest rectangle.

[0030] Finally, a certain number of preprocessed stone image samples with annotation information are used to train the target detection model. The target detection model continuously optimizes the initial model parameters during the training process by recognizing the annotation information of the stone image samples, thereby obtaining the trained target detection model.

[0031] In this embodiment, after the trained target detection model receives an image of the current frame containing the stone, it can quickly and accurately identify the stone to be tracked in the current frame image and output the current detection result information corresponding to the stone to be tracked. The current detection result information includes at least one of the following: current location information, current pixel size information, current stone type information, current stone composition information, and current stone type confidence level. Furthermore, the current detection result information can be displayed in real time at a preset position on the software interface of the system, so that it can be used as a reference by doctors or surgeons.

[0032] The data backtracking module 130 is used to obtain historical detection results information of tracked stones from the image of the previous frame of the current frame.

[0033] The data backtracking module 130 can obtain historical detection results of tracked stones from the previous frame of the current frame. Furthermore, based on these historical detection results, it can predict the state of the tracked stones in the current frame using a target tracking algorithm, thereby obtaining predicted state information for the tracked stones. This predicted state information may include, but is not limited to, predicted location information, predicted appearance information, and predicted velocity information. The predicted state information of the tracked stones in the current frame, calculated based on their historical detection results, can be used together with the current detection results of the stones to be tracked to calculate the similarity between the stones to be tracked and the tracked stones.

[0034] For example, target tracking algorithms include, but are not limited to, SORT (Simple Online and Realtime Tracking), Deep Sort (Simple Online and Realtime Tracking with Deep association metric), Byte Track (a tracker based on byte association), and Bo T-SORT (Bottoms-Up Tracking from Online Streaming Toolbox).

[0035] The stone tracking module 140 is used to calculate the similarity between the stone to be tracked and the stone that has been tracked based on the current detection result information and the historical detection result information. In response to the similarity being greater than a preset threshold, the unique identification information corresponding to the stone that has been tracked is configured to the stone to be tracked.

[0036] The stone tracking module 140 compares the similarity between tracked stones and untracked stones in two consecutive frames. If the similarity exceeds a preset threshold, the module determines that the tracked stone and the untracked stone in the two frames are the same stone. It then assigns the unique identifier information of the tracked stone to the untracked stone. In other words, the untracked stone in the current frame uses the unique identifier information of the tracked stone from the previous frame that matches the similarity requirement. This ensures consistency in the identification of the same stone, achieving continuous, stable, and accurate spatiotemporal tracking and consistent identification of stones.

[0037] In some embodiments, the stone detection module 140 can further be used to: generate corresponding unique identification information for the stone to be tracked in response to a similarity not exceeding a preset threshold.

[0038] Specifically, for newly appearing stones that do not match the trajectory of any already tracked stones, the stone detection module 140 can automatically generate a new identifier for them, serving as the unique identifier for the newly appearing stone. Furthermore, if no stone matching an already tracked stone is detected in several consecutive frames, the unique identifier for that already tracked stone can be cancelled or frozen. Through this time-series-based data association mechanism, the consistency of the identifier information for the same stone can be maintained across consecutive video frames, thereby achieving continuous and accurate spatiotemporal tracking of each stone.

[0039] In some embodiments, the stone tracking module 140 is specifically configured to perform the following steps to realize the calculation process of the similarity between the stone to be tracked and the already tracked stone; The current detection result information includes current location information, current pixel size information, and current stone type information, while the historical detection result information includes historical location information, historical pixel size information, and historical stone type information. The stone tracking module 140 can be used for: Based on historical location information, the predicted location information of the tracked stones in the current frame image is determined. Based on the current location information and the predicted location information, the spatial overlap and center distance difference between the tracked stones and the stones to be tracked are calculated. Based on the current pixel size information and historical pixel size information, the size consistency between the tracked stones and the stones to be tracked is calculated. Based on the current stone category information and historical stone category information, the category consistency between the tracked stones and the stones to be tracked is calculated. Finally, the similarity between the tracked stones and the stones to be tracked is calculated based on the spatial overlap, center distance difference, size consistency, and category consistency.

[0040] The predicted location information refers to the possible location information of the tracked stone in the previous frame in the current frame. For example, it can be calculated using the Mann filter prediction algorithm.

[0041] For example, the current location information may include the current detection box, the current target center, etc., while the predicted location information may include the predicted detection box and predicted target center that the tracked stone in the previous frame might correspond to in the current frame. The spatial overlap between the current detection box of the stone to be tracked and the predicted detection box of the tracked stone, and the center distance between the current target center of the stone to be tracked and the predicted target center of the tracked stone, can be used as the basis for judging their similarity. The higher the spatial overlap and the closer the center distance, the higher the similarity score. The spatial overlap, for example, can be calculated using IoU (Intersection over Union).

[0042] In addition, the pixel size information of the stone to be tracked (current pixel size information) and the pixel size information of the tracked stones (historical pixel size information) can be used as one of the criteria for similarity judgment. The closer the pixel size information of the two, the higher the size consistency between them, and the higher the similarity score. Furthermore, the stone category information of the stone to be tracked (current stone category information) and the stone category information of the tracked stones (historical stone category information) obtained through the object detection model can also be used as one of the criteria for similarity judgment. If the stone category information of the two is consistent, the similarity score is higher.

[0043] In this embodiment, by combining the calculations of spatial overlap, center distance difference, size consistency, and category consistency, the similarity between the stone to be tracked and the stone already tracked can be determined more comprehensively and accurately, thereby ensuring the accuracy of tracking the same stone in a continuous video stream.

[0044] The aforementioned stone tracking system divides a continuous video stream containing stones into frames and combines target detection and tracking technologies. Specifically, it uses a target detection model to detect each frame of the image and calculates the matching relationship between the stone to be tracked in the current frame and the previously tracked stone by comparing the current detection result information of the stone to be tracked in the current frame with the historical detection result information of the tracked stone in the previous frame. This allows for efficient and accurate identification of stones in a spatiotemporally continuous video stream and stable tracking of the same stone, thereby improving the efficiency and accuracy of stone identification and tracking in laser lithotripsy.

[0045] In practical application scenarios, you can refer to Figure 3 As shown, Figure 3Schematic diagrams of the software interface of the stone tracking system involved in this application in some embodiments are shown.

[0046] According to the above embodiments, the trained target detection model and target tracking algorithm can be combined to generate an image processing framework and embedded into the software system to construct the stone tracking system involved in the embodiments of this application. Furthermore, the PyQt graphical user interface (GUI) framework can be combined to develop the software interface of the stone tracking system, so as to realize the purpose of real-time stone tracking through the video stream acquired during the operation, and to display the current detection result information and the identification information of the tracked stones in real time. In addition, human-computer interaction can be realized through the software interface, and functions such as stone classification and quantity statistics, manual measurement of stones, and automatic calibration of the current conversion rate can be realized according to different user selections.

[0047] For example, the software interface of the stone tracking system can refer to Figure 2 In designing the software interface, it can mainly include four display areas: (1) Function menu display area: Basic function menu buttons such as "Camera", "Select Video", "Play Video", "Take Screenshot", "Start Statistics", "Manual Measurement", and "Size Calibration" can be set in this area. The main functions are to control the opening ("Camera"), selection ("Select Video"), playback ("Play Video"), saving ("Take Screenshot"), identification, tracking and classification of stones in the current frame image ("Start Statistics"), measurement ("Manual Measurement"), and calibration of the current conversion rate ("Size Calibration").

[0048] (2) Video image display area: mainly used for real-time imaging display of video stream, image screenshot display, brief current detection result information (e.g., location box), and display of unique identification information (ID) of stones.

[0049] (3) Current detection result display area: mainly used to display the current detection result information of the stones in the current frame image after “start statistics”, such as including but not limited to the frame number of the current frame, the current location information of the stones, the current pixel size information, the current stone category information, the current stone composition information, the current stone category confidence, unique identifier information (ID) and the converted size information, etc.

[0050] (4) Display area for classification and quantity statistics: Estimate the stone size information based on the current conversion rate, and classify and count according to the preset size range and display it.

[0051] In some embodiments, the stone tracking system supports a reasoning mode, which is a mode for identifying and tracking stones in the current frame image. In this mode, as described in the foregoing embodiments, through the cooperation of the video processing module 110, the stone detection module 120, the data backtracking module 130, and the stone tracking module 140, stones can be dynamically identified and tracked in the video stream. Furthermore, the current detection result information and the unique identifier information corresponding to the same stone can be displayed in the corresponding position on the software interface, so as to provide timely reference for doctors or surgeons.

[0052] In some embodiments, reference may be made to Figure 2 As shown, Figure 2 A schematic diagram of the structure of a stone tracking system in some other embodiments is shown. The current detection result information includes the current pixel size information. The stone tracking system 100 may also include a stone statistics module 150. The stone statistics module 150 is used to: calculate the size information of the stone to be tracked based on the current pixel size information and the current conversion rate; and classify and count the number of stones in different size ranges based on the size information and unique identification information.

[0053] In this embodiment, the stone tracking system can also support the classification and quantity statistics of stones. In response to a user triggering entry into the classification and quantity statistics mode, the system can obtain a pre-set or stored current conversion rate and convert the pixel size information of the stones to be tracked into size information (e.g., at the millimeter level). The formula for calculating the size information can be found as follows: Size information = pixel size information * current conversion rate.

[0054] Furthermore, in response to users' needs for classifying and counting the number of stones, the software can classify and count the number of stones with unique identification information according to preset size ranges (e.g., 0 to 1 mm, 1 to 2 mm and above 2 mm, etc., which can be set according to actual needs), and display the classification and counting results in real time at a preset position on the software interface.

[0055] The current conversion rate can be obtained by using the laser fiber under the endoscope as the calibration reference target, and by calculating its actual size information and the pixel size information in the image displayed on the interface. The laser fiber used as the calibration reference target can include, but is not limited to, holmium laser fiber, thulium laser fiber, etc.

[0056] In some embodiments, continue to refer to Figure 2The stone tracking system 100 also includes a size measurement module 160, which is used to: determine the stone to be measured in response to the user's selection operation; obtain the starting coordinates and ending coordinates of the drawn line segment in response to the user's drawing operation on the stone to be measured; and calculate the size information of the stone to be measured based on the starting coordinates, ending coordinates and the current conversion rate.

[0057] In this embodiment, the stone tracking system also supports manual measurement of the stones. (See reference...) Figure 4 As shown, Figure 4 The diagram shows a schematic of the software interface of the stone tracking system in manual measurement mode in some embodiments.

[0058] Specifically, in response to a user triggering the manual measurement mode, the user can select one or more stones from the video image display area as the stones to be measured. Then, the user can draw lines on the stones using a mouse or touchscreen. For example, the user can press and drag the mouse to draw lines. After detecting the user's drawing operation, the system identifies the starting coordinates (mouse click point) and ending coordinates (mouse release point) of the line segment drawn by the user. Using the starting and ending coordinates and the current conversion rate, the system calculates the size information of the stone to be measured. In this embodiment, by implementing manual stone measurement through human-computer interaction, the flexibility of stone measurement can be improved, thereby combining artificial intelligence with the professional experience of medical personnel, which is beneficial to the efficiency and accuracy of stone identification in complex intraoperative scenarios.

[0059] In some embodiments, continue to refer to Figure 2 The stone tracking system 100 may also include a conversion rate calibration module 170, which is used to: acquire a calibration reference target selected by the user and the size information of the calibration reference target; in response to the user's line drawing operation, obtain the line segment corresponding to the calibration reference target and read the pixel size information of the line segment; and calibrate the current conversion rate according to the pixel size information of the line segment and the size information of the calibration reference target.

[0060] In this embodiment, the stone tracking system also supports automatic size calibration of the conversion rate. (See reference...) Figure 5 As shown, Figure 5 A schematic diagram of the software interface of the stone tracking system in size calibration mode is shown in some embodiments.

[0061] In response to a user-triggered size calibration mode, the user can select a standard target with known actual dimensions (such as a fiber optic head or spot) as a calibration reference target. The user can then draw lines on the calibration reference target by pressing and dragging the mouse. In size calibration mode, for example... Figure 5The interface will then display a prompt box reminding the user to enter the size information of the calibration reference target. After the user enters the information, the system will automatically calibrate the current conversion rate based on the user-inputted size information of the calibration reference target and the pixel size information of the calibration reference target corresponding to the line segment drawn on the interface. Then, the system can use the calibrated conversion rate to recalculate the stone size information and count the number of stones, thereby further improving the accuracy of stone detection and tracking.

[0062] Each module in the aforementioned stone tracking system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0063] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0064] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0065] In some embodiments, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps: acquiring a video stream and segmenting the video stream into frames; wherein the video stream is a video stream including stones; identifying the stones to be tracked from the image of the current frame and obtaining the current detection result information of the stones to be tracked using a target detection model; acquiring the historical detection result information of the tracked stones in the image of the previous frame of the current frame; and calculating the similarity between the stones to be tracked and the tracked stones based on the current detection result information and the historical detection result information, and configuring the unique identification information corresponding to the tracked stones to the stones to be tracked in response to the similarity being greater than a preset threshold.

[0066] In some embodiments, when the processor executes the computer program, it further implements the following steps: in response to a similarity not exceeding a preset threshold, generating corresponding unique identification information for the stone to be tracked.

[0067] In some embodiments, when the processor executes the computer program, it further implements the following steps: acquiring a historical video stream, extracting frames from the historical video stream at preset fixed intervals to obtain stone image samples; wherein, the historical video stream is a video stream acquired by an in vivo image acquisition device during laser lithotripsy; preprocessing the stone image samples to obtain processed stone image samples; identifying annotation information in the processed stone image samples; wherein, the annotation information includes location information, pixel size information, stone category information, and stone composition information; and training a target detection model based on the annotation information of the processed stone image samples.

[0068] In some embodiments, when the processor executes the computer program, it further implements the following steps: determining the predicted location information of the tracked stone in the current frame image based on historical location information, and calculating the spatial overlap and center distance difference between the tracked stone and the stone to be tracked based on the current location information and the predicted location information; calculating the size consistency between the tracked stone and the stone to be tracked based on the current pixel size information and historical pixel size information; calculating the category consistency between the tracked stone and the stone to be tracked based on the current stone category information and historical stone category information; and calculating the similarity between the tracked stone and the stone to be tracked based on the spatial overlap, center distance difference, size consistency, and category consistency.

[0069] In some embodiments, when the processor executes the computer program, it also performs the following steps: calculating the size information of the stone to be tracked based on the current pixel size information and the current conversion rate; and classifying and counting the number of stones in different size ranges based on the size information and unique identification information.

[0070] In some embodiments, when the processor executes the computer program, it further performs the following steps: in response to a user's selection operation, determining the stone to be measured; in response to a user's drawing operation on the stone to be measured, obtaining the starting coordinates and ending coordinates of the drawn line segment; and calculating the size information of the stone to be measured based on the starting coordinates, ending coordinates, and the current conversion rate.

[0071] In some embodiments, when the processor executes the computer program, it further implements the following steps: obtaining a calibration reference target selected by the user and the size information of the calibration reference target; in response to the user's line drawing operation, obtaining the line segment corresponding to the calibration reference target and reading the pixel size information of the line segment; and calibrating the current conversion rate according to the pixel size information of the line segment and the size information of the calibration reference target.

[0072] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, performs the following steps: acquiring a video stream and segmenting the video stream into frames; wherein the video stream is a video stream including stones; identifying stones to be tracked from the image of the current frame and obtaining current detection result information of the stones to be tracked using a target detection model; acquiring historical detection result information of tracked stones in the image of the previous frame of the current frame; and calculating the similarity between the stones to be tracked and the tracked stones based on the current detection result information and the historical detection result information, and configuring the unique identification information corresponding to the tracked stones to the stones to be tracked in response to the similarity being greater than a preset threshold.

[0073] In some embodiments, when the computer program is executed by the processor, it further implements the following steps: in response to a similarity not exceeding a preset threshold, generating corresponding unique identification information for the stone to be tracked.

[0074] In some embodiments, when the computer program is executed by the processor, it further performs the following steps: acquiring a historical video stream, extracting frames from the historical video stream at preset fixed intervals to obtain stone image samples; wherein, the historical video stream is a video stream acquired by an in vivo image acquisition device during laser lithotripsy; preprocessing the stone image samples to obtain processed stone image samples; identifying annotation information in the processed stone image samples; wherein, the annotation information includes location information, pixel size information, stone category information, and stone composition information; and training a target detection model based on the annotation information of the processed stone image samples.

[0075] In some embodiments, when the computer program is executed by the processor, it further implements the following steps: determining the predicted location information of the tracked stone in the current frame image based on historical location information, and calculating the spatial overlap and center distance difference between the tracked stone and the stone to be tracked based on the current location information and the predicted location information; calculating the size consistency between the tracked stone and the stone to be tracked based on the current pixel size information and historical pixel size information; calculating the category consistency between the tracked stone and the stone to be tracked based on the current stone category information and historical stone category information; and calculating the similarity between the tracked stone and the stone to be tracked based on the spatial overlap, center distance difference, size consistency, and category consistency.

[0076] In some embodiments, when the computer program is executed by the processor, it further performs the following steps: calculating the size information of the stone to be tracked based on the current pixel size information and the current conversion rate; and classifying and counting the number of stones in different size ranges based on the size information and unique identification information.

[0077] In some embodiments, when the computer program is executed by the processor, it further performs the following steps: in response to a user's selection operation, determining the stone to be measured; in response to a user's drawing operation on the stone to be measured, obtaining the starting coordinates and ending coordinates of the drawn line segment; and calculating the size information of the stone to be measured based on the starting coordinates, ending coordinates, and the current conversion rate.

[0078] In some embodiments, when the computer program is executed by the processor, it further implements the following steps: obtaining a calibration reference target selected by the user and the size information of the calibration reference target; in response to the user's line drawing operation, obtaining the line segment corresponding to the calibration reference target and reading the pixel size information of the line segment; and calibrating the current conversion rate according to the pixel size information of the line segment and the size information of the calibration reference target.

[0079] Those skilled in the art will understand that implementing all or part of the processes in the above embodiments can be accomplished by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0080] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0081] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

[0082] It should be noted that in the embodiments of this application, data involving user information or user data (e.g., video streams of kidney stones, historical video streams, etc.) must be obtained and processed only after user authorization and consent. When the embodiments of this application are applied to specific products or technologies, user permission or consent must be obtained, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A stone tracking system, the system comprising: A video processing module is used to acquire a video stream and segment the video stream into frames; wherein the video stream includes stones. The stone detection module is used to identify the stones to be tracked from the image of the current frame and to obtain the current detection result information of the stones to be tracked using the target detection model; The data backtracking module is used to obtain historical detection results information of tracked stones from the image of the previous frame of the current frame; and The stone tracking module is used to calculate the similarity between the stone to be tracked and the tracked stone based on the current detection result information and the historical detection result information, and in response to the similarity being greater than a preset threshold, to configure the unique identification information corresponding to the tracked stone to the stone to be tracked.

2. The system according to claim 1, characterized in that, The stone tracking module is also used for: In response to the similarity not being greater than a preset threshold, a unique identifier is generated for the stone to be tracked.

3. The system according to claim 1, characterized in that, The current detection result information includes at least one of the following: current location information, current pixel size information, current stone type information, current stone composition information, and current stone type confidence level.

4. The system according to claim 1, characterized in that, The stone detection module is also used for: A historical video stream is acquired, and frames are extracted from the historical video stream at preset fixed intervals to obtain stone image samples; wherein, the historical video stream is a video stream acquired by an in vivo image acquisition device during laser lithotripsy. The stone image samples are preprocessed to obtain processed stone image samples; Identify the annotation information in the processed stone image sample; wherein, the annotation information includes location information, pixel size information, stone type information, and stone composition information; The target detection model is trained based on the annotation information of the processed stone image samples.

5. The system according to claim 1, characterized in that, The current detection result information includes current location information, current pixel size information, and current stone type information; the historical detection result information includes historical location information, historical pixel size information, and historical stone type information; the stone tracking module is specifically used for: Based on the historical location information, the predicted location information of the tracked stone in the current frame image is determined, and based on the current location information and the predicted location information, the spatial overlap and center distance difference between the stone to be tracked and the tracked stone are calculated. Based on the current pixel size information and the historical pixel size information, the size consistency between the stone to be tracked and the stone already tracked is calculated; Based on the current stone category information and the historical stone category information, calculate the category consistency between the stone to be tracked and the previously tracked stones; as well as The similarity between the stone to be tracked and the stone already tracked is calculated based on the spatial overlap, the center distance difference, the size consistency, and the category consistency.

6. The system according to claim 1, characterized in that, The current detection result information includes the current pixel size information. The system also includes a stone statistics module, which is used for: Calculate the size information of the stone to be tracked based on the current pixel size information and the current conversion rate; and Based on the size information and the unique identifier information, the stones in different size ranges are classified and their numbers are counted.

7. The system according to claim 6, characterized in that, The system also includes a dimension measurement module, which is used for: In response to the user's selection, the stone to be measured is determined; In response to the user's drawing operation on the stone to be measured, the starting coordinates and ending coordinates of the drawn line segment are obtained; The size information of the stone to be measured is calculated based on the starting coordinates, the ending coordinates, and the current conversion rate.

8. The system according to claim 1, characterized in that, The system also includes a conversion rate calibration module, which is used for: Obtain the calibration reference target selected by the user and the size information of the calibration reference target; In response to the user's line drawing operation, the line segment corresponding to the calibration reference target is obtained, and the pixel size information of the line segment is read; as well as The current conversion rate is calibrated based on the pixel size information of the line segment and the size information of the calibration reference target.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it achieves the following: A video stream is acquired and segmented into frames; wherein the video stream includes a stone; the stone to be tracked is identified from the image of the current frame, and the current detection result information of the stone to be tracked is obtained using a target detection model; historical detection result information of the tracked stone in the image of the previous frame of the current frame is acquired; and the similarity between the stone to be tracked and the tracked stone is calculated based on the current detection result information and the historical detection result information, and in response to the similarity being greater than a preset threshold, the unique identifier information corresponding to the tracked stone is configured to the stone to be tracked.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by the processor to achieve the following: A video stream is acquired and segmented into frames; wherein the video stream includes a stone; the stone to be tracked is identified from the image of the current frame, and the current detection result information of the stone to be tracked is obtained using a target detection model; historical detection result information of the tracked stone in the image of the previous frame of the current frame is acquired; and the similarity between the stone to be tracked and the tracked stone is calculated based on the current detection result information and the historical detection result information, and in response to the similarity being greater than a preset threshold, the unique identifier information corresponding to the tracked stone is configured to the stone to be tracked.

Images