A method and system for dynamic lesion tracking via endoscopic video

Abstract

This invention discloses a method and system for dynamic lesion tracking in endoscopic video, relating to the field of image data processing. The method includes: constructing a lesion change behavior feature library; extracting at least one candidate lesion region from the endoscopic video stream; calculating corresponding lesion change behavior features for each candidate lesion region in consecutive frames; calling the lesion change behavior feature library to obtain prior lesion change behavior features; determining whether an abnormal tracking event has occurred by comparing the prior lesion change behavior features and the lesion change behavior features; outputting lesion tracking results if no abnormal tracking event occurs; and initiating a dynamic relocation mechanism to track and relocate the candidate lesion region where the abnormal tracking event occurred if an abnormal tracking event occurs. This method solves the technical problems of insufficient accuracy and reliability in existing dynamic lesion tracking of endoscopic video, achieving the technical effect of improving the accuracy and stability of lesion tracking.

Description

Technical Field

[0001] This application relates to the field of image data processing, and in particular to a method and system for dynamic lesion tracking in endoscopic video. Background Technology

[0002] In endoscopic examinations, continuous and stable tracking of specific lesions is a crucial technological foundation for ensuring surgical navigation accuracy, quantifying dynamic changes in lesions, and generating complete examination reports. Currently, mainstream endoscopic lesion tracking methods primarily rely on general target tracking algorithms. These algorithms extract the lesion's surface features and establish a motion model to predict its location within a video sequence. However, due to the complexities of endoscopic scenarios, such as tissue deformation, rapid lens movement, intracavitary fluid interference, and the morphological evolution of the lesion itself, general tracking algorithms struggle to distinguish between changes in the actual lesion and drastic changes in target features caused by tracking failure. This leads to tracking drift and even target loss, and the lack of effective anomaly detection and self-correction mechanisms results in insufficient robustness.

[0003] Currently, endoscopic video dynamic lesion tracking suffers from technical problems of insufficient accuracy and reliability. Summary of the Invention

[0004] This application provides a method and system for dynamic lesion tracking in endoscopic video. It employs techniques such as constructing a feature library of lesion change behavior obtained through sampling and training from known real lesion samples, extracting candidate lesion regions from the endoscopic video stream, calculating the lesion change behavior features in consecutive frames, comparing these features with prior features in the feature library to determine if any abnormal tracking events have occurred. If no abnormal events occur, the lesion tracking result is output; if abnormal events occur, a dynamic relocation mechanism is activated to track and relocate the abnormal candidate lesion regions. These techniques address the technical problems of insufficient accuracy and reliability in existing dynamic lesion tracking methods for endoscopic video, achieving the technical effect of improving the accuracy and stability of lesion tracking.

[0005] This application provides a method for dynamic lesion tracking in endoscopic video, comprising: constructing a lesion change behavior feature library, wherein the lesion change behavior feature library is obtained by sampling and training prior lesion change behavior features on known real lesion samples; extracting at least one candidate lesion region from the endoscopic video stream, calculating the corresponding lesion change behavior features for each candidate lesion region in consecutive frames, calling the lesion change behavior feature library to obtain prior lesion change behavior features of the lesion change behavior features, and determining whether an abnormal tracking event has occurred by comparing the prior lesion change behavior features and the lesion change behavior features; wherein, if no abnormal tracking event has occurred, outputting the lesion tracking result; if an abnormal tracking event has occurred, initiating a dynamic relocation mechanism to track and relocate the candidate lesion region where the abnormal tracking event occurred.

[0006] In a possible implementation, after extracting at least one candidate lesion region from the endoscopic video stream, the following processing is performed: the input endoscopic video stream is divided into segments, and continuous endoscopic video frames are output; a tracking detection step size is set for each candidate lesion region, and the continuous endoscopic video frames are divided according to the tracking detection step size, and a tracking continuous frame for each candidate lesion region is output; for each candidate lesion region, the corresponding lesion change behavior features are calculated under the tracking continuous frame.

[0007] In a possible implementation, a tracking detection step size is set for each candidate lesion region, and the following processing is performed: the lesion type of each candidate lesion region is analyzed according to the lesion change behavior feature library; the movement speed and scene complexity are identified according to the lesion type of each candidate lesion region, and a tracking detection step size is configured for each candidate lesion region based on the movement speed and scene complexity.

[0008] In a possible implementation, a lesion change behavior feature library is constructed, and the following processing is performed: An endoscopic video training set corresponding to known real lesion samples is obtained, the endoscopic video training set including lesion type annotation information; by analyzing the endoscopic video training set, a lesion change behavior feature training set based on temporal changes is collected, the lesion change behavior feature training set including instantaneous velocity and acceleration calculated from the centroid position corresponding to each lesion sample, area positive value calculated based on the lesion contour, pixel distribution similarity, and boundary ambiguity; a classification model is trained on the lesion change behavior feature training set according to the lesion type annotation information to obtain prior lesion change behavior features corresponding to each lesion type; a lesion change behavior feature library is constructed according to the mapping relationship between each lesion type and prior lesion change behavior features.

[0009] In a possible implementation, the prior lesion change behavior features of the lesion change behavior features are obtained by calling the lesion change behavior feature library, and the following processing is performed: the lesion type of the lesion change behavior features is analyzed by calling the lesion change behavior feature library, and the prior lesion change behavior features are obtained by mapping according to the lesion type; wherein, the prior lesion change behavior features include the instantaneous velocity change range, acceleration change range, area value change range, pixel distribution similarity change range, and boundary ambiguity change range.

[0010] In a possible implementation, the presence or absence of an abnormality tracking event is determined by comparing the prior lesion change behavior features with the lesion change behavior features, and the following processing is performed: A change anomaly discrimination model based on multidimensional feature deviations is constructed, including instantaneous velocity change deviation, acceleration deviation, area positive value deviation, pixel distribution similarity deviation, and boundary blurring deviation; the deviation values ​​of the prior lesion change behavior features and the lesion change behavior features are compared according to the change anomaly discrimination model, and a change anomaly score is output; if the change anomaly score is greater than a preset threshold, a change anomaly tracking event is determined to have occurred; otherwise, no change anomaly tracking event is determined to have occurred.

[0011] In a possible implementation, if an abnormal tracking event occurs, a dynamic relocation mechanism is initiated to track and relocate the candidate lesion region where the abnormal tracking event occurred. The following processing is performed: determine the current video frame in which the abnormal tracking event occurred in the candidate lesion region, backtrack based on the current video frame to obtain the previous video frame; predict the predicted position of the candidate lesion region in the current video frame according to the previous video frame, generate a candidate search region with the predicted position, and re-track and locate the candidate lesion region in the candidate search region.

[0012] In a possible implementation, the candidate lesion region is re-tracked and repositioned within the candidate search area, and the following processing is performed: multimodal features of the lesion in the current video frame are acquired; feature similarity matching is performed between the multimodal features of the lesion in the current video frame and the candidate lesion region; a similarity matching result is returned, including the re-tracked and repositioned candidate lesion region; wherein, if the similarity matching result returns empty, the candidate lesion region is marked, and the predicted position of the candidate lesion region in the next video frame is predicted and re-tracked and repositioned.

[0013] In a possible implementation, at least one candidate lesion region is extracted from the endoscopic video stream, and the following processing is performed: a target detection model is trained using a multi-scale convolutional neural network, and the lesion region is extracted from the endoscopic video stream according to the target detection model to obtain an initial candidate lesion region; the initial candidate lesion region is subjected to reflection interference fusion screening to output at least one candidate lesion region.

[0014] This application also provides an endoscopic video dynamic lesion tracking system, comprising: a lesion change behavior feature library construction module, used to construct a lesion change behavior feature library, wherein the lesion change behavior feature library is obtained by sampling and training prior lesion change behavior features on known real lesion samples; a judgment module, used to extract at least one candidate lesion region from the endoscopic video stream, calculate the corresponding lesion change behavior features for each candidate lesion region in consecutive frames, call the lesion change behavior feature library to obtain the prior lesion change behavior features of the lesion change behavior features, and determine whether an abnormal tracking event has occurred by comparing the prior lesion change behavior features and the lesion change behavior features; and a processing module, used to output the lesion tracking result if no abnormal tracking event has occurred, and to activate a dynamic relocation mechanism to track and relocate the candidate lesion region where the abnormal tracking event has occurred if an abnormal tracking event has occurred.

[0015] This application proposes a method and system for dynamic lesion tracking in endoscopic video. First, a lesion change behavior feature library is constructed. This library is obtained through prior lesion change behavior feature sampling training on known real lesion samples. Next, at least one candidate lesion region is extracted from the endoscopic video stream. For each candidate lesion region, corresponding lesion change behavior features are calculated in consecutive frames. The prior lesion change behavior features of these features are obtained by calling the lesion change behavior feature library. By comparing the prior lesion change behavior features with the proposed lesion change behavior features, it is determined whether an abnormal tracking event has occurred. If no abnormal tracking event occurs, the lesion tracking result is output. If an abnormal tracking event occurs, a dynamic relocation mechanism is initiated to track and relocate the candidate lesion region where the abnormal tracking event occurred. This achieves the technical effect of improving the accuracy and stability of lesion tracking. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments of the present invention will be briefly described below. Flowcharts are used in this application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from these processes.

[0017] Figure 1 This is a flowchart illustrating an endoscopic video dynamic lesion tracking method provided in an embodiment of this application.

[0018] Figure 2 This is a schematic diagram of the structure of an endoscopic video dynamic lesion tracking system provided in an embodiment of this application.

[0019] Figure labeling: Module 10 for constructing the feature library of lesion changes, Module 20 for judgment, and Module 30 for processing. Detailed Implementation

[0020] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0021] This application provides a method for dynamic lesion tracking via endoscopic video, such as... Figure 1 As shown, the method includes:

[0022] Step S100: Construct a lesion change behavior feature library, which is obtained by sampling and training known real lesion samples to obtain prior lesion change behavior features.

[0023] Specifically, by collecting video data of known real lesion samples, the dynamic change features of these samples over time are extracted, and a classification model is trained based on these features to establish a database containing typical change patterns of various lesions. This database serves as a benchmark for subsequent judgments on whether lesion changes are abnormal.

[0024] In one possible implementation, a feature library of lesion changes is constructed. Step S100 further includes step S110, obtaining a training set of endoscopic videos corresponding to known real lesion samples. The training set of endoscopic videos includes lesion type annotation information. Specifically, data is prepared by collecting a large number of endoscopic video clips that have been reviewed and annotated by medical experts. These annotations, i.e., labels, indicate the lesion areas appearing in the videos and their specific pathological types, such as polyps, ulcers, erosions, and tumors. These annotated datasets form the basis for supervised machine learning.

[0025] Step S120: By analyzing the endoscopic video training set, a training set of lesion change behavior features based on temporal changes is collected. This training set includes the instantaneous velocity and acceleration calculated from the centroid position of each lesion sample, the area value calculated based on the lesion contour, pixel distribution similarity, and boundary ambiguity. Specifically, feature engineering is used to quantify the dynamic behavior of the lesions. Specifically, in consecutive video frames, the centroid, i.e., the geometric center, of the lesion is located using target tracking algorithms such as optical flow or correlation filtering. The displacement of the centroid between adjacent frames is calculated, and combined with the video frame rate, the instantaneous velocity is obtained. Further, the rate of change of velocity at adjacent time points is calculated to obtain the instantaneous acceleration. This reflects the macroscopic motion characteristics of the lesion. In each frame, the pixels of the segmented lesion region contour are counted to obtain the area. Within a set time window, the maximum and minimum areas are calculated, and the difference or ratio is the area value, used to measure drastic changes in lesion size. RGB / HSV color and texture features within the lesion region are extracted to form a feature histogram. Texture features are calculated using Local Binary Pattern (LBP) or Gray-Level Co-occurrence Matrix (GLCM). The similarity of the lesion region feature histograms between consecutive frames is calculated using Bach's distance or correlation coefficient to measure the stability of the lesion's internal appearance features. Gradient strength is calculated for pixels in the lesion boundary region using the Sobel or Canny operator. Clear boundaries result in high and concentrated gradient values, while blurred boundaries result in low and dispersed gradient values. The degree of blurring is quantified by statistically analyzing the distribution of gradient magnitudes such as the mean and variance of the boundary region.

[0026] Step S130: A classification model is trained on the lesion change behavior feature training set according to the lesion type labeling information to obtain prior lesion change behavior features corresponding to each lesion type. Specifically, a machine learning model is used to learn the feature patterns of different lesion types. Multiple features extracted in step S120 are combined into a multi-dimensional feature vector, and each vector is associated with the lesion type labeled in step S110. Then, classification algorithms such as Support Vector Machine (SVM), Random Forest, or Gradient Boosting Decision Tree are used to train this labeled feature dataset. After training, the model can learn the typical distribution range of dynamic feature vectors for each lesion type, such as polyps and ulcers. These distribution ranges, such as common velocity intervals and common similarity thresholds, constitute the prior lesion change behavior features.

[0027] Step S140: Construct a lesion change behavior feature library according to the mapping relationship between each lesion type and prior lesion change behavior features. Specifically, solidify the training results, that is, store the classification model trained in step S130 and the feature parameter ranges corresponding to each lesion type learned by the model in the form of a lookup table, hash table, or database. This database is the lesion change behavior feature library, in which lesion category is used as the search basis, and each category stores the standard value range of its various features.

[0028] Step S200: Extract at least one candidate lesion region from the endoscopic video stream; calculate the corresponding lesion change behavior features for each candidate lesion region in consecutive frames; call the lesion change behavior feature library to obtain the prior lesion change behavior features of the lesion change behavior features; and determine whether an abnormal change tracking event has occurred by comparing the prior lesion change behavior features and the lesion change behavior features.

[0029] Specifically, real-time monitoring and anomaly detection are performed by first detecting suspected lesion areas in the endoscopic video stream, then tracking these areas and calculating their dynamic characteristics, and finally comparing them with prior knowledge in the lesion change behavior feature database to determine whether there may be deviations in the tracking process or abnormal changes in the target.

[0030] In one possible implementation, at least one candidate lesion region is extracted from the endoscopic video stream. Step S200 further includes step S210, which involves training a target detection model using a multi-scale convolutional neural network, and extracting lesion regions from the endoscopic video stream according to the target detection model to obtain initial candidate lesion regions. Specifically, automatic initial localization of lesions is performed using a deep learning-based object detection framework, such as Faster R-CNN, YOLO, or SSD. The core of these models is a convolutional neural network. Multi-scale refers to incorporating mechanisms such as feature pyramid networks into the network structure, enabling the model to simultaneously utilize deep semantic features and shallow detail features, thereby effectively detecting lesions of different sizes. The network is trained using the labeled data from step S110. The trained model can infer from the input video frames and directly output bounding box coordinates and confidence scores. The regions within these bounding boxes are the initial candidate lesion regions.

[0031] Step S220 involves fusing and filtering the initial candidate lesion regions to remove reflective interference, outputting at least one candidate lesion region. Specifically, common reflective false positive interference in endoscopic images is removed. This is achieved by using image processing algorithms, such as threshold-based or color space-based segmentation, to detect bright reflective regions in the image. Then, the overlap between each initial candidate lesion region and these reflective regions is calculated based on the Intersection over Union (IoU). If the overlap exceeds a certain threshold, the candidate region is considered a reflective artifact rather than a real lesion and is removed from the candidate list. The regions that remain are the regions of the real lesions, which are the output candidate lesion regions.

[0032] In one possible implementation, after extracting at least one candidate lesion region from the endoscopic video stream, step S200 further includes step S230, dividing the input endoscopic video stream and outputting consecutive frames of endoscopic video. Specifically, the video stream is read sequentially through a video decoding library or image acquisition interface and decomposed into static images frame by frame to form an image sequence.

[0033] Step S240 involves setting a tracking detection step size for each candidate lesion region, dividing the continuous frames of the endoscopic video according to the tracking detection step size, and outputting the tracking continuous frames for each candidate lesion region. Specifically, an adaptive tracking strategy is implemented, meaning that instead of performing complete feature calculation and anomaly judgment for every frame, different detection frequencies are set according to the target characteristics to balance computational overhead and tracking accuracy. Step S240 further includes step S241, where the lesion type of each candidate lesion region is analyzed based on the lesion change behavior feature library. Specifically, static features of the current candidate lesion region, such as color, texture, and shape, are extracted, and then a pre-trained lesion classifier is used to classify it to obtain the most probable lesion type. The lesion classifier can be part of the model in step S130 or an independent lightweight classification network.

[0034] Step S242: Identify the movement speed and scene complexity of each candidate lesion region based on its lesion type, and configure a tracking detection step size for each candidate lesion region based on the movement speed and scene complexity. Specifically, a dynamic strategy is configured by pre-setting a configuration table within the system, associating the lesion type with the expected movement speed and scene complexity. For situations with fast movement speed and large scene changes, such as in environments with strong peristalsis like the stomach or where the lesion itself moves quickly, tracking is easily lost. In this case, a shorter tracking detection step size is set to improve the tracking response speed; for example, a complete feature calculation and anomaly judgment are performed every 2-3 frames. For situations with slow movement speed and stable scenes, such as in relatively static intestinal segments, a longer step size is set to save computational resources; for example, a complete detection is performed every 10-15 frames. Based on the type determined in step S241, this configuration table is queried to assign an appropriate detection step size to each candidate region. Then, a subsequence is extracted from the continuous frames of the endoscopic video according to this step size; this subsequence is the tracking continuous frame used for tracking and feature calculation.

[0035] Step S250: For each candidate lesion region, calculate the corresponding lesion change behavior features in the tracking continuous frames. Specifically, this step is similar to S120, but applies to candidate regions in real-time video. The specific technical means are exactly the same: on the assigned tracking continuous frame subsequence, calculate the instantaneous velocity, acceleration, area value, pixel distribution similarity, and boundary blur of the candidate region.

[0036] In one possible implementation, the prior lesion change behavior features of the lesion change behavior features are obtained by calling the lesion change behavior feature library. Step S200 further includes step S260, which involves calling the lesion change behavior feature library to analyze the lesion type of the lesion change behavior features, and mapping the prior lesion change behavior features according to the lesion type. The prior lesion change behavior features include the instantaneous velocity change range, acceleration change range, area positive value change range, pixel distribution similarity change range, and boundary ambiguity change range. Specifically, a benchmark is queried; that is, the real-time feature vector calculated in step S250 is input again into the lesion classifier or the result of S241 is directly used to determine the most likely lesion type of the currently tracked candidate region. Then, using this type as the retrieval identifier, the lesion change behavior feature library constructed in step S140 is queried to obtain the normal variation range of various features of this type of lesion.

[0037] In one possible implementation, step S200 further includes step S270, which involves constructing a change anomaly discrimination model based on multidimensional feature deviations. These multidimensional feature deviations include instantaneous velocity change deviation, acceleration deviation, area positive value deviation, pixel distribution similarity deviation, and boundary blurring deviation. Specifically, an anomaly measurement standard is established, and a comprehensive scoring function is defined. For each feature, the absolute difference between its real-time value and the value within the normal range for that type retrieved from the lesion change behavior feature database is calculated, or it is determined whether it exceeds the normal range. Then, a weight is assigned to the deviation of each feature, and finally, the deviations of all features are weighted and fused to form a total change anomaly score.

[0038] Step S280: Based on the deviation value between the prior lesion change behavior characteristics and the lesion change behavior characteristics according to the change anomaly discrimination model, an anomaly score is output. Specifically, anomaly judgment is performed by substituting the real-time feature value calculated in step S250 and the normal range queried in step S260 into the scoring function defined in step S270 for calculation, and finally outputting a quantified anomaly score.

[0039] Step S290: If the abnormality score is greater than a preset threshold, an abnormality tracking event is determined to have occurred; otherwise, no abnormality tracking event is determined to have occurred. Specifically, an empirical threshold is set, and the abnormality score calculated in step S280 is compared with this threshold. If the score exceeds the threshold, it indicates that the behavior of the currently tracked target differs too much from the typical behavior of similar lesions, meaning that the tracker has lost track of it. That is, after the target is lost, the tracking box attaches to irrelevant tissue or noise, resulting in drastic changes in features, or the lesion itself has undergone acute abnormal changes. Both of these situations are defined as abnormality tracking events.

[0040] Step S300: If no abnormal tracking event occurs, output the lesion tracking result; if an abnormal tracking event occurs, start the dynamic relocation mechanism to track and relocate the candidate lesion area where the abnormal tracking event occurred.

[0041] Specifically, if tracking is normal, the location and bounding box of the lesion in the current frame are continuously output. If an abnormal event is detected, a relocation process is triggered to attempt to retrieve the lost tracking target.

[0042] In one possible implementation, if an abnormal tracking event occurs, a dynamic relocation mechanism is initiated to track and relocate the candidate lesion region where the abnormal tracking event occurred. Step S300 further includes step S310, which determines the current video frame where the abnormal tracking event occurred in the candidate lesion region, and performs backtracking based on the current video frame to obtain the previous video frame. Specifically, relocation initialization is performed, and the video frame that triggered the abnormal alarm is recorded, denoted as frame T. Then, the video frame that was successfully tracked before the abnormal frame is read from memory or cache, denoted as frame T-1.

[0043] Step S320: Based on the predicted position of the candidate lesion region in the current video frame according to the previous video frame, a candidate search region is generated using the predicted position. The candidate lesion region is then re-tracked and located within the candidate search region. Specifically, after triggering an anomaly alarm, a reasonable range is determined in the current video frame to re-search for any potentially lost lesion targets. Specifically, a linear uniform velocity model or a Kalman filter is used as the motion model. Based on the lesion's position, size, and historical motion trajectory in frame T-1, its approximate location in frame T is predicted. Using this predicted location as the center, a rectangular or elliptical region larger than the original lesion region is defined as the candidate search region to avoid searching the entire image and improve the efficiency and accuracy of relocalization.

[0044] The size of the search area is determined based on the maximum expected motion speed. A maximum expected pixel displacement value is preset based on the currently tracked lesion type and the typical movement speed of the endoscope. For example, the system is set so that the lesion movement between adjacent frames will not exceed a certain value, such as 50 pixels. Then, the radius or side length of the search area is based on this maximum displacement value, multiplied by a safety factor, such as 2, to ensure that even if the target moves slightly faster, it will still be covered. The calculation formula can be simplified to: Search area side length = Original lesion area side length + 2 × (Maximum expected pixel displacement × Safety factor). The resulting area is larger than the original area.

[0045] In one possible implementation, the candidate lesion region is re-tracked and located within the candidate search area. Step S320 further includes step S321: acquiring the lesion multimodal features of the current video frame, performing feature similarity matching between the lesion multimodal features of the current video frame and the candidate lesion region, and returning the similarity matching result. The similarity matching result includes the re-tracked and located candidate lesion region. If the similarity matching result returns empty, the candidate lesion region is marked, and its predicted position in the next video frame is predicted for re-tracking and relocation. Specifically, precise matching is performed within the search area. That is, within the candidate search area defined in step S320, multiple sub-regions are generated using a sliding window or a Region Proposal Network (RPN). For each sub-region, its multimodal features, including appearance features and shape features, are extracted. The features of the successfully tracked template in frame T-1 of step S310 are compared with the features of each sub-region within the search area using cosine similarity calculation to find the sub-region with the highest similarity. If the highest similarity exceeds a preset confidence threshold, the relocalization is considered successful, the sub-region is updated as a new candidate lesion region, and tracking continues. If the similarity of all sub-regions within the search area is below the threshold, the relocalization in the current frame is considered to have failed. The target is marked as temporarily lost, and based on the previous motion model, its possible location in the next frame is predicted. The relocalization process continues in the next frame, i.e., it jumps back to step S320, until the relocalization is successful or the target is determined to have permanently left the field of view.

[0046] This application employs a technique of constructing a lesion change behavior feature library obtained through sampling and training of known real lesion samples, extracting candidate lesion regions from the endoscopic video stream, calculating the lesion change behavior features in consecutive frames, comparing them with prior features in the feature library to determine whether an abnormal tracking event has occurred. If no abnormal event occurs, the lesion tracking result is output; if an abnormal event occurs, a dynamic relocation mechanism is activated to track and relocate the abnormal candidate lesion region. This technique solves the technical problem of insufficient accuracy and reliability in existing dynamic lesion tracking of endoscopic videos, and achieves the technical effect of improving the accuracy and stability of lesion tracking.

[0047] In the above text, refer to Figure 1 A method for dynamic lesion tracking via endoscopic video according to an embodiment of the present invention is described in detail. Next, reference will be made to... Figure 2 This invention describes an endoscopic video dynamic lesion tracking system according to an embodiment of the present invention.

[0048] An endoscopic video dynamic lesion tracking system according to an embodiment of the present invention addresses the technical problems of insufficient accuracy and reliability in existing endoscopic video dynamic lesion tracking systems, thereby improving the accuracy and stability of lesion tracking. The endoscopic video dynamic lesion tracking system includes: a lesion change behavior feature database construction module 10, a judgment module 20, and a processing module 30.

[0049] The lesion change behavior feature library construction module 10 is used to construct a lesion change behavior feature library, which is obtained by sampling and training prior lesion change behavior features on known real lesion samples; the judgment module 20 is used to extract at least one candidate lesion region from the endoscopic video stream, calculate the corresponding lesion change behavior features for each candidate lesion region in consecutive frames, call the lesion change behavior feature library to obtain the prior lesion change behavior features of the lesion change behavior features, and determine whether an abnormal change tracking event has occurred by comparing the prior lesion change behavior features and the lesion change behavior features; the processing module 30 is used to output the lesion tracking result if no abnormal change tracking event has occurred, and to start a dynamic relocation mechanism to track and relocate the candidate lesion region where the abnormal tracking event has occurred if an abnormal change tracking event has occurred.

[0050] The specific configuration of the judgment module 20 is described in detail below: As mentioned above, after extracting at least one candidate lesion region from the endoscopic video stream, the judgment module 20 may further include: a video stream segmentation unit for segmenting the input endoscopic video stream and outputting continuous endoscopic video frames; a frame segmentation unit for setting a tracking detection step size for each candidate lesion region, segmenting the continuous endoscopic video frames according to the tracking detection step size, and outputting a tracking continuous frame for each candidate lesion region; and a lesion change behavior feature calculation unit for calculating the corresponding lesion change behavior features for each candidate lesion region under the tracking continuous frame.

[0051] Specifically, a tracking detection step size is set for each candidate lesion region, and the frame segmentation unit may further include: a lesion type analysis subunit for analyzing the lesion type of each candidate lesion region based on the lesion change behavior feature library; and a tracking detection step size configuration subunit for identifying the motion speed and scene complexity based on the lesion type of each candidate lesion region, and configuring a tracking detection step size for each candidate lesion region based on the motion speed and scene complexity.

[0052] The detailed description of the specific configuration of the lesion change behavior feature library construction module 10 is as follows: As mentioned above, the lesion change behavior feature library construction module 10 may further include: an endoscopy video training set acquisition unit for acquiring an endoscopy video training set corresponding to known real lesion samples, the endoscopy video training set including lesion type annotation information; a lesion change behavior feature training set acquisition unit for acquiring a lesion change behavior feature training set based on time-series changes by analyzing the endoscopy video training set, the lesion change behavior feature training set including instantaneous velocity and acceleration calculated from the centroid position corresponding to each lesion sample, area positive value calculated based on the lesion contour, pixel distribution similarity and boundary ambiguity; a classification model training unit for training a classification model on the lesion change behavior feature training set according to the lesion type annotation information, and acquiring prior lesion change behavior features corresponding to each lesion type; and a lesion change behavior feature library construction unit for constructing a lesion change behavior feature library according to the mapping relationship between each lesion type and prior lesion change behavior features.

[0053] The judgment module 20 may further include: a priori lesion change behavior feature acquisition unit, which calls the lesion change behavior feature library to obtain the priori lesion change behavior features of the lesion change behavior features, and maps the priori lesion change behavior features according to the lesion type. The priori lesion change behavior features include the instantaneous velocity change range, acceleration change range, area value change range, pixel distribution similarity change range, and boundary ambiguity change range.

[0054] The determination module 20, which determines whether an abnormal change tracking event has occurred by comparing the prior lesion change behavior characteristics with the lesion change behavior characteristics, may further include: an abnormal change discrimination model construction unit for constructing an abnormal change discrimination model based on multi-dimensional feature deviations, wherein the multi-dimensional feature deviations include instantaneous velocity change deviation, acceleration deviation, area positive value deviation, pixel distribution similarity deviation, and boundary blurring deviation; a deviation value comparison unit for comparing the deviation values ​​of the prior lesion change behavior characteristics and the lesion change behavior characteristics according to the abnormal change discrimination model, and outputting an abnormal change score; and a determination unit for determining that an abnormal change tracking event has occurred if the abnormal change score is greater than a preset threshold, otherwise determining that no abnormal change tracking event has occurred.

[0055] The detailed description of the specific configuration of the processing module 30 is explained as follows: As mentioned above, if a change abnormal tracking event occurs, a dynamic relocation mechanism is activated to track and relocate the candidate lesion area where the abnormal tracking event occurs. The processing module 30 may further include: a backtracking unit for determining the current video frame where the abnormal tracking event occurs in the candidate lesion area, and backtracking based on the current video frame to obtain the previous video frame; and a re-tracking and relocation unit for predicting the predicted position of the candidate lesion area in the current video frame according to the previous video frame, generating a candidate search area with the predicted position, and re-tracking and relocating the candidate lesion area in the candidate search area.

[0056] The re-tracking and repositioning unit in the candidate search area may further include: a feature similarity matching subunit for collecting multimodal features of lesions in the current video frame, performing feature similarity matching between the multimodal features of lesions in the current video frame and the candidate lesion area, and returning a similarity matching result. The similarity matching result includes the re-tracked and repositioned candidate lesion area. If the similarity matching result returns empty, the candidate lesion area is marked and the predicted position of the candidate lesion area in the next video frame is predicted for re-tracking and repositioning.

[0057] The judgment module 20, which extracts at least one candidate lesion region from the endoscopic video stream, may further include: a target detection model training unit for training a target detection model using a multi-scale convolutional neural network, and extracting lesion regions from the endoscopic video stream according to the target detection model to obtain initial candidate lesion regions; and a reflection interference fusion screening unit for performing reflection interference fusion screening on the initial candidate lesion regions to output at least one candidate lesion region.

[0058] The endoscopic video dynamic lesion tracking system provided in this embodiment of the invention can execute the endoscopic video dynamic lesion tracking method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0059] Although this application makes various references to certain modules in the system according to the embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of this invention.

[0060] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. An endoscopic video dynamic lesion tracking method, characterized by, The method comprises: constructing a lesion change behavior feature library, the lesion change behavior feature library is obtained by training the prior lesion change behavior features of the known real lesion samples; extracting at least one candidate lesion area of the endoscope video stream, calculating the corresponding lesion change behavior features of each candidate lesion area under the continuous frames, calling the lesion change behavior feature library to obtain the prior lesion change behavior features of the lesion change behavior features, and judging whether a change abnormal tracking event occurs by comparing the prior lesion change behavior features and the lesion change behavior features; If no change abnormal tracking event occurs, output the lesion tracking result, if a change abnormal tracking event occurs, start a dynamic relocation mechanism to track and relocate the candidate lesion area where the abnormal tracking event occurs, including: determining the current video frame where the candidate lesion area appears the abnormal tracking event, and performing backtracking based on the current video frame to obtain a previous video frame, including reading the last successfully tracked video frame before the abnormal frame; predicting the predicted position of the candidate lesion area in the current video frame according to the previous video frame, generating a candidate search area at the predicted position, and re-tracking and positioning the candidate lesion area in the candidate search area; constructing a lesion change behavior feature library, the method comprising: obtaining an endoscope video training set corresponding to a known real lesion sample, the endoscope video training set comprising lesion type annotation information; by analyzing the endoscope video training set, collecting a lesion change behavior feature training set based on time sequence change, the lesion change behavior feature training set comprising instantaneous speed and acceleration calculated based on the center of mass position of each lesion sample, area extreme value calculated based on lesion contour, pixel distribution similarity and boundary fuzziness; performing classification model training on the lesion change behavior feature training set according to the lesion type annotation information, and obtaining prior lesion change behavior features corresponding to each lesion type; constructing a lesion change behavior feature library according to the mapping relationship between each lesion type and the prior lesion change behavior features; calling the lesion change behavior feature library to obtain the prior lesion change behavior features of the lesion change behavior features, the method comprising: calling the lesion change behavior feature library to analyze the lesion type of the lesion change behavior features, and mapping to obtain the prior lesion change behavior features according to the lesion type; The prior lesion change behavior features include instantaneous speed change range, acceleration change range, area extreme value change range, pixel distribution similarity change range and boundary fuzziness change range.

2. The method of claim 1, wherein, After extracting at least one candidate lesion area of the endoscope video stream, the method further comprises: dividing the input endoscope video stream to output endoscope video continuous frames; setting a tracking detection step length for each candidate lesion area, dividing the endoscope video continuous frames according to the tracking detection step length to output tracking continuous frames of each candidate lesion area; calculating the corresponding lesion change behavior features of each candidate lesion area under the tracking continuous frames.

3. The endoscopic video dynamic lesion tracking method of claim 2, wherein, setting a tracking detection step length for each candidate lesion area, the method comprising: According to the lesion change behavior feature library, a lesion type of each candidate lesion region is analyzed; According to the lesion type of each candidate lesion region, a motion speed and a scene complexity are identified, and a tracking detection step length is configured for each candidate lesion region based on the motion speed and the scene complexity.

4. The endoscopic video dynamic lesion tracking method of claim 1, wherein, By comparing the prior lesion change behavior feature and the lesion change behavior feature, it is judged whether a change abnormal tracking event occurs, and the method comprises: A change abnormality discrimination model based on multi-dimensional feature deviation is constructed, and the multi-dimensional feature deviation comprises instantaneous speed change deviation, acceleration deviation, area extreme value deviation, pixel distribution similarity deviation, and boundary blur deviation; According to the change abnormality discrimination model, the deviation values of the prior lesion change behavior feature and the lesion change behavior feature are compared, and a change abnormality score is output; If the change abnormality score is greater than a preset threshold, it is determined that a change abnormal tracking event occurs; Otherwise, it is determined that no change abnormal tracking event occurs.

5. The endoscopic video dynamic lesion tracking method of claim 1, wherein, In the candidate search region, a candidate lesion region is re-tracked and positioned, and the method comprises: Lesion multi-modal features of a current video frame are collected, feature similarity matching is performed on the lesion multi-modal features of the current video frame and the candidate lesion region, and a similarity matching result is returned, wherein the similarity matching result comprises the candidate lesion region after re-tracking and positioning; If the similarity matching result is empty, the candidate lesion region is marked and a predicted position of the candidate lesion region in a next video frame is predicted for tracking repositioning, including: if the similarity of all sub-regions in the search region is lower than a threshold, it is considered that the repositioning in the current frame fails, the target is temporarily lost, and based on the previous motion model, it is predicted that the target may appear in the position of the next frame, and the repositioning process is continued in the next frame until the repositioning is successful or the target is determined to permanently leave the field of view.

6. The endoscopic video dynamic lesion tracking method of claim 1, wherein, At least one candidate lesion region of an endoscope video stream is extracted, and the method comprises: A multi-scale convolutional neural network is used to train a target detection model, lesion region extraction is performed on the endoscope video stream according to the target detection model, and an initial candidate lesion region is obtained; The initial candidate lesion region is subjected to anti-reflection interference fusion screening, and at least one candidate lesion region is output.

7. An endoscopic video dynamic lesion tracking system, comprising: The system is used to implement the endoscope video dynamic lesion tracking method of any one of claims 1-6, and the system comprises: A lesion change behavior feature library construction module is configured to construct a lesion change behavior feature library, wherein the lesion change behavior feature library is obtained by training prior lesion change behavior features of known real lesion samples; A judgment module is configured to extract at least one candidate lesion region of an endoscope video stream, calculate corresponding lesion change behavior features of each candidate lesion region in consecutive frames, call prior lesion change behavior features of the lesion change behavior features from the lesion change behavior feature library, and judge whether a change abnormal tracking event occurs by comparing the prior lesion change behavior features and the lesion change behavior features. The processing module is used for outputting the lesion tracking result if no change abnormal tracking event occurs, and starting a dynamic repositioning mechanism to track and reposition the candidate lesion area of the abnormal tracking event if the change abnormal tracking event occurs.

Images