Multi-target tracking and detection methods, apparatus, equipment and readable storage media

Abstract

A multi-object tracking and detection method, apparatus, device, and readable storage medium are disclosed. The multi-object tracking and detection method includes: decomposing a colonoscopy video to be detected into multiple frames in chronological order; sequentially detecting multiple polyps in each frame using a polyp detection model to obtain detection bounding boxes and confidence scores; predicting multiple current trajectories based on multiple historical trajectories using a Kalman filter; constructing a multi-objective utility matrix of intersection-over-union ratio (IoU), confidence score, and appearance similarity based on each detection bounding box, predicted bounding box, confidence score, first appearance feature vector, and second appearance feature vector; solving the multi-objective utility matrix using a multi-objective optimization algorithm to obtain the optimal solution; and updating and managing the trajectories based on the optimal solution. This application improves the accuracy of multi-polyp target tracking and detection compared to multi-object tracking techniques that rely solely on single indicators such as IoU.

Description

Technical Field

[0001] This application relates to the field of colonoscopy polyp detection technology, and in particular to a multi-target tracking detection method, apparatus, device, and readable storage medium. Background Technology

[0002] Multi-target tracking technology has important application value in the field of medical image processing, especially in the dynamic monitoring of polyps in colonoscopy videos, which can provide continuous lesion localization and identification for intelligent diagnosis and treatment systems.

[0003] However, existing multi-target tracking technologies face multiple challenges when tracking and detecting multiple polyps in colonoscopy videos, such as abrupt changes in polyp movement, complex deformation, and noise interference, resulting in inaccurate detection results. Summary of the Invention

[0004] This application provides a multi-target tracking and detection method, apparatus, device, and readable storage medium, aiming to solve the technical problem that existing multi-target tracking technologies face multiple challenges when tracking and detecting multiple polyps in colonoscopy videos, such as abrupt changes in polyp movement, complex deformation, and noise interference, resulting in inaccurate detection results.

[0005] In a first aspect, embodiments of this application provide a multi-target tracking and detection method, the multi-target tracking and detection method comprising:

[0006] The colonoscopy video to be tested was decomposed into multiple frames in chronological order;

[0007] For each frame of the image, the polyp detection model is used to detect the image to obtain multiple detection bounding boxes and confidence scores for the detected polyps. Based on each detection bounding box, the first appearance feature vector of each detected polyp is extracted from the image.

[0008] Multiple current trajectories are predicted based on multiple historical trajectories using a Kalman filter, and a second appearance feature vector for each current trajectory is extracted from the image based on the predicted bounding box of each current trajectory.

[0009] Based on each detected bounding box, predicted bounding box, confidence score, first appearance feature vector, and second appearance feature vector, construct a multi-objective utility matrix of intersection-over-union ratio, confidence score, and appearance similarity.

[0010] The optimal solution is obtained by solving the multi-objective utility matrix using a multi-objective optimization algorithm.

[0011] The trajectory is updated and managed based on the optimal solution.

[0012] Optionally, before using a Kalman filter to predict multiple current trajectories based on multiple historical trajectories, the following steps are included:

[0013] Based on the detection bounding boxes of multiple polyps detected in historical frames by the polyp detection model, multiple historical trajectories are established.

[0014] Based on multiple historical trajectories, a Kalman filter is used to model the motion state of the detected polyps, resulting in a well-established Kalman filter.

[0015] Optionally, the step of constructing a multi-objective utility matrix of intersection-over-union ratio, confidence score, and appearance similarity based on each detected bounding box, predicted bounding box, confidence score, first appearance feature vector, and second appearance feature vector includes:

[0016] Based on each detected bounding box and each predicted bounding box, the intersection-union utility matrix is ​​constructed.

[0017] Based on each confidence level, a confidence-utility matrix is ​​constructed.

[0018] Based on each first appearance feature vector and each second appearance feature vector, an appearance similarity utility matrix is ​​constructed.

[0019] A multi-objective utility matrix is ​​obtained by using the intersection-union utility matrix, the confidence utility matrix, and the appearance similarity utility matrix.

[0020] Optionally, the step of using a multi-objective optimization algorithm to solve the multi-objective utility matrix to obtain the optimal solution includes:

[0021] The Pareto front solution set is obtained by solving the multi-objective utility matrix using a multi-objective optimization algorithm.

[0022] For the Pareto front solution set, a two-stage optimization mechanism of knee-guided selection and human-inspired decision-making is used to select the optimal solution.

[0023] Optionally, the step of using a multi-objective optimization algorithm to solve the multi-objective utility matrix to obtain the Pareto front solution set includes:

[0024] The objective function is established based on the intersection-union utility matrix, the confidence utility matrix, and the appearance similarity utility matrix, and constraints are set.

[0025] Multiple solutions are generated as the initial population;

[0026] Stratify and rank the population according to Pareto dominance;

[0027] Two-point crossover and bit-flip mutation are used to generate offspring solutions, which are then merged into the population, and a first preset number of optimal solutions are retained in the population.

[0028] If the number of iterations does not reach the preset number, return to the step of stratifying and sorting the population according to the Pareto dominance relationship;

[0029] If the number of iterations reaches the preset number, the Pareto front solution set will be output.

[0030] Optionally, the optimal solution is obtained by using a two-stage selection mechanism of knee-guided screening and human-inspired decision-making for the Pareto front solution set, including:

[0031] Solution set of Pareto front f The Pareto front solution set is calculated using the formula for each pair of solutions. i ,p j The trade-off The formula is:

[0032] Where f1, f2, and f3 represent the crossover union ratio, confidence score, and appearance similarity objective function, respectively;

[0033] Retain satisfaction The solution set K of the knee points, where, ;

[0034] The optimal solution is determined from the knee point solution set according to the priority order of highest intersection-union ratio, highest appearance similarity, and highest confidence.

[0035] Optionally, the optimal solution includes whether each detected polyp matches each current trajectory, the trajectories are divided into formal trajectories and temporary trajectories, and the step of updating and managing the trajectories according to the optimal solution includes:

[0036] For each detected polyp, if the detected polyp has a matching current trajectory, and the matching current trajectory is a formal trajectory, then the Kalman filter is updated using the bounding box of the detected polyp.

[0037] For each detected polyp, if there is no matching current trajectory for the detected polyp and the confidence of the detected polyp is greater than the preset confidence, a temporary trajectory is created for the detected polyp, and the Kalman filter is updated using the bounding box of the detected polyp.

[0038] For each current trajectory, if the current trajectory is a temporary trajectory, and if the temporary trajectory has a second consecutive preset number of detected polyps that match it, then the temporary trajectory is converted into a formal trajectory; if the temporary trajectory does not have a third consecutive preset number of detected polyps that match it, then the temporary trajectory is deleted.

[0039] For each current trajectory, if the current trajectory is a formal trajectory and there is no fourth preset number of detected polyps matching it, then the formal trajectory is terminated.

[0040] Secondly, embodiments of this application provide a multi-target tracking and detection device, the multi-target tracking and detection device comprising:

[0041] The extraction module is used to decompose the colonoscopy video to be tested into multiple frames of images in chronological order;

[0042] The detection module is used to sequentially detect polyps in each frame of the image using a polyp detection model, obtain multiple detection bounding boxes and confidence scores for the detected polyps, and extract the first appearance feature vector of each detected polyp from the image based on each detection bounding box.

[0043] The prediction module is used to predict multiple current trajectories based on multiple historical trajectories using a Kalman filter, and extract the second appearance feature vector of each current trajectory from the image based on the predicted bounding box of each current trajectory.

[0044] The module is used to construct a multi-objective utility matrix of intersection-over-union ratio, confidence, and appearance similarity based on each detected bounding box, predicted bounding box, confidence score, first appearance feature vector, and second appearance feature vector.

[0045] The solution module is used to solve the multi-objective utility matrix using a multi-objective optimization algorithm to obtain the optimal solution;

[0046] The update module is used to update and manage the trajectory based on the optimal solution.

[0047] Thirdly, embodiments of this application provide a multi-target tracking and detection device, which includes a processor, a memory, and a multi-target tracking and detection program stored in the memory and executable by the processor. When the multi-target tracking and detection program is executed by the processor, it implements the steps of the multi-target tracking and detection method described above.

[0048] Fourthly, embodiments of this application provide a readable storage medium storing a multi-target tracking and detection program, wherein when the multi-target tracking and detection program is executed by a processor, it implements the steps of the multi-target tracking and detection method as described above.

[0049] The beneficial effects of the technical solutions provided in this application include:

[0050] In this embodiment, the colonoscopy video to be detected is decomposed into multiple frames in chronological order. For each frame, a polyp detection model is used to detect multiple polyps and obtain detection bounding boxes and confidence scores. Based on each detection bounding box, a first appearance feature vector of each polyp is extracted from the image. A Kalman filter is used to predict multiple current trajectories based on multiple historical trajectories. Based on the predicted bounding box of each current trajectory, a second appearance feature vector of each current trajectory is extracted from the image. Based on each detection bounding box, predicted bounding box, confidence score, first appearance feature vector, and second appearance feature vector, a multi-objective utility matrix of intersection-union ratio, confidence score, and appearance similarity is constructed. A multi-objective optimization algorithm is used to solve the multi-objective utility matrix to obtain the optimal solution. Based on the optimal solution, the trajectory is updated and managed. Through the embodiments of this application, based on the detection bounding box of each detected polyp and the predicted bounding box of each current trajectory, the intersection-union ratio (IU) of the bounding boxes of each detected polyp and each current trajectory can be obtained. Based on the first appearance feature vector of each detected polyp and the second appearance feature vector of each current trajectory, the appearance similarity between each detected polyp and each current trajectory can be obtained. Then, the confidence of each detected polyp is added, and a multi-objective utility matrix is ​​constructed to solve for the optimal solution. Compared with multi-objective tracking technology that only relies on a single indicator such as IU, it can improve the accuracy of multi-polyp target tracking and detection, and can better cope with multiple challenges such as sudden motion, complex deformation and interference noise. If only IU is relied on, when the polyp undergoes rapid displacement due to endoscopic operation, causing the IU to drop to zero, it will easily lead to trajectory interruption or identity switching. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating an embodiment of the multi-target tracking and detection method of this application;

[0052] Figure 2 This is a schematic diagram comparing experimental results data of an embodiment of the multi-target tracking and detection method of this application;

[0053] Figure 3 This is a schematic diagram of the model structure of an embodiment of the multi-target tracking and detection method of this application;

[0054] Figure 4 This is a schematic diagram of the system flow of an embodiment of the multi-target tracking and detection method of this application;

[0055] Figure 5 This is a schematic diagram of the functional modules of an embodiment of the multi-target tracking and detection device of this application;

[0056] Figure 6 This is a schematic diagram of the hardware structure of the multi-target tracking and detection device involved in the embodiments of this application. Detailed Implementation

[0057] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0059] Firstly, embodiments of this application provide a multi-target tracking and detection method.

[0060] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the multi-target tracking and detection method of this application, as shown below. Figure 1 As shown, multi-target tracking and detection methods include:

[0061] Step S10: Decompose the colonoscopy video to be tested into multiple frames in chronological order.

[0062] In this embodiment, the colonoscopy video to be detected consists of multiple frames. Since multiple polyps need to be tracked and detected in the colonoscopy video, it is decomposed into multiple frames in chronological order so that each frame can be detected sequentially in the same order. Preprocessing such as denoising and enhancement can be performed on the extracted images; the specific processing steps are not described further here.

[0063] Step S20: For each frame of the image, the polyp detection model is used to detect the image to obtain multiple detection bounding boxes and confidence scores of the detected polyps, and the first appearance feature vector of each detected polyp is extracted from the image based on each detection bounding box.

[0064] In this embodiment, the polyp detection model can be, for example, the YOLO-OB model. Before use, the YOLO-OB model is pre-trained using a dataset to obtain a trained YOLO-OB model. During use, for each frame of the image, the image is input into the YOLO-OB model, and the YOLO-OB model predicts the detection bounding box and confidence score of each detected polyp in the output image. YOLO-OB is an anchorless real-time polyp detection model based on the YOLO framework. Addressing the problem of missed detection of multi-scale polyps in colonoscopy images, it proposes a solution combining cross-level feature fusion and dynamic training optimization. The YOLO-OB model can significantly improve the detection accuracy of small polyps while maintaining real-time performance.

[0065] Furthermore, for each detected polyp, the location of the polyp in the image can be determined based on its detection bounding box. Then, the first appearance feature vector of the detected polyp is extracted from the image by the AdaSP re-identification model. The AdaSP re-identification model captures multi-granular spatial features through an adaptive spatial pyramid pooling layer and generates a 2048-dimensional feature vector by combining global average pooling. The extracted feature vector is then processed by L2 normalization and bound to the detection box for storage, forming a trajectory feature description that combines spatial localization and biometric recognition characteristics. This can provide a measurable appearance similarity criterion for multi-target data association.

[0066] Step S30: Use a Kalman filter to predict multiple current trajectories based on multiple historical trajectories, and extract the second appearance feature vector of each current trajectory from the image according to the predicted bounding box of each current trajectory.

[0067] In this embodiment, the Kalman filter is an efficient state estimation algorithm widely used in problems requiring estimation of system state from a series of noisy measurement data. Assuming multiple polyps are detected in each historical frame image, for each polyp, multiple historical trajectories are established based on its position in the historical frames. The Kalman filter is then used to predict multiple current trajectories for the current frame image based on these historical trajectories. Each current trajectory represents the predicted position of each polyp using the Kalman filter. The current trajectory information includes a predicted bounding box. Therefore, based on the predicted bounding box of each current trajectory, the predicted position of each polyp in the image can be determined. Finally, the AdaSP re-identification model extracts the second appearance feature vector of each current trajectory from the image.

[0068] Step S40: Construct a multi-objective utility matrix of intersection-union ratio, confidence and appearance similarity based on each detected bounding box, predicted bounding box, confidence, first appearance feature vector and second appearance feature vector.

[0069] In this embodiment, after the aforementioned steps, multiple polyps and their locations are detected using the polyp detection model for the current frame image. Multiple current trajectories are predicted using a Kalman filter, and each current trajectory represents the location of each polyp predicted by the Kalman filter. Therefore, it is important to correlate and compare the two to improve the accuracy of multi-polyp target tracking and detection. First, the intersection-union utility matrix of the two bounding boxes can be constructed based on each detected bounding box and the predicted bounding box. The confidence utility matrix is ​​constructed using the confidence of the detected polyp. The multi-objective utility matrix of appearance similarity is constructed based on the first appearance feature vector of the detected polyp and the second appearance feature vector of the predicted polyp. Finally, the multi-objective utility matrix is ​​constructed.

[0070] Step S50: Use a multi-objective optimization algorithm to solve the multi-objective utility matrix to obtain the optimal solution.

[0071] In this embodiment, solving the multi-objective utility matrix requires combining multi-objective optimization algorithms and utility theory. Given a three-dimensional multi-objective utility matrix, the first two dimensions represent candidate solutions (solutions), and the third dimension represents the objective function (or utility index). By imposing certain constraints (such as each current trajectory matching at most one detected polyp) and balancing the trade-offs of multiple objectives, the optimal solution set is selected, and then the optimal solution is determined. The optimal solution includes whether each detected polyp matches each current trajectory.

[0072] Step S60: Update and manage the trajectory based on the optimal solution.

[0073] In this embodiment, the trajectory is updated and managed based on the optimal solution result of whether each detected polyp matches each current trajectory. For example, the Kalman filter is updated based on the bounding box information of the matched detected polyps, a new trajectory is created for newly detected polyps, or false detection is marked for newly detected polyps, thereby achieving accurate multi-polyp target tracking and detection.

[0074] In this embodiment, compared to multi-target tracking techniques that rely solely on a single metric such as the cross-union ratio (CUNR), the accuracy of multi-polyp target tracking and detection can be improved. It can better address multiple challenges such as sudden motion changes, complex deformations, and interference noise. For example, relying solely on CUNR can easily lead to trajectory interruption or identity switching when a polyp undergoes rapid displacement due to endoscopic manipulation, causing the CUNR to drop to zero. Through a multi-target collaborative optimization mechanism, the limitations of traditional linear weighted or cascaded matching are overcome. The data association problem in polyp tracking is transformed into a multi-target optimization problem, simultaneously processing three types of discriminative cues: location, confidence level, and appearance similarity. This avoids matching bias caused by a single metric dominating the process.

[0075] In this embodiment, a self-built medical image dataset and a public dataset were used for joint validation. The self-built dataset was constructed from 24 colonoscopy videos, covering polyps of various shapes, complex shooting angles, and lighting scenarios. After frame-by-frame conversion of the videos, invalid edges were cropped, and professional physicians annotated the data to form rectangular bounding boxes and instance IDs for 67,996 polyp instances, generating 58,548 valid sample images. Simultaneously, two public datasets, CVC-ClinicDB (612 images / 29 sequences) and SUN-SEG (49,136 images / 500 sequences), were used for cross-validation. A multi-physician review mechanism ensured the reliability of the annotations, and multi-scenario data fusion testing improved the model's robustness. Using the standardized format of the MOT17 dataset, colonoscopy video sequences containing polyp movement trajectories were collected. Each video segment was stored as a JPG image frame by frame, and annotation files conforming to the MOT17 standard were generated simultaneously.

[0076] Reference Figure 2 , Figure 2 This is a schematic diagram comparing experimental results data of an embodiment of the multi-target tracking and detection method of this application, as shown below. Figure 2 As shown, experiments on three colonoscopy video datasets (including a private dataset Union and two public datasets CVC-ClinicDB and SUN-SEG) demonstrate that the proposed method outperforms existing MOT models in almost all metrics. Specifically, experiments on the public dataset SUN-SEG show that the proposed method improves MOTA, HOTA, and IDF1 metrics by 4.1%, 7.2%, and 5.8% respectively compared to the existing best model (BoT-SORT), while reducing the number of identity switching (IDSw) by 41.5%. In tests on the self-built dataset Union, the proposed method comprehensively surpasses existing technologies with 75.9% MOTA, 44.3% HOTA, and 45.2% IDF1, with the number of ID switching reduced by 22.6% compared to the best competitor (BoT-SORT-ReID). On the CVC-ClinicDB public dataset in low frame rate scenarios, the proposed method overcomes the trajectory breakage problem of traditional trackers, achieving a breakthrough improvement of 73.4% MOTA, which is 12 percentage points higher than the second-best solution. In tests on the high-quality video SUN-SEG dataset, this method achieved outstanding performance of 98.5% MOTA and 98.1% IDF1, fully validating the practical value of the solution in the field of medical image tracking.

[0077] Further, in one embodiment, before step S30, the following steps are included:

[0078] Based on the detection bounding boxes of multiple polyps detected in historical frames by the polyp detection model, multiple historical trajectories are established.

[0079] Based on multiple historical trajectories, a Kalman filter is used to model the motion state of the detected polyps, resulting in a well-established Kalman filter.

[0080] In this embodiment, it is assumed that m polyps are detected from historical frame images. For each polyp, m historical trajectories are established based on the positions of the m polyps in multiple historical frames. An 8-dimensional Kalman filter is used to model the polyp motion state for the m historical trajectories of the current frame image. The specific parameters are defined as follows: State variables: Where (x, y) represents the center coordinates of the polyp bounding box. The aspect ratio is , w is the width of the detection frame, h is the height of the detection frame, and v is the height of the detection frame. x ,v y ,v a ,v h Let be the rate at which the corresponding parameter changes with time. State transition equation: State transition matrix Defined as: ,in, w is the time interval k ~N(0,Q) represents the process noise, and its covariance matrix is... This reflects the tolerance for deviation between the assumption of uniform linear motion and the actual motion. Observation equation: Observation matrix Observation noise v k ~N(0,R), covariance matrix This reflects a high degree of trust in the detector's output. Through prediction-update iteration, the Kalman filter outputs the optimal state estimate, and the prediction formula is: When a new detection arrives, the Kalman gain K is used to... k Implement state correction: .

[0081] Further, in one embodiment, step S40 includes:

[0082] Based on each detected bounding box and each predicted bounding box, the intersection-union utility matrix is ​​constructed.

[0083] Based on each confidence level, a confidence-utility matrix is ​​constructed.

[0084] Based on each first appearance feature vector and each second appearance feature vector, an appearance similarity utility matrix is ​​constructed.

[0085] A multi-objective utility matrix is ​​obtained by using the intersection-union utility matrix, the confidence utility matrix, and the appearance similarity utility matrix.

[0086] In this embodiment, a three-dimensional utility matrix based on a multi-objective optimization framework is constructed. Where m is the number of current trajectories predicted by the Kalman filter, n is the number of detected polyps in the current frame image, and 3 represents the number of objective functions to be optimized. Three objective functions are defined to characterize the intersection-over-union (IoU) ratio of each detected bounding box and each predicted bounding box, the confidence score of detected polyps, and the appearance similarity, respectively. The utility matrix U measures the intersection-over-union (IoU) ratio of the detected bounding boxes and each predicted bounding box. IoU Calculate the intersection-union ratio (IoU) between the i-th detected bounding box and the j-th predicted bounding box. Confidence-utility matrix U Conf : Directly extract the confidence score output by the polyp detection model: Appearance similarity utility matrix U ReID : Calculate each first appearance feature vector f i With each second appearance feature vector f j Cosine similarity: The last two rows of the utility matrix represent the "new trajectory identifier row" and the "false detection identifier row," which are used to constrain new trajectory creation and false detection filtering. By introducing "new trajectory" and "false detection" pseudo-nodes into the utility matrix, combined with the trajectory survival time threshold mechanism, it is possible to effectively distinguish between real polyp targets and transient interference (such as bubbles and specular reflections), thereby reducing the false alarm rate.

[0087] Further, in one embodiment, step S50 includes:

[0088] Step S501: Solve the multi-objective utility matrix using a multi-objective optimization algorithm to obtain the Pareto front solution set;

[0089] Step S502: For the Pareto front solution set, the optimal solution is obtained by a two-stage optimization mechanism of knee-guided selection and human-inspired decision-making.

[0090] In this embodiment, solving the multi-objective utility matrix requires combining multi-objective optimization algorithms with utility theory. Given a three-dimensional multi-objective utility matrix, the first two dimensions represent candidate solutions (solutions), and the third dimension represents the objective function (or utility index). By imposing certain constraints (such as each current trajectory matching at most one detected polyp) and balancing the trade-offs of multiple objectives, the Pareto front solution set is first obtained. The Pareto front solution set includes multiple solutions. Further, the optimal solution is obtained by using a two-stage optimization mechanism of knee-guided screening and human-inspired decision-making on the Pareto front solution set.

[0091] Further, in one embodiment, step S501 includes:

[0092] The objective function is established based on the intersection-union utility matrix, the confidence utility matrix, and the appearance similarity utility matrix, and constraints are set.

[0093] Multiple solutions are generated as the initial population;

[0094] Stratify and rank the population according to Pareto dominance;

[0095] Two-point crossover and bit-flip mutation are used to generate offspring solutions, which are then merged into the population, and a first preset number of optimal solutions are retained in the population.

[0096] If the number of iterations does not reach the preset number, return to the step of stratifying and sorting the population according to the Pareto dominance relationship;

[0097] If the number of iterations reaches the preset number, the Pareto front solution set will be output.

[0098] In this embodiment, based on the above multi-objective utility matrix, the following multi-objective optimization problem is solved, and the objective function is defined as a weighted aggregation of the three-dimensional utility matrix:

[0099] ,

[0100] The constraints are satisfied: ;

[0101] The non-dominated sorting genetic algorithm with elitist strategy (NSGA-II) is used to solve this multi-objective optimization problem. The specific steps include: (1) Initializing the population: can generate We use 10 random solutions as the initial population, with the minimum and maximum values ​​set to 30 and 60 respectively, where n is the number of targets to be detected. Each solution is... (2) Non-dominated sorting: Sort the population in a hierarchical manner according to the Pareto dominance relationship. (3) Crossover mutation: Generate offspring by using two-point crossover (probability 0.8) and bit flip mutation (probability 0.1). (4) Elite retention: Merge the parent and offspring populations and retain the top 50 optimal solutions. (5) Iterative optimization: Repeat the above (2)-(4) process until the iteration terminates after 10 generations, and output the Pareto front solution set.

[0102] Further, in one embodiment, step S502 includes:

[0103] The solution set P of the Pareto front f The formula is used to calculate each pair of solutions (p) in the Pareto front solution set. i ,p j The trade-off The formula is:

[0104] Where f1, f2, and f3 represent the crossover union ratio, confidence score, and appearance similarity objective function, respectively;

[0105] Retain satisfaction The solution set K of the knee points, where, ;

[0106] The optimal solution is determined from the knee point solution set according to the priority order of highest intersection-union ratio, highest appearance similarity, and highest confidence.

[0107] In this embodiment, a bimodal Pareto selection strategy is adopted: a hybrid decision-making paradigm combining knee-guided selection and human heuristic selection is proposed. The former identifies the naturally optimal solution by calculating the utility loss rate between solution sets, while the latter simulates the cognitive priority of human visual tracking to achieve stable decision-making in complex scenarios. Specifically, the knee-guided selection selects key solutions from the Pareto front solution set, aiming to retain solutions that are sensitive to the decision-maker's trade-offs and avoid redundancy caused by uniform distribution.

[0108] The optimal solution is determined from the knee solution set according to the priority order of highest intersection-union ratio, highest appearance similarity, and highest confidence level. This is human-inspired decision-making, and the following formula can be used to determine the optimal solution. The candidate solutions in the process are evaluated hierarchically: ;

[0109] The above steps yield the optimal solution to the data association problem in the polyp target tracking and detection task. , It is a decision matrix with dimensions (m+2,n), containing all decisions for n polyps. All values ​​in the matrix are either 0 or 1. If the value in the i-th row and j-th column is 1, it means that the j-th polyp out of n polyps matches the i-th trajectory out of m trajectories. The matrix index with a value of 1 in the first m rows indicates that a certain polyp (column index) belongs to the next frame of a certain trajectory (row index), that is, a successful match; The matrix index with a value of 1 in the (m+1)th row represents a polyp, which is marked as a provisional new trajectory and initialized with a counter. If the trajectory can be matched continuously in subsequent matching processes, a new trajectory is officially added; if the provisional new trajectory does not match continuously in subsequent processes, it is marked as a false detection and the trajectory is deleted.

[0110] Further, in one embodiment, the optimal solution includes whether each detected polyp matches each current trajectory, the trajectory being divided into formal trajectories and temporary trajectories, and step S60 includes:

[0111] For each detected polyp, if the detected polyp has a matching current trajectory, and the matching current trajectory is a formal trajectory, then the Kalman filter is updated using the bounding box of the detected polyp.

[0112] For each detected polyp, if there is no matching current trajectory for the detected polyp and the confidence of the detected polyp is greater than the preset confidence, a temporary trajectory is created for the detected polyp, and the Kalman filter is updated using the bounding box of the detected polyp.

[0113] For each current trajectory, if the current trajectory is a temporary trajectory, and if the temporary trajectory has a second consecutive preset number of detected polyps that match it, then the temporary trajectory is converted into a formal trajectory; if the temporary trajectory does not have a third consecutive preset number of detected polyps that match it, then the temporary trajectory is deleted.

[0114] For each current trajectory, if the current trajectory is a formal trajectory and there is no fourth preset number of detected polyps matching it, then the formal trajectory is terminated.

[0115] In this embodiment, the optimal solution includes whether each detected polyp matches each current trajectory. Trajectories are divided into formal trajectories and temporary trajectories. Based on whether each detected polyp matches each current trajectory, the trajectories are updated and managed according to different scenarios. For a temporary trajectory, if a detected polyp matches it for three consecutive frames, the temporary trajectory is converted into a formal trajectory. If no detected polyp matches it for three consecutive frames, the temporary trajectory is deleted, indicating a false detection, such as a misidentification by the polyp detection model due to reflection, where normal tissue in one frame is identified as a polyp, but it does not appear in subsequent frames. For a formal trajectory, if no detected polyp matches it for ten consecutive frames, the formal trajectory is terminated. The formal trajectory is resumed after a detected polyp matches it in a subsequent frame. This effectively addresses situations where there is occlusion or the target leaves the screen and then reappears. The preset number can be set according to specific needs. For example, if the endoscopist's operation is smooth, the fourth preset number can be set smaller; otherwise, it can be set larger.

[0116] In this embodiment, refer to Figure 3 , Figure 3 This is a schematic diagram of the model structure of an embodiment of the multi-target tracking and detection method of this application, as shown below. Figure 3 As shown, Figure 3 In the diagram, DET represents the polyp detection model. The input is a frame of image, and the output is the detection bounding box of the detected polyp in the image and its confidence score. ReID is the re-identification module, which calculates the first appearance feature vector of the detected polyp. MOEA is a multi-objective optimization algorithm. KO / HI are two Pareto front selection schemes: knee-guided and human heuristic selection, respectively. KF represents the Kalman filter. predict predicts the polyp position in the next frame for a certain trajectory, and update refers to updating the state of the Kalman filter after a successful match.

[0117] In this embodiment, refer to Figure 4 , Figure 4 This is a schematic diagram of the system flow of an embodiment of the multi-target tracking and detection method of this application, as shown below. Figure 4 As shown, the multi-object tracking and detection system first receives the image of the t-th frame from the polyp detection module, outputs the detected polyps, and extracts the image appearance feature vector through the re-identification module. The multi-objective optimization and association module constructs a multi-objective utility matrix of intersection-union ratio, confidence, and appearance similarity based on each detected bounding box, predicted bounding box, confidence, first appearance feature vector, and second appearance feature vector. The multi-objective optimization algorithm is used to solve the multi-objective utility matrix and outputs the solution set with respect to the Pareto front. The optimal solution is obtained by a two-stage optimization mechanism of knee-guided screening and human-inspired decision-making. Then, it enters the trajectory management module. If the detected polyp has a matching current trajectory, and the matching current trajectory is a formal trajectory, the bounding box of the detected polyp is used to update the Kalman filter. The Kalman filter is the trajectory prediction module.

[0118] Secondly, embodiments of this application also provide a multi-target tracking and detection device.

[0119] In one embodiment, reference is made to Figure 5 , Figure 5 This is a functional module diagram of an embodiment of the multi-target tracking and detection device of this application, as shown below. Figure 5 As shown, the multi-target tracking and detection device includes:

[0120] Extraction module 10 is used to decompose the colonoscopy video to be tested into multiple frames of images in chronological order;

[0121] The detection module 20 is used to sequentially detect each frame of the image using a polyp detection model to obtain multiple detection bounding boxes and confidence scores for detected polyps, and extract the first appearance feature vector of each detected polyp from the image based on each detection bounding box.

[0122] The prediction module 30 is used to predict multiple current trajectories based on multiple historical trajectories using a Kalman filter, and extract the second appearance feature vector of each current trajectory from the image based on the predicted bounding box of each current trajectory.

[0123] Module 40 is used to construct a multi-objective utility matrix of intersection-over-union ratio, confidence and appearance similarity based on each detected bounding box, predicted bounding box, confidence, first appearance feature vector and second appearance feature vector;

[0124] The solver module 50 is used to solve the multi-objective utility matrix using a multi-objective optimization algorithm to obtain the optimal solution;

[0125] The update module 60 is used to update and manage the trajectory based on the optimal solution.

[0126] Furthermore, in one embodiment, the multi-target tracking and detection device further includes an establishment module for:

[0127] Based on the detection bounding boxes of multiple polyps detected in historical frames by the polyp detection model, multiple historical trajectories are established.

[0128] Based on multiple historical trajectories, a Kalman filter is used to model the motion state of the detected polyps, resulting in a well-established Kalman filter.

[0129] Furthermore, in one embodiment, the construction module 40 is used for:

[0130] Based on each detected bounding box and each predicted bounding box, the intersection-union utility matrix is ​​constructed.

[0131] Based on each confidence level, a confidence-utility matrix is ​​constructed.

[0132] Based on each first appearance feature vector and each second appearance feature vector, an appearance similarity utility matrix is ​​constructed.

[0133] A multi-objective utility matrix is ​​obtained by using the intersection-union utility matrix, the confidence utility matrix, and the appearance similarity utility matrix.

[0134] Furthermore, in one embodiment, the solving module 50 includes:

[0135] The multi-objective solution unit is used to solve the multi-objective utility matrix using a multi-objective optimization algorithm to obtain the Pareto front solution set;

[0136] The screening unit is used to select the optimal solution from the Pareto front solution set using a two-stage optimization mechanism of knee-guided screening and human-inspired decision-making.

[0137] Furthermore, in one embodiment, the multi-objective solving unit is used for:

[0138] The objective function is established based on the intersection-union utility matrix, the confidence utility matrix, and the appearance similarity utility matrix, and constraints are set.

[0139] Multiple solutions are generated as the initial population;

[0140] Stratify and rank the population according to Pareto dominance;

[0141] Two-point crossover and bit-flip mutation are used to generate offspring solutions, which are then merged into the population, and a first preset number of optimal solutions are retained in the population.

[0142] If the number of iterations does not reach the preset number, return to the step of stratifying and sorting the population according to the Pareto dominance relationship;

[0143] If the number of iterations reaches the preset number, the Pareto front solution set will be output.

[0144] Furthermore, in one embodiment, the filtering unit is used for:

[0145] The solution set P of the Pareto front f The formula is used to calculate each pair of solutions (p) in the Pareto front solution set. i ,p j The trade-off The formula is:

[0146] Where f1, f2, and f3 represent the crossover union ratio, confidence score, and appearance similarity objective function, respectively;

[0147] Retain satisfaction The solution set K of the knee points, where, ;

[0148] The optimal solution is determined from the knee point solution set according to the priority order of highest intersection-union ratio, highest appearance similarity, and highest confidence.

[0149] Further, in one embodiment, the optimal solution includes whether each detected polyp matches each current trajectory, the trajectories being divided into formal trajectories and temporary trajectories, and the update module 60 is used for:

[0150] For each detected polyp, if the detected polyp has a matching current trajectory, and the matching current trajectory is a formal trajectory, then the Kalman filter is updated using the bounding box of the detected polyp.

[0151] For each detected polyp, if there is no matching current trajectory for the detected polyp and the confidence of the detected polyp is greater than the preset confidence, a temporary trajectory is created for the detected polyp, and the Kalman filter is updated using the bounding box of the detected polyp.

[0152] For each current trajectory, if the current trajectory is a temporary trajectory, and if the temporary trajectory has a second consecutive preset number of detected polyps that match it, then the temporary trajectory is converted into a formal trajectory; if the temporary trajectory does not have a third consecutive preset number of detected polyps that match it, then the temporary trajectory is deleted.

[0153] For each current trajectory, if the current trajectory is a formal trajectory and there is no fourth preset number of detected polyps matching it, then the formal trajectory is terminated.

[0154] The functions of each module in the multi-target tracking and detection device correspond to the steps in the multi-target tracking and detection method embodiment, and their functions and implementation processes will not be described in detail here.

[0155] Thirdly, embodiments of this application provide a multi-target tracking and detection device.

[0156] Reference Figure 6 , Figure 6 This is a schematic diagram of the hardware structure of the multi-target tracking and detection device involved in the embodiments of this application. In the embodiments of this application, the multi-target tracking and detection device may include a processor, a memory, a communication interface, and a communication bus.

[0157] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0158] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting internal components of the multi-target tracking and detection equipment, as well as interfaces used for interconnecting the multi-target tracking and detection equipment with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0159] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0160] The processor can be a general-purpose processor, which can call the multi-target tracking and detection program stored in memory and execute the multi-target tracking and detection method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the multi-target tracking and detection program is called can be referred to in the various embodiments of the multi-target tracking and detection method of this application, and will not be repeated here.

[0161] Those skilled in the art will understand that Figure 6 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0162] Fourthly, embodiments of this application also provide a readable storage medium.

[0163] This application stores a multi-target tracking and detection program on a readable storage medium, wherein when the multi-target tracking and detection program is executed by a processor, it implements the steps of the multi-target tracking and detection method described above.

[0164] The method implemented when the multi-target tracking and detection program is executed can be referred to in various embodiments of the multi-target tracking and detection method of this application, and will not be repeated here.

[0165] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0166] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0167] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0168] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0169] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0171] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A multi-target tracking and detection method, characterized in that, The multi-target tracking and detection method includes: The colonoscopy video to be tested was decomposed into multiple frames in chronological order; For each frame of the image, the polyp detection model is used to detect the image and obtain multiple detection bounding boxes and confidence scores for the detected polyps. Based on each detection bounding box, the first appearance feature vector of each detected polyp is extracted from the image. Multiple current trajectories are predicted based on multiple historical trajectories using a Kalman filter, and a second appearance feature vector for each current trajectory is extracted from the image based on the predicted bounding box of each current trajectory. Based on each detected bounding box, predicted bounding box, confidence score, first appearance feature vector, and second appearance feature vector, construct a multi-objective utility matrix of intersection-over-union ratio, confidence score, and appearance similarity. The optimal solution is obtained by solving the multi-objective utility matrix using a multi-objective optimization algorithm. The trajectory is updated and managed based on the optimal solution. The construction of a multi-objective utility matrix of intersection-over-union ratio, confidence level, appearance similarity, and other metrics based on each detected bounding box, predicted bounding box, confidence level, first appearance feature vector, and second appearance feature vector includes: Based on each detected bounding box and each predicted bounding box, the intersection-union utility matrix is ​​constructed. Based on each confidence level, a confidence-utility matrix is ​​constructed. Based on each first appearance feature vector and each second appearance feature vector, an appearance similarity utility matrix is ​​constructed. A multi-objective utility matrix is ​​obtained by using the intersection-union utility matrix, the confidence utility matrix, and the appearance similarity utility matrix. The method of using a multi-objective optimization algorithm to solve the multi-objective utility matrix to obtain the optimal solution includes: The Pareto front solution set is obtained by solving the multi-objective utility matrix using a multi-objective optimization algorithm. For the Pareto front solution set, a two-stage optimization mechanism of knee-guided selection and human-inspired decision-making is used to select the optimal solution; The optimal solution obtained by using a two-stage selection mechanism of knee-guided screening and human-inspired decision-making for the Pareto front solution set includes: The solution set P of the Pareto front f The formula is used to calculate each pair of solutions (p) in the Pareto front solution set. i ,p j The trade-off The formula is: Where f1, f2, and f3 represent the crossover union ratio, confidence score, and appearance similarity objective function, respectively; Retain satisfaction The solution set K of the knee points, where, ; The optimal solution is determined from the knee point solution set according to the priority order of highest intersection-union ratio, highest appearance similarity, and highest confidence.

2. The multi-target tracking and detection method as described in claim 1, characterized in that, Before using a Kalman filter to predict multiple current trajectories based on multiple historical trajectories, the following steps are included: Based on the detection bounding boxes of multiple polyps detected in historical frames by the polyp detection model, multiple historical trajectories are established. Based on multiple historical trajectories, a Kalman filter is used to model the motion state of the detected polyps, resulting in a well-established Kalman filter.

3. The multi-target tracking and detection method as described in claim 1, characterized in that, The method of using a multi-objective optimization algorithm to solve the multi-objective utility matrix yields the Pareto front solution set, which includes: The objective function is established based on the intersection-union utility matrix, the confidence utility matrix, and the appearance similarity utility matrix, and constraints are set. Multiple solutions are generated as the initial population; Stratify and rank the population according to Pareto dominance; Two-point crossover and bit-flip mutation are used to generate offspring solutions, which are then merged into the population, and a first preset number of optimal solutions are retained in the population. If the number of iterations does not reach the preset number, return to the step of stratifying and sorting the population according to the Pareto dominance relationship; If the number of iterations reaches the preset number, the Pareto front solution set will be output.

4. The multi-target tracking and detection method as described in claim 1, characterized in that, The optimal solution includes whether each detected polyp matches each current trajectory. The trajectories are divided into formal trajectories and temporary trajectories. The step of updating and managing the trajectories based on the optimal solution includes: For each detected polyp, if the detected polyp has a matching current trajectory, and the matching current trajectory is a formal trajectory, then the Kalman filter is updated using the bounding box of the detected polyp. For each detected polyp, if there is no matching current trajectory for the detected polyp and the confidence of the detected polyp is greater than the preset confidence, a temporary trajectory is created for the detected polyp, and the Kalman filter is updated using the bounding box of the detected polyp. For each current trajectory, if the current trajectory is a temporary trajectory, and if the temporary trajectory has a second consecutive preset number of detected polyps that match it, then the temporary trajectory is converted into a formal trajectory; if the temporary trajectory does not have a third consecutive preset number of detected polyps that match it, then the temporary trajectory is deleted. For each current trajectory, if the current trajectory is a formal trajectory and there is no fourth preset number of detected polyps matching it, then the formal trajectory is terminated.

5. A multi-target tracking and detection device, characterized in that, The multi-target tracking and detection device includes: The extraction module is used to decompose the colonoscopy video to be tested into multiple frames of images in chronological order; The detection module is used to sequentially detect polyps in each frame of the image using a polyp detection model, obtain multiple detection bounding boxes and confidence scores for the detected polyps, and extract the first appearance feature vector of each detected polyp from the image based on each detection bounding box. The prediction module is used to predict multiple current trajectories based on multiple historical trajectories using a Kalman filter, and extract the second appearance feature vector of each current trajectory from the image based on the predicted bounding box of each current trajectory. The module is used to construct a multi-objective utility matrix of intersection-over-union ratio, confidence, and appearance similarity based on each detected bounding box, predicted bounding box, confidence score, first appearance feature vector, and second appearance feature vector. The solution module is used to solve the multi-objective utility matrix using a multi-objective optimization algorithm to obtain the optimal solution; The update module is used to update and manage the trajectory based on the optimal solution; Modules are also used for: Based on each detected bounding box and each predicted bounding box, the intersection-union utility matrix is ​​constructed. Based on each confidence level, a confidence-utility matrix is ​​constructed. Based on each first appearance feature vector and each second appearance feature vector, an appearance similarity utility matrix is ​​constructed. A multi-objective utility matrix is ​​obtained by using the intersection-union utility matrix, the confidence utility matrix, and the appearance similarity utility matrix. The solver module is also used for: The Pareto front solution set is obtained by solving the multi-objective utility matrix using a multi-objective optimization algorithm. For the Pareto front solution set, a two-stage optimization mechanism of knee-guided selection and human-inspired decision-making is used to select the optimal solution; The optimal solution obtained from the Pareto front solution set is selected using a two-stage optimization mechanism combining knee-guided selection and human-inspired decision-making. The solution set P of the Pareto front f The formula is used to calculate each pair of solutions (p) in the Pareto front solution set. i ,p j The trade-off The formula is: Where f1, f2, and f3 represent the crossover union ratio, confidence score, and appearance similarity objective function, respectively; Retain satisfaction The solution set K of the knee points, where, ; The optimal solution is determined from the knee point solution set according to the priority order of highest intersection-union ratio, highest appearance similarity, and highest confidence.

6. A multi-target tracking and detection device, characterized in that, The multi-target tracking and detection device includes a processor, a memory, and a multi-target tracking and detection program stored in the memory and executable by the processor, wherein when the multi-target tracking and detection program is executed by the processor, it implements the steps of the multi-target tracking and detection method as described in any one of claims 1 to 4.

7. A readable storage medium, characterized in that, The readable storage medium stores a multi-target tracking and detection program, wherein when the multi-target tracking and detection program is executed by a processor, it implements the steps of the multi-target tracking and detection method as described in any one of claims 1 to 4.

Images