Multi-target tracking method and device based on diffusion model

Abstract

The application relates to a multi-target tracking method and device based on a diffusion model, wherein the method comprises the following steps: acquiring ROI features of input front and rear image frames through a feature extraction module; detecting targets in the front and rear image frames by using a diffusion detection head structure based on a DDIM process, and simultaneously predicting a track of a previous image frame by using a track prediction module based on a decoupled diffusion model; effectively associating the predicted track and the obtained detection result by using a distance calculation algorithm of a hybrid IoU and ReID, so that a final target tracking result is obtained. Thus, the problems that the existing multi-target tracking technology is difficult to simultaneously cope with linear and nonlinear motion targets, ID switching is prone to occurring in a dense scene, and the stability and robustness of tracking are poor are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, in particular to a multi-target tracking method and device based on a diffusion model. BACKGROUND

[0002] Multi-target tracking technology has a wide range of applications in video surveillance, autonomous driving, intelligent security, sports events, etc. The core goal is to accurately and continuously track multiple moving targets from video streams, thereby providing reliable target trajectory information for downstream tasks such as behavior analysis, decision support, etc. Traditional multi-target tracking methods mainly include two stages of target detection and data association: first, a target detection model is used to identify targets in video frames, and then a data association algorithm is used to associate the same targets in consecutive frames to form a moving trajectory. However, the complexity of real-world scenarios poses many challenges to existing multi-target tracking technology.

[0003] In real-world scenarios, targets often have complex motion patterns, such as linear and non-linear motion, frequent occlusion, and appearance similarity between targets. In addition, different camera devices and environmental conditions (such as lighting, weather, etc.) can significantly affect the performance of target detection and tracking. Therefore, an efficient multi-target tracking system needs to have strong robustness, which can effectively handle occlusion and target overlap in dense target scenarios while maintaining high-precision tracking of linear and non-linear motion targets.

[0004] Although deep learning-based multi-target tracking technology has made significant progress, existing methods are difficult to balance when dealing with linear and non-linear motion targets, and the tracking effect in target dense scenarios is still not ideal. How to implement an efficient and robust multi-target tracking system in complex scenarios is still a technical problem to be solved.

[0005] Target tracking is an important research direction in computer vision. In recent years, existing technologies can introduce diffusion models into the field of multi-target tracking, considering target detection and association as a consistent denoising diffusion process from pairs of noisy boxes to target association, successfully achieving tracking of targets. However, it only considers the position transformation of the target and does not consider the appearance of the target, and does not consider the non-linear motion scene, so its effect is poor in dense scenes and when dealing with non-linear motion scenes. In addition, DiffMOT is another multi-target tracking method based on diffusion model, which introduces a diffusion model when predicting trajectories, improving the tracking effect of non-linear motion targets. However, DiffMOT has poor effect when tracking linear motion targets, and is prone to ID switching problems in dense scenes.

[0006] In summary, the existing multi-target tracking technology is difficult to cope with linear and nonlinear motion targets at the same time, and ID switching problems easily occur in dense scenes, and the stability and robustness of tracking are poor, which need to be solved urgently. SUMMARY

[0007] The present application provides a multi-target tracking method and device based on a diffusion model to solve the problems that the existing multi-target tracking technology is difficult to cope with linear and nonlinear motion targets at the same time, and ID switching problems easily occur in dense scenes, and the stability and robustness of tracking are poor.

[0008] The first aspect embodiment of the present application provides a multi-target tracking method based on a diffusion model, comprising the following steps: obtaining ROI features of front and rear two frames of to-be-detected images through a preset feature extraction module, and inputting the ROI features into a preset diffusion detection head module, and performing target detection operation on the front and rear two frames of to-be-detected images to generate target detection results of each target in the front and rear two frames of to-be-detected images, wherein the front and rear two frames of to-be-detected images include a previous frame of to-be-detected image and a current frame of to-be-detected image; calculating target displacement of each target in the front and rear two frames of to-be-detected images according to the target detection results, and inputting the target displacement into a preset decoupling diffusion trajectory prediction module to predict target trajectories of each target in the previous frame of to-be-detected image to obtain a trajectory prediction result; performing matching and association operation on the target detection results and the trajectory prediction result based on a preset mixed IoU and ReID distance calculation strategy to generate a final target tracking result.

[0009] Optionally, in an embodiment of the present application, the ROI features of the two frames of to-be-detected images are obtained by a preset feature extraction module, and the ROI features are input into a preset diffusion detection head module, and target detection is performed on the two frames of to-be-detected images to generate a target detection result of each target in the two frames of to-be-detected images, including: inputting the two frames of to-be-detected images into the feature extraction module to output ROI features of a target size corresponding to the two frames of to-be-detected images; generating a plurality of random boxes in the two frames of to-be-detected images, and determining the plurality of random boxes as initial detection boxes corresponding to the two frames of to-be-detected images; based on a preset multi-scale attention module, a dynamic convolution module and a data association head module, the diffusion detection head module is constructed, and the ROI features and the initial detection boxes are input into the diffusion detection head module to generate a predicted box corresponding to the two frames of to-be-detected images by the multi-scale attention module and the dynamic convolution module, and to calculate noise between the plurality of random boxes and the predicted box; the predicted box and the noise are input into the data association head module to obtain a detection box of the current iteration, and the initial detection box is updated using the detection box of the current iteration; the ROI features and the updated initial detection box are re-input into the diffusion detection head module to iteratively perform target detection until a preset time step iteration termination condition is met to generate the target detection result.

[0010] Optionally, in an embodiment of the present application, the target displacement is input into a preset decoupled diffusion trajectory prediction module to predict a target trajectory of each target in the previous frame of to-be-detected images to obtain a trajectory prediction result, including: constructing the decoupled diffusion trajectory prediction module by a plurality of preset multi-scale attention modules and a motion fusion module; adding Gaussian noise to the target displacement, and inputting the target displacement after adding Gaussian noise into the decoupled diffusion trajectory prediction module to perform post-diffusion on the target displacement after adding Gaussian noise by the multi-scale attention module and the motion fusion module to generate a target predicted displacement of the current frame of to-be-detected images; based on the target predicted displacement and the target detection result of the previous frame of to-be-detected images, the trajectory prediction result of the current frame of to-be-detected images is calculated.

[0011] Optionally, in an embodiment of the present application, the matching and associating operation of the target detection result and the trajectory prediction result based on the preset mixed IoU and ReID distance calculation strategy to generate the final target tracking result comprises: calculating the IoU distance and the ReID distance between the target detection result and the trajectory prediction result, and calculating the distance weight corresponding to each coordinate distance between the IoU distance and the ReID distance; calculating the final mixed distance corresponding to the IoU distance and the ReID distance according to the distance weight and a preset weight factor; based on the final mixed distance and a preset linear distribution algorithm, calculating the target matching pair corresponding to the target detection result and the trajectory prediction result, and updating the target trajectory through the target matching pair to obtain the final target tracking result.

[0012] The second aspect embodiment of the present application provides a multi-target tracking device based on a diffusion model, comprising: a target detection module, configured to obtain ROI features of two frames of to-be-detected images before and after through a preset feature extraction module, and input the ROI features into a preset diffusion detection head module, and perform target detection operation on the two frames of to-be-detected images before and after to generate target detection results of each target in the two frames of to-be-detected images before and after, wherein the two frames of to-be-detected images include a previous frame of to-be-detected image and a current frame of to-be-detected image; a trajectory prediction module, configured to calculate target displacement of each target in the two frames of to-be-detected images before and after according to the target detection result, and input the target displacement into a preset decoupling diffusion trajectory prediction module to predict target trajectory of each target in the previous frame of to-be-detected image to obtain a trajectory prediction result; a matching and associating module, configured to perform matching and associating operation on the target detection result and the trajectory prediction result based on a preset mixed IoU and ReID distance calculation strategy to generate a final target tracking result.

[0013] Optionally, in an embodiment of the present application, the target detection module comprises: an input unit configured to input the two frames of to-be-detected images into the feature extraction module to output ROI features of the two frames of to-be-detected images; a generation unit configured to generate a plurality of random boxes in the two frames of to-be-detected images, and determine the plurality of random boxes as initial detection boxes corresponding to the two frames of to-be-detected images; a first calculation unit configured to construct the diffusion detection head module based on a preset multi-scale attention module, a dynamic convolution module and a data association head module, and input the ROI features and the initial detection boxes into the diffusion detection head module to generate prediction boxes corresponding to the two frames of to-be-detected images through the multi-scale attention module and the dynamic convolution module, and calculate noises between the plurality of random boxes and the prediction boxes; an update unit configured to input the prediction boxes and the noises into the data association head module to obtain detection boxes of a current iteration, and update the initial detection boxes using the detection boxes of the current iteration; and an iteration unit configured to re-input the ROI features and the updated initial detection boxes into the diffusion detection head module to iteratively perform target detection operations until a preset time step iteration termination condition is met, so as to generate the target detection result.

[0014] Optionally, in an embodiment of the present application, the trajectory prediction module comprises: a construction unit configured to construct the decoupled diffusion trajectory prediction module through a plurality of preset multi-scale attention modules and a motion fusion module; a noise adding unit configured to add Gaussian noise to the target displacement, and input the target displacement after adding the Gaussian noise into the decoupled diffusion trajectory prediction module to perform post-diffusion on the target displacement after adding the Gaussian noise through the multi-scale attention module and the motion fusion module, so as to generate a target predicted displacement of the current frame of to-be-detected image; and a second calculation unit configured to calculate a trajectory prediction result of the current frame of to-be-detected image based on the target predicted displacement and a target detection result of the previous frame of to-be-detected image.

[0015] Optionally, in an embodiment of the present application, the matching association module comprises: a third calculation unit configured to calculate an IoU distance and a ReID distance between the target detection result and the trajectory prediction result, and calculate a distance weight corresponding to each coordinate distance between the IoU distance and the ReID distance; a fourth calculation unit configured to calculate a final hybrid distance corresponding to the IoU distance and the ReID distance according to the distance weight and a preset weight factor; and a linear distribution unit configured to calculate a target matching pair corresponding to the target detection result and the trajectory prediction result based on the final hybrid distance and a preset linear distribution algorithm, and update a target trajectory through the target matching pair to obtain the final target tracking result.

[0016] The third aspect of the embodiments of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the program to implement the multi-target tracking method based on the diffusion model as described in the above embodiments.

[0017] The fourth aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the multi-target tracking method based on the diffusion model as described above.

[0018] The fifth aspect of the embodiments of the present application provides a computer program product, comprising a computer program, and the computer program is executed to implement the multi-target tracking method based on the diffusion model as described above.

[0019] Therefore, the embodiments of the present application have the following beneficial effects:

[0020] The embodiments of the present application can obtain the ROI features of the front and rear two frames of to-be-detected images through the preset feature extraction module, and input the ROI features into the preset diffusion detection head module, and perform target detection operation on the front and rear two frames of to-be-detected images to generate the target detection result of each target in the front and rear two frames of to-be-detected images, wherein the front and rear two frames of to-be-detected images include a previous frame of to-be-detected image and a current frame of to-be-detected image; calculate the target displacement of each target in the front and rear two frames of to-be-detected images according to the target detection result, and input the target displacement into the preset decoupling diffusion trajectory prediction module to predict the target trajectory of each target in the previous frame of to-be-detected image to obtain the trajectory prediction result; and perform matching and association operation on the target detection result and the trajectory prediction result based on the preset mixed IoU and ReID distance calculation strategy to generate the final target tracking result. The present application realizes accurate tracking of multiple moving targets in complex motion state by combining the diffusion detection module and the diffusion trajectory prediction module, which can be applied to multiple fields such as security, rescue, sports events and automatic driving. Therefore, the existing multi-target tracking technology is difficult to cope with linear and nonlinear motion targets at the same time, and ID switching problem is easy to occur in dense scenes, and the stability and robustness of tracking are poor.

[0021] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0023] Figure 1A flow chart of a multi-target tracking method based on a diffusion model according to an embodiment of the present application is provided.

[0024] Figure 2 An execution logic diagram of a multi-target tracking method based on a diffusion model according to an embodiment of the present application is provided.

[0025] Figure 3 A multi-scale attention module network structure diagram according to an embodiment of the present application is provided.

[0026] Figure 4 A motion fusion module network structure diagram according to an embodiment of the present application is provided.

[0027] Figure 5 A multi-target tracking effect diagram according to an embodiment of the present application is provided.

[0028] Figure 6 An example diagram of a multi-target tracking device based on a diffusion model according to an embodiment of the present application is provided.

[0029] Figure 7 A structure diagram of an electronic device according to an embodiment of the present application is provided.

[0030] Wherein, 10 - a multi-target tracking device based on a diffusion model; 100 - a target detection module, 200 - a trajectory prediction module, 300 - a matching correlation module; 701 - a memory, 702 - a processor, 703 - a communication interface. DETAILED DESCRIPTION

[0031] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0032] A method and device for multi-target tracking based on a diffusion model are described below with reference to the accompanying drawings. To address the problems mentioned in the background, the present application provides a method for multi-target tracking based on a diffusion model. In the method, the ROI features of two consecutive frames of images to be detected are obtained by a pre-set feature extraction module, and the ROI features are input into a pre-set diffusion detection head module. Target detection is performed on the two consecutive frames of images to be detected to generate target detection results for each target in the two consecutive frames of images to be detected. The target displacement of each target in the two consecutive frames of images to be detected is calculated based on the target detection results, and the target displacement is input into a pre-set decoupled diffusion trajectory prediction module to predict the target trajectory of each target in the previous frame of image to be detected to obtain a trajectory prediction result. Based on a pre-set mixed IoU and ReID distance calculation strategy, the target detection results and the trajectory prediction results are matched and associated to generate a final target tracking result. By combining the diffusion detection module and the diffusion trajectory prediction module, the present application realizes accurate tracking of multiple moving targets in complex motion states and can be applied to multiple fields such as security, rescue, sports events, and autonomous driving. Thus, the existing multi-target tracking technology is difficult to cope with linear and nonlinear motion targets at the same time, and ID switching is prone to occur in dense scenes, and the stability and robustness of tracking are poor.

[0033] Specifically, Figure 1 A flowchart of a method for multi-target tracking based on a diffusion model is provided in the present application.

[0034] As Figure 1 shown, the method for multi-target tracking based on a diffusion model includes the following steps:

[0035] In step S101, the ROI features of two consecutive frames of images to be detected are obtained by a pre-set feature extraction module, and the ROI features are input into a pre-set diffusion detection head module. Target detection is performed on the two consecutive frames of images to be detected to generate target detection results for each target in the two consecutive frames of images to be detected. The two consecutive frames of images to be detected include a previous frame of image to be detected and a current frame of image to be detected.

[0036] The embodiment of the present application can first obtain the ROI features of the input two consecutive frames of images to be detected by a feature extraction module, wherein the feature extraction module is a backbone network in YOLOX. Secondly, the embodiment of the present application can input the ROI features into a diffusion detection head module to detect each target in the two consecutive frames of images to be detected.

[0037] Optionally, in an embodiment of the present application, the ROI features of the front and rear two frames of to-be-detected images are obtained by a preset feature extraction module, and the ROI features are input into a preset diffusion detection head module, and target detection operations are performed on the front and rear two frames of to-be-detected images to generate target detection results of each target in the front and rear two frames of to-be-detected images, including: inputting the front and rear two frames of to-be-detected images into the feature extraction module to output the ROI features corresponding to the target size of the front and rear two frames of to-be-detected images; generating a plurality of random boxes in the front and rear two frames of to-be-detected images, and determining the plurality of random boxes as the initial detection boxes corresponding to the front and rear two frames of to-be-detected images; based on a preset multi-scale attention module, a dynamic convolution module and a data association head module, a diffusion detection head module is constructed, and the ROI features and the initial detection boxes are input into the diffusion detection head module to generate the prediction boxes corresponding to the front and rear two frames of to-be-detected images through the multi-scale attention module and the dynamic convolution module, and to calculate the noise between the plurality of random boxes and the prediction boxes; the prediction boxes and the noise are input into the data association head module to obtain the detection boxes of the current iteration, and the initial detection boxes are updated using the detection boxes of the current iteration; the ROI features and the updated initial detection boxes are re-input into the diffusion detection head module to iteratively perform target detection operations until a preset time step iteration termination condition is met to generate the target detection results.

[0038] It should be noted that the embodiment of the present application can input the previous frame of to-be-detected image F T-1 and the current frame of to-be-detected image F T into the feature extraction module to obtain the ROI features R T-1 and R T with a scale of 4000x320x7x7, and record them as P T =(R T ,R T-1 ).

[0039] Secondly, the embodiment of the present application can input the ROI features into the diffusion detection head module to detect each target in the front and rear two frames of to-be-detected images, wherein the diffusion detection head module is composed of a multi-scale attention module, a dynamic convolution module and a data association head module, as shown in Figure 2 .

[0040] Specifically, the steps of the embodiment of the present application for target detection by the diffusion detection head module are as follows:

[0041] Step 1: set N=500, generate N random boxes B T-1 and B T on the previous frame of to-be-detected image F T-1 and the current frame of to-be-detected image F T , and take them as initial detection boxes, recorded as Z T =(BT ,B T-1 );

[0042] Step 2: input the ROI feature P T and the initial detection frame Z T into the diffusion detection head module, and the detection process can be regarded as a denoising diffusion process from the noise frame to the target frame;

[0043] The embodiments of the present application can obtain the prediction frame B pred through the multi-scale attention module and the dynamic convolution module; given the data distribution X0~q(X0), and the forward noise disturbance process at time t is defined as q(X T |X T-1 ), and according to the variance table β1,...β T , gradually add Gaussian noise, as shown in the following formula:

[0044]

[0045] Given X0, the embodiments of the present application can obtain X T by sampling Gaussian vector ε~N(0, I) and applying transformation, which is specifically represented as:

[0046]

[0047] Wherein,

[0048] Then, the embodiments of the present application can substitute B pred and Z T into X T and X0 to calculate the noise introduced by the process from the random frame to the prediction frame. The neural network f θ (X T , T) is used to train X0 from X T , and the loss is as follows:

[0049]

[0050] Step 3: input the noise Noise T and the prediction frame B pred obtained in step 2 into the data association head module to obtain the detection frame B temp of the current iteration through denoising diffusion, and the process is represented as:

[0051]

[0052] Wherein, and are B pred and Noise T, s T may be expressed as:

[0053]

[0054] Therefore, the X T-1 generated in the above calculation is the detection box B temp generated in the current iteration.

[0055] Step 4: using the generated detection box B temp in step 3, update Z T in step 2, and input the updated Z T and ROI feature P T into the diffusion detection head module again for a new round of iteration until the time step T = 0, at which time the B temp obtained is the final detection box B det .

[0056] In step S102, the target displacement of each target in the two frames of to-be-detected images before and after the target detection result is calculated, and the target displacement is input into a preset decoupling diffusion trajectory prediction module to predict the target trajectory of each target in the previous frame of to-be-detected image, so as to obtain the trajectory prediction result.

[0057] Further, the embodiment of the present application also needs to calculate the target displacement in the two frames of to-be-detected images before and after, and input it into the decoupling diffusion trajectory prediction module to predict the trajectory of the target in the previous frame of to-be-detected image.

[0058] Therefore, the embodiment of the present application realizes accurate tracking of multiple targets in a complex motion state by innovatively combining the diffusion detection module and the diffusion trajectory prediction module.

[0059] Optionally, in an embodiment of the present application, inputting the target displacement into the preset decoupling diffusion trajectory prediction module to predict the target trajectory of each target in the previous frame of to-be-detected image to obtain the trajectory prediction result, comprising: constructing the decoupling diffusion trajectory prediction module through a plurality of preset multi-scale attention modules and a motion fusion module; adding Gaussian noise to the target displacement, and inputting the target displacement after adding Gaussian noise into the decoupling diffusion trajectory prediction module to generate the target prediction displacement of the current frame of to-be-detected image through the multi-scale attention module and the motion fusion module; based on the target prediction displacement and the target detection result of the previous frame of to-be-detected image, calculating the trajectory prediction result of the current frame of to-be-detected image.

[0060] Specifically, the specific steps of the embodiment of the present application for target trajectory prediction through the decoupling diffusion trajectory prediction module are as follows:

[0061] Step 1: according to the target detection results D of the previous and next frames of the to-be-detected image T and D T-1 , calculate the target displacement M T , the calculation process is as shown in the following formula:

[0062] M T = D T - D T-1 = (ΔX T , ΔY T , ΔW T , ΔH T )

[0063] wherein X and Y are the horizontal and vertical coordinates of the center point of the target, and W and H are the width and height of the detection frame;

[0064] Step 2: add Gaussian noise to the target displacement M T obtained in step 1, input a decoupling diffusion trajectory prediction module composed of a multi-scale attention module, a dynamic convolution module and a data association head module, and diffuse to generate the target predicted displacement M pred of the current frame of the to-be-detected image through a multi-scale attention module as shown in Figure 3 and a motion fusion module as shown in Figure 4 ;

[0065] Step 3: use the target predicted displacement M pred of step 2 and the detection result D T-1 of the previous frame of the to-be-detected image to obtain the predicted trajectory D pred of the current frame of the to-be-detected image, and the process is represented as:

[0066] D pred = D T-1 + M pred

[0067] Therefore, the embodiment of the present application predicts the trajectory of each target in the previous frame of the to-be-detected image through the decoupling diffusion trajectory prediction module, thereby providing guidance and basis for the execution of the subsequent matching and association operation.

[0068] In step S103, the target detection result and the trajectory prediction result are matched and associated based on the preset mixed IoU and ReID distance calculation strategy, to generate the final target tracking result.

[0069] Further, the embodiment of the present application also needs to match and associate the target detection result and the predicted trajectory through the mixed IoU and ReID distance calculation algorithm to obtain the final target tracking result, and iteratively update, thereby continuously tracking the subsequent images.

[0070] Therefore, the embodiment of the present application effectively reduces target occlusion and ID switching during interaction and improves the stability of target tracking by establishing a hybrid IoU and ReID distance calculation strategy.

[0071] Optionally, in an embodiment of the present application, based on the preset hybrid IoU and ReID distance calculation strategy, the target detection result and the trajectory prediction result are matched and associated to generate the final target tracking result, including: calculating the IoU distance and the ReID distance between the target detection result and the trajectory prediction result, and calculating the distance weight corresponding to each coordinate distance between the IoU distance and the ReID distance; calculating the final hybrid distance corresponding to the IoU distance and the ReID distance according to the distance weight and the preset weight factor; based on the final hybrid distance and the preset linear distribution algorithm, calculating the target matching pair corresponding to the target detection result and the trajectory prediction result, and updating the target trajectory through the target matching pair to obtain the final target tracking result.

[0072] Specifically, the embodiment of the present application performs data association on the detection result and the predicted trajectory of the current frame to be detected image according to the hybrid IoU and ReID distance calculation strategy to obtain the final target tracking result as follows:

[0073] Step 1: calculate the IoU distance d I and the ReID distance d E between the above obtained detection box B det and the predicted trajectory D pred , wherein the embodiment of the present application can use the SBS-S50 network to extract target feature embedding;

[0074] Step 2: calculate the distance weight corresponding to each coordinate of d I and d E in step 1, and the value of each coordinate can be represented as:

[0075] D I = d I [i][j]D E = d E [i][j]

[0076] Set the distance threshold of IoU and ReID as τ I and τ E , and the proportional distance of each coordinate is calculated as follows:

[0077]

[0078] Wherein, the weight calculation strategy is F~1 / e X , and the embodiment of the present application can calculate I Δ and E ΔCompute the distance weight W of IoU and ReID I and W E , in D I <τ I & D E <τ E , the hybrid distance is calculated as follows:

[0079] d mix [i][j] = W I · D I + W E · D E

[0080] In other cases, a weight factor η is added to the distance calculation of IoU, which is expressed as:

[0081] d mix [i][j] = W I · D I · η + W E · D E

[0082] After one iteration, the final hybrid distance d mix is obtained.

[0083] Step 3: Calculate the hybrid distance d mix of step 2 using the linear assignment algorithm to obtain the target matching pair m τ of the predicted trajectory D pred and the detection result B det , update the target trajectory according to the matching pair m τ to obtain the target tracking result of the current frame.

[0084] Subsequently, the subsequent frame target tracking of the present embodiment can be performed, at this time, the current frame F T becomes F T-1 , by iterating the above target detection, trajectory prediction and other operations to obtain the tracking result of the subsequent frame, until all frames are completed tracking, and the effect of the multi-target tracking of the present embodiment is as shown in Figure 5 .

[0085] Therefore, the detection and trajectory prediction results in the present embodiment are generated using the diffusion model, taking into account the linear and nonlinear motion of the target, and considering the occlusion of the dense scene, having better tracking effect and stronger robustness in real scene; in addition, the present embodiment can also improve the sampling speed of the diffusion process through parallel sampling technology, thereby reducing the inference time, and being suitable for application scenarios with high real-time requirements.

[0086] To sum up, the embodiment of the application can first obtain the ROI features of the input front and rear two frames of images through the feature extraction module; secondly, the target in the front and rear two frames is detected using the diffusion detection head structure based on the DDIM process, and the trajectory of the previous frame of image is predicted using the trajectory prediction module based on the decoupled diffusion model; then, the predicted trajectory and the obtained detection result are effectively associated using a distance calculation algorithm of mixed IoU and ReID, so as to obtain the final target tracking result. Therefore, the embodiment of the application can be well applied to multiple fields such as security, rescue, sports events and automatic driving, and can effectively and accurately track the moving target under complex motion state.

[0087] The application can also construct a corresponding multi-target tracking system based on a diffusion model according to the multi-target tracking method based on the diffusion model. The multi-target tracking system based on the diffusion model of the application is briefly described and introduced as follows.

[0088] The multi-target tracking system based on the diffusion model of the application mainly includes a feature extraction unit, a target detection unit, a motion trajectory prediction unit and a target tracking unit.

[0089] The feature extraction unit is used to extract the ROI features of the front and rear frames of input images to be detected;

[0090] The target detection unit is used to diffuse to generate the target detection result of the front and rear frames of input images to be detected;

[0091] The motion trajectory prediction unit is used to diffuse to generate the motion trajectory of the target of the current frame of image to be detected;

[0092] The target tracking unit is used to data associate the detection result and the predicted trajectory of the current frame of image to be detected to obtain the final target tracking result.

[0093] According to the multi-target tracking method based on the diffusion model provided in the embodiments of the present application, the ROI features of the front and rear two frames of to-be-detected images are obtained through a preset feature extraction module, and the ROI features are input into a preset diffusion detection head module, and target detection operations are performed on the front and rear two frames of to-be-detected images to generate target detection results of each target in the front and rear two frames of to-be-detected images, wherein the front and rear two frames of to-be-detected images include a previous frame of to-be-detected image and a current frame of to-be-detected image; the target displacement of each target in the front and rear two frames of to-be-detected images is calculated according to the target detection results, and the target displacement is input into a preset decoupling diffusion trajectory prediction module to predict the target trajectory of each target in the previous frame of to-be-detected image to obtain a trajectory prediction result; and the target detection results and the trajectory prediction results are matched and associated based on a preset mixed IoU and ReID distance calculation strategy to generate a final target tracking result. By combining the diffusion detection module and the diffusion trajectory prediction module, the present application realizes accurate tracking of multiple moving targets in a complex motion state, and can be applied to multiple fields such as security, rescue, sports events and automatic driving.

[0094] Secondly, the multi-target tracking device based on the diffusion model according to the embodiments of the present application is described with reference to the accompanying drawings.

[0095] Figure 6 is a block schematic diagram of the multi-target tracking device based on the diffusion model of the embodiments of the present application.

[0096] As shown in Figure 6 , the multi-target tracking device based on the diffusion model 10 includes a target detection module 100, a trajectory prediction module 200 and a matching association module 300.

[0097] The target detection module 100 is configured to obtain the ROI features of the front and rear two frames of to-be-detected images through a preset feature extraction module, and input the ROI features into a preset diffusion detection head module, and perform target detection operations on the front and rear two frames of to-be-detected images to generate target detection results of each target in the front and rear two frames of to-be-detected images, wherein the front and rear two frames of to-be-detected images include a previous frame of to-be-detected image and a current frame of to-be-detected image.

[0098] The trajectory prediction module 200 is configured to calculate the target displacement of each target in the front and rear two frames of to-be-detected images according to the target detection results, and input the target displacement into a preset decoupling diffusion trajectory prediction module to predict the target trajectory of each target in the previous frame of to-be-detected image to obtain a trajectory prediction result.

[0099] The matching association module 300 is configured to perform matching and association operations on the target detection results and the trajectory prediction results based on a preset mixed IoU and ReID distance calculation strategy to generate a final target tracking result.

[0100] Optionally, in an embodiment of the present application, the target detection module 100 comprises an input unit, a generation unit, a first calculation unit, an update unit and an iteration unit.

[0101] The input unit is configured to input the front and rear two frames of to-be-detected images into the feature extraction module to output the ROI features of the target size corresponding to the front and rear two frames of to-be-detected images.

[0102] The generation unit is configured to generate a plurality of random boxes in the front and rear two frames of to-be-detected images, and determine the plurality of random boxes as the initial detection boxes corresponding to the front and rear two frames of to-be-detected images.

[0103] The first calculation unit is configured to construct a diffusion detection head module based on a preset multi-scale attention module, a dynamic convolution module and a data association head module, and input the ROI features and the initial detection boxes into the diffusion detection head module to generate the predicted boxes corresponding to the front and rear two frames of to-be-detected images through the multi-scale attention module and the dynamic convolution module, and calculate the noise between the plurality of random boxes and the predicted boxes.

[0104] The update unit is configured to input the predicted boxes and the noise into the data association head module to obtain the detection boxes of the current iteration, and update the initial detection boxes by using the detection boxes of the current iteration.

[0105] The iteration unit is configured to re-input the ROI features and the updated initial detection boxes into the diffusion detection head module to iteratively perform the target detection operation until a preset time step iteration termination condition is met, so as to generate the target detection result.

[0106] Optionally, in an embodiment of the present application, the trajectory prediction module 200 comprises a construction unit, a noise adding unit and a second calculation unit.

[0107] The construction unit is configured to construct a decoupled diffusion trajectory prediction module through a plurality of preset multi-scale attention modules and a motion fusion module.

[0108] The noise adding unit is configured to add Gaussian noise to the target displacement, and input the target displacement with the added Gaussian noise into the decoupled diffusion trajectory prediction module to perform post-diffusion on the target displacement with the added Gaussian noise through the multi-scale attention module and the motion fusion module, so as to generate the target predicted displacement of the current frame of to-be-detected image.

[0109] The second calculation unit is configured to calculate the trajectory prediction result of the current frame of to-be-detected image based on the target predicted displacement and the target detection result of the previous frame of to-be-detected image.

[0110] Optionally, in an embodiment of the present application, the matching association module 300 comprises a third calculation unit, a fourth calculation unit and a linear distribution unit.

[0111] The third calculation unit is configured to calculate an IoU distance and a ReID distance between the target detection result and the trajectory prediction result, and calculate a distance weight corresponding to each coordinate distance between the IoU distance and the ReID distance.

[0112] The fourth calculation unit is configured to calculate a final mixed distance corresponding to the IoU distance and the ReID distance according to the distance weight and a preset weight factor.

[0113] The linear distribution unit is configured to calculate a target matching pair corresponding to the target detection result and the trajectory prediction result based on the final mixed distance and a preset linear distribution algorithm, and update the target trajectory through the target matching pair to obtain a final target tracking result.

[0114] It should be noted that the foregoing explanation and description of the embodiment of the multi-target tracking method based on the diffusion model also applies to the embodiment of the multi-target tracking device based on the diffusion model, which will not be described here again.

[0115] The multi-target tracking device based on the diffusion model according to the embodiment of the present application comprises a target detection module 100, a trajectory prediction module 200, and a matching association module 300.

[0116] Figure 7 The electronic device provided by the embodiment of the present application has the structure shown in the structural schematic diagram of the electronic device. The electronic device can comprise:

[0117] The memory 701, the processor 702, and the computer program stored in the memory 701 and executable on the processor 702.

[0118] The processor 702 implements the diffusion model based multi-target tracking method provided in the above embodiments when executing a program.

[0119] Further, the electronic device further comprises:

[0120] The communication interface 703 is configured to communicate between the memory 701 and the processor 702.

[0121] The memory 701 is configured to store a computer program executable on the processor 702.

[0122] The memory 701 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0123] If the memory 701, the processor 702 and the communication interface 703 are implemented independently, the communication interface 703, the memory 701 and the processor 702 can be connected through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 In the figure, only one thick line is used to represent, but it does not mean that there is only one bus or one type of bus.

[0124] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can complete communication between each other through an internal interface.

[0125] The processor 702 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0126] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the diffusion model based multi-target tracking method as above.

[0127] The embodiment of the present application further provides a computer program product comprising a computer program, which, when executed, is configured to implement the multi-target tracking method based on the diffusion model.

[0128] In the description of the present specification, the description referring to the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or N embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples, without contradiction.

[0129] In addition, the terms "first", "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0130] Any process or method descriptions in flow charts or described elsewhere herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for performing specific logic functions or steps in the process, and that the various embodiments of the preferred implementation include the use of hardware and software configured to implement such functions or steps, and that the modules or segments can be implemented wherever, when, and how they are required for the specified purpose with the use of one or more machine-readable media. The preferred implementation of the present application thus includes a computer program product which can be implemented as a computer program, which is executable by a computer processing unit and / or which can be downloaded to a mobile terminal and / or a computer or network element via, for example, the Internet.

[0131] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of executable instructions stored in a computer readable medium, which can be executed by an instruction execution system, apparatus or device, such as a computer-based system, a processor-based system, or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or a combination of them. For the purposes of this specification, a "computer readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus or device. The computer readable medium can be a computer readable storage medium or a computer readable signal medium. The computer readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a propagation medium. The computer readable signal medium can include, but is not limited to, a computer readable medium that facilitates transfer of the program from one place to another. A specific example of a computer readable medium is a non-transitory computer-readable storage medium. A specific example of a computer readable signal medium is a source or destination of the computer readable medium. Another specific example of a computer readable signal medium is a computer readable signal travelling through space. Thus, a computer readable medium can take many forms of hardware to carry out the program for use by or in connection with the instruction execution system, apparatus or device.

[0132] It should be understood that aspects of the application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware and in another embodiment, the hardware can be implemented with any or a combination of the following technologies, which are all well known in the art: a discrete logic circuit(s) having logic gates for implementing logic functions upon an application of data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array(s) (PGA), a field programmable gate array (FPGA), etc.

[0133] Those of skill in the art would understand that the steps carried out by the above-mentioned embodiments can be implemented by a program instructing the relevant hardware to complete all or part of the steps, and the program can be stored in a computer readable storage medium. When the program is executed, it includes one of the steps of the method embodiments or a combination thereof.

[0134] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0135] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A multi-target tracking method based on a diffusion model, characterized in that, Includes the following steps: The ROI features of the two frames to be detected are obtained by a preset feature extraction module, and the ROI features are input into a preset diffusion detection head module. Target detection is then performed on the two frames to be detected to generate target detection results for each target in the two frames to be detected. The two frames to be detected include the previous frame to be detected and the current frame to be detected. The target displacement of each target in the two frames of images to be detected is calculated based on the target detection results, and the target displacement is input into a preset decoupling diffusion trajectory prediction module to predict the target trajectory of each target in the previous frame of images to be detected, so as to obtain the trajectory prediction result. Based on a preset hybrid IoU and ReID distance calculation strategy, the target detection results and the trajectory prediction results are matched and associated to generate the final target tracking results; The step of matching and associating the target detection results and trajectory prediction results based on a preset hybrid IoU and ReID distance calculation strategy to generate the final target tracking result includes: Calculate the IoU distance and ReID distance between the target detection result and the trajectory prediction result, and calculate the distance weight corresponding to each coordinate distance between the IoU distance and the ReID distance; The final mixed distance corresponding to the IoU distance and the ReID distance is calculated based on the distance weight and the preset weight factor; Based on the final hybrid distance and the preset linear allocation algorithm, the target matching pairs corresponding to the target detection results and the trajectory prediction results are calculated, and the target trajectory is updated through the target matching pairs to obtain the final target tracking results.

2. The method according to claim 1, characterized in that, The process involves obtaining the Region of Interest (ROI) features of two consecutive frames of images to be detected through a preset feature extraction module, inputting the ROI features into a preset diffusion detection head module, and performing target detection on the two consecutive frames of images to generate target detection results for each target in the two consecutive frames of images. This includes: The two frames of images to be detected are input into the feature extraction module to output the ROI features of the target size corresponding to the two frames of images to be detected. Multiple random boxes are generated in the two consecutive frames of images to be detected, and the multiple random boxes are determined as the initial detection boxes corresponding to the two consecutive frames of images to be detected. Based on the preset multi-scale attention module, dynamic convolution module and data association head module, the diffusion detection head module is constructed, and the ROI features and the initial detection box are input into the diffusion detection head module so that the prediction boxes corresponding to the two frames of images to be detected are generated by the multi-scale attention module and the dynamic convolution module, and the noise between the multiple random boxes and the prediction boxes is calculated. The predicted bounding box and the noise are input into the data association header module to obtain the detection bounding box of the current iteration, and the initial detection bounding box is updated using the detection bounding box of the current iteration; The ROI features and the updated initial detection box are re-input into the diffusion detection head module to perform target detection operations iteratively until the preset time step iteration cutoff condition is met, so as to generate the target detection result.

3. The method according to claim 1, characterized in that, The step of inputting the target displacement into a preset decoupled diffusion trajectory prediction module to predict the target trajectory of each target in the previous frame of the image to be detected, and obtaining the trajectory prediction result, includes: The decoupled diffusion trajectory prediction module is constructed by using multiple preset multi-scale attention modules and motion fusion modules; Gaussian noise is added to the target displacement, and the target displacement after adding Gaussian noise is input to the decoupled diffusion trajectory prediction module, so that the target displacement after adding Gaussian noise is back-diffused by the multi-scale attention module and the motion fusion module to generate the target predicted displacement of the current frame image to be detected; Based on the predicted displacement of the target and the target detection result of the previous frame image to be detected, the trajectory prediction result of the current frame image to be detected is calculated.

4. A multi-target tracking device based on a diffusion model, characterized in that, include: The target detection module is used to obtain the ROI features of the two frames of images to be detected by a preset feature extraction module, input the ROI features into a preset diffusion detection head module, and perform target detection operation on the two frames of images to be detected to generate the target detection result of each target in the two frames of images to be detected. The two frames of images to be detected include the previous frame image to be detected and the current frame image to be detected. The trajectory prediction module is used to calculate the target displacement of each target in the two frames of images to be detected in the previous and next frames based on the target detection results, and input the target displacement into a preset decoupling diffusion trajectory prediction module to predict the target trajectory of each target in the previous frame of images to be detected, so as to obtain the trajectory prediction result. The matching and association module is used to perform matching and association operations on the target detection results and the trajectory prediction results based on a preset hybrid IoU and ReID distance calculation strategy to generate the final target tracking result; The matching and association module includes: The third calculation unit is used to calculate the IoU distance and ReID distance between the target detection result and the trajectory prediction result, and to calculate the distance weight corresponding to each coordinate distance between the IoU distance and the ReID distance; The fourth calculation unit is used to calculate the final mixed distance corresponding to the IoU distance and the ReID distance based on the distance weight and the preset weight factor; A linear allocation unit is used to calculate the target matching pair corresponding to the target detection result and the trajectory prediction result based on the final mixed distance and a preset linear allocation algorithm, and update the target trajectory through the target matching pair to obtain the final target tracking result.

5. The apparatus according to claim 4, characterized in that, The target detection module includes: The input unit is used to input the two frames of images to be detected into the feature extraction module to output the ROI features of the target size corresponding to the two frames of images to be detected. The generation unit is used to generate multiple random boxes in the two frames of images to be detected before and after the previous and current frames, and to determine the multiple random boxes as the initial detection boxes corresponding to the two frames of images to be detected before and after the previous and current frames. The first computing unit is used to construct the diffusion detection head module based on a preset multi-scale attention module, dynamic convolution module and data association head module, and input the ROI features and the initial detection box into the diffusion detection head module so as to generate the prediction boxes corresponding to the two frames of images to be detected through the multi-scale attention module and dynamic convolution module, and calculate the noise between the multiple random boxes and the prediction boxes. An update unit is used to input the predicted bounding box and the noise into the data association head module to obtain the detection bounding box of the current iteration, and to update the initial detection bounding box using the detection bounding box of the current iteration; An iterative unit is used to re-input the ROI features and the updated initial detection box into the diffusion detection head module to iteratively perform target detection operations until a preset time step iteration cutoff condition is met, so as to generate the target detection result.

6. The apparatus according to claim 4, characterized in that, The trajectory prediction module includes: The construction unit is used to construct the decoupled diffusion trajectory prediction module through multiple preset multi-scale attention modules and motion fusion modules; A noise addition unit is used to add Gaussian noise to the target displacement and input the target displacement after adding Gaussian noise to the decoupled diffusion trajectory prediction module, so that the target displacement after adding Gaussian noise is back-diffused through the multi-scale attention module and the motion fusion module to generate the target predicted displacement of the current frame image to be detected. The second calculation unit is used to calculate the trajectory prediction result of the current frame image to be detected based on the target predicted displacement and the target detection result of the previous frame image to be detected.

7. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the multi-target tracking method based on a diffusion model as described in any one of claims 1-3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the multi-target tracking method based on the diffusion model as described in any one of claims 1-3.

9. A computer program product, comprising a computer program, characterized in that, The computer program is executed by a processor to implement the multi-target tracking method based on a diffusion model as described in any one of claims 1-3.

Images