Multi-target tracking method based on matter waves in frequent occlusion scenes

Abstract

A multi-target tracking method based on matter waves in frequent occlusion scenes, by constructing and training a target detection network based on matter waves, in the online stage, the coordinates of the target in the image frame with occlusion are obtained from the image frame with no occlusion of the target through quantum evolution and quantum measurement, which is used to predict and match the coordinates of the occluded target in dense scenes, so that the ID of the same target before and after occlusion remains unchanged, thereby achieving continuous tracking of multiple targets. The present invention can connect the target trajectories before and after occlusion while accurately detecting the target, realize multi-target tracking under frequent occlusion, reduce the number of identity switching and loss of tracking multiple targets in extremely dense scenes as a whole, and improve the accuracy of the algorithm in multi-target tracking on public data sets.

Description

Technical Field

[0001] The present invention relates to a technology in the field of computer vision multi-target tracking, specifically a multi-target tracking method based on a motion prediction algorithm based on a matter wave function and a high-precision target detection algorithm in frequent occlusion scenes. Background Art

[0002] The existing deep learning-based multi-target tracking algorithm implements tracking. When a target appears again after being completely occluded for a period of time, the target detection algorithm cannot detect the target, and the undetectable target track will inevitably be cut off, and a new identity label will be given to it when it appears again. This leads to undesirable identity switching (IDSwitch). Since the deep learning-based multi-target tracking algorithm has a high dependence on vision, high-precision tracking cannot be achieved as long as the target cannot be detected. Therefore, the occlusion problem has become a key factor affecting the multi-target tracking effect. Summary of the invention

[0003] In view of the fact that existing multi-target tracking algorithms rely on a large number of manually annotated re-identification datasets but do not pay attention to the differences in appearance features caused by target motion in upper and lower frames of the video and the problem of mutual occlusion between targets, that is, the problem of continuous tracking of targets in frequent occlusion scenarios, the present invention proposes a multi-target tracking method based on matter waves in frequent occlusion scenarios. The method can accurately detect the target and connect the target trajectories before and after the occlusion to achieve multi-target tracking under frequent occlusion, reduce the number of identity switches and losses of multiple targets in extremely dense scenarios as a whole, and improve the accuracy (MOTA) of multi-target tracking of the algorithm on public datasets.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a multi-target tracking method based on matter waves in frequent occlusion scenes. By constructing and training a target detection network based on matter waves, the coordinates of the target in the image frame with occlusion are obtained from the image frame with no occlusion of the target through quantum evolution and quantum measurement in the online stage. The method is used to predict and match the coordinates of the occluded target in dense scenes, so that the identity of the same target before and after occlusion remains unchanged, thereby achieving continuous tracking of multiple targets.

[0006] The training refers to: setting a training loss function to optimize the synthesized matter wave coefficients.

[0007] The matter wave feature refers to: as the number of image frames blocked by the target increases, the quantum evolution of the target The obtained material wave function The variance of the material wave function gradually increases, that is, the central energy of the material wave function waveform radiates to both sides. The material wave function is a vector describing the state of an object in Hilbert space. Where: matter wave coefficient represents the projection of the matter wave in the Hilbert space consisting of the basis set ψi, and the non-negative scalar j satisfies φ j represents the phase, r j Represents the phase function The weighting coefficient is the center coordinate of the bounding box output by the detection algorithm.

[0008] The material wave function is obtained by solving the Schrödinger equation, specifically: Where: φ is the material wave function, Hamiltonian in: Planck's constant, potential energy function k is the elastic coefficient and the distance from the Δx moving point to the equilibrium point, m is the mass of the matter. When the Hamiltonian at a certain moment is known, the matter wave function corresponding to that moment can be calculated.

[0009] The method specifically comprises:

[0010] Step 1) After obtaining the center coordinates of the unobstructed target on the image through the target detection network, project it into the basis set and use the relationship between the material wave functions of two adjacent frames to calculate the center coordinates of the target. After obtaining Φ(r, t+1), we define the formula based on the material wave function: Substitute the known Φ and ψi to solve for the matter wave coefficient That is, the target state is transferred from an uncertain state to a certain state. In the absence of measurement, the target state is uncertain because it is simultaneously in all possible measurements. After measurement, the target state will collapse to a certain ground state. Where: Φ(r, t) is the matter wave of the current frame, Φ(r, t) multiplied by the phase motion function Get the matter wave function Φ(r, t+1) of the next frame.

[0011] Step 2) Use quantum measurement to predict the center coordinates of the object in the next frame: The center coordinates of the object in the next frame (Xc, Yc) are obtained by calculating the integral expression of quantum measurement according to the center coordinates of the object in the current frame (Xc, Yc). 1 c , Y 1 c ), specifically: X 1 c =∫(Φ(r,t+1) * X c Φ(r,t+1))dx,Y 1c =∫(Φ(r,t+1) * Y c Φ(r, t+1))dy, where: X c Indicates the center coordinate of the object on the X axis in the current frame. 1 c Indicates the center coordinate of the object on the X-axis in the next frame.

[0012] Step 3) Calculate the center coordinates of the object in the continuous frame by continuously looping quantum measurement and quantum evolution, specifically: 1 c and Y 1 c Bring in X 2 c =∫(Φ(r,t+1) * X 1 c Φ(r, t+1))dx and Y 2 c =∫(Φ(r,t+1) * Y 1 c Φ(r, t+1))dy gets X 2 c and Y 2 c , and further cyclically calculate to obtain (X 3 c , Y 3 c ),......,(X n c , Y n c ), realizing continuous frame prediction of target trajectory.

[0013] The present invention relates to a multi-target tracking system based on matter wave function for realizing the above method, comprising: a target detection network, a matter wave calculation module and a matter wave motion prediction module, wherein: the target detection network collects the target boundary frame and image features of the input image, the matter wave calculation module brings the image of the target before being blocked, the Planck constant, the imaginary unit and the Hamiltonian of the target into the Schrödinger equation to solve and obtain the target initialized matter wave function, the matter wave motion prediction module obtains the coordinate position of the target when it is blocked according to the image of the target before being blocked combined with the quantum evolution prediction of the matter wave, correctly connects the target trajectory before and after the blockage with the target identity, regenerates the target from annihilation through quantum measurement and quantum evolution and repairs the detection error, thereby realizing multi-target tracking of continuous frames.

[0014] The target detection network includes: a backbone network (backbone), a feature pyramid (FPN) and a network output unit, wherein: the backbone uses pooling kernels of different sizes to pool the features of the image respectively, and then stacks and convolves to obtain 128-dimensional high-dimensional image features; the FPN fuses the features of feature layers of different sizes to enhance feature extraction, and the network output unit outputs the confidence, bounding box and category information of the target.

[0015] The matter wave motion prediction module includes: a quantum measurement unit and a quantum evolution unit, wherein: the quantum measurement unit encodes the target detection result into the matter wave, integrates the encoded matter wave with the coordinates of the target in the previous frame to obtain the two-dimensional coordinates (Xc, Yc) of the target in the current frame, and the quantum evolution unit integrates the matter wave function of the current frame with the phase function Multiply them to get the matter wave function of the next frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a flow chart of the present invention;

[0017] Figure 2 is the potential energy function of the resonator system Schematic diagram of changes;

[0018] Figure 3 It is a schematic diagram of the system of the present invention;

[0019] Figure 4 Schematic diagram of quantum evolution E and quantum measurement M;

[0020] Figure 5 Schematic diagram for comparing the effects of the embodiments. DETAILED DESCRIPTION

[0021] like Figure 3 and Figure 4 As shown, this embodiment involves a multi-target tracking system based on matter wave function. In this embodiment, the target detection network is implemented by the YOLOX network. The backbone network in the YOLOX network utilizes the easy optimization characteristics of the residual network and can improve the accuracy by increasing the depth. FPN performs enhanced feature extraction and feature fusion of feature layers of different sizes is beneficial to extracting better features. The network output unit outputs the confidence, bounding box and category information of the target.

[0022] like Figure 1 and Figure 3 As shown, a multi-target tracking method based on material wave functions in a frequent occlusion scene involved in this embodiment includes:

[0023] Step 1: Use the ground state solution of the oscillator system as the basis set of matter waves, where the ground state solution of the oscillator system is obtained by the following method: Solve the Schrödinger equation Where: φ is the matter wave function, is the Hamiltonian, is the Laplace operator, Planck's constant, the potential energy function of the oscillator system satisfies the equation k is the elastic coefficient and the distance from the moving point to the equilibrium point, and m is the mass of the substance, so it is used as the potential energy function for solving the Schrödinger equation; the basis set and matter wave obtained from the oscillator are as follows Figure 2 As shown, the matter wave function that satisfies the Schrödinger equation is φ; when the Hamiltonian at a specific time is known, the current matter wave function is solved, and the components of the matter wave function are the basis set of the matter wave.

[0024] Step 2: Solve the matter wave coefficient according to the coordinates (Xc, Yc) of the target before being occluded, that is, use the target information before occlusion to predict the center coordinates of the target after occlusion. Specifically, obtain the center coordinates of the unoccluded object on the image through the YOLOX network, project the center coordinates of the object into the basis set of the matter wave obtained in step 1, and calculate the matter wave coefficient.

[0025] In quantum measurement, the object state is projected into the matter wave basis set through PVM, transferring the object state from an uncertain state to a certain event. In the absence of measurement, the state is uncertain because it is simultaneously on all possible measurements. After measurement, the state will collapse to a certain state. The matter wave and its component matter wave basis set have been obtained in step 1. If the matter wave shows a small variance, it means that the degree of obstruction of the object is small. If the object is severely obstructed, the variance of the matter wave will become large. The variance of the matter wave function can reflect the degree of obstruction of the target.

[0026] Step 3: Figure 3 As shown in the figure, the quantum measurement unit encodes the target detection result into the matter wave, and then integrates the encoded matter wave with the coordinates of the target in the previous frame to obtain the two-dimensional coordinates (Xc, Yc) of the target in the current frame. Then the quantum evolution unit integrates the matter wave function of the current frame with the phase function Multiply to get the material wave function of the next frame, and bring the material wave function into equation X 1 c =∫(Φ(r,t+1) * X c Φ(r, t+1))dx and Y 1 c =∫(Φ(r,t+1) * Y cΦ(r, t+1))dy, so as to obtain the coordinates of the target in the next frame (X 1 c , Y 1 c ), where: X c Indicates the center coordinate of the object on the X axis in the current frame. 1 c Indicates the center coordinate of the object on the X axis in the next frame. 1 c Indicates the center coordinate of the object on the Y axis in the current frame, Y 1 c Indicates the center coordinates of the object on the Y axis in the next frame. This process can obtain the center coordinates of the object in the next frame (Xc, Yc) from the center coordinates of the object in the current frame 1 c , Y 1 c ). Through continuous cycles of quantum measurement and quantum evolution, the center coordinates of the object in the continuous frame are calculated. After obtaining the center coordinates, X 1 c and Y 1 c Bring in X 2 c =∫(Φ(r,t+1) * X 1 c Φ(r, t+1))dx and Y 2 c =∫(Φ(r,t+1) * Y 1 c Φ(r, t+1))dy gets X 2 c and Y 2 c :The center coordinates of the occluded object in the continuous frames are obtained by continuously looping the above calculations, that is, (X 3 c , Y 3 c ),......,(X n c , Y n c ), and finally form a continuous frame prediction of the target trajectory.

[0027] like Figure 4 As shown, in step 3, ψi is input into the quantum evolution E module to solve the material wave function of the current frame. Among them: coefficient represents the projection of the matter wave in the Hilbert space consisting of the basis set ψi, and the non-negative scalar j satisfies φ j Indicates the phase; then the material wave function of the current frame and the phase function Multiply and calculate to get the material wave function when blocked Since the material wave function is in an uncertain state, the material wave function is integrated by quantum measurement M. 1 c =∫(Φ(r,t+1) * X c Φ(r, t+1))dx and Y 1 c =∫(Φ(r,t+1) * Y c Φ(r, t+1))dy gets the predicted coordinates under occlusion.

[0028] Step 4: Initialize the multi-target tracking system and perform offline training, including:

[0029] 4.1) Set the convergence conditions and maximum number of iterations for multi-target tracking system training, initialize the network model parameters and set the hyperparameters of the optimization objective function.

[0030] 4.2) Normalize and enhance the multi-target tracking data.

[0031] The data enhancement processing includes but is not limited to operations such as flipping, rotating, translating, scaling, and random cropping.

[0032] The original image data set used in this embodiment is derived from MOT15, MOT17 and MOT20. The main feature of these data is that there are many targets and they belong to data sets collected in dense scenes. The targets are divided into two states. The first state is that the detector detects correctly and there is no obstruction, and the target can be accurately tracked; the second state is that the detector cannot detect correctly and there is an occlusion. For targets in the first state, multi-target tracking can be achieved by using the characteristics of the target or setting a threshold for the intersection and union ratio of the target bounding boxes of the previous and next frames. For the second state, it is necessary to use the Schrödinger equation to generate the material wave function of the corresponding target, so as to construct a motion model. Restore the target from the occlusion.

[0033] The multi-target tracking data is data of mutual occlusion between real targets collected from scenes such as different streets and different intersections. In this embodiment, the MOT20 dataset is used, and a total of 8 sequences are obtained, of which: half are used as training sets and half are used as test sets. To avoid overfitting on the test set, the labels of the test set are not made public. These sequences are recorded in 3 different scenes. There are recorded sequences in each scene, and they are assigned to the training set and the test set. However, one scene is not used as a training set to measure the generalization ability of the model. The MOT20 dataset contains about 3 times the bounding box of the MOT17 dataset. All sequences are shot from high altitudes, and the average crowd density reaches 247 pedestrians per frame.

[0034] 4.3) Offline training:

[0035] 4.3.1) Input the multi-target tracking data processed in step 4.2 into the initialized multi-target tracking system. According to the difference between the predicted value and the true value, the multi-target tracking system calculates the gradient according to the difference, and continuously optimizes the parameters of the multi-target tracking system, so that the difference between the predicted value and the true value continues to decrease until the difference is infinitely close to zero. In summary, the offline training of the multi-target tracking system is realized.

[0036] The error includes: cross entropy loss function as confidence prediction loss And the GIOU loss function as the positioning loss, where: p(y = y s |x s ) represents the target input category x s Predicted as category y s The probability size, N s Represents the total number of target samples, log() function is the logarithmic function with e as the base, D s Represents the set of categories of the target.

[0037] The update preferably uses the stochastic gradient descent method to update the network parameters, that is, the overall optimization goal Where: p(y=y s |x s ) represents the target input category x s Predicted as category y s The probability size, N s Represents the total number of target samples, log() function is the logarithmic function with e as the base, D s Represents the target category set, and the ultimate goal is to minimize L and obtain the required model.

[0038] 4.3.2) When the loss function converges or the maximum number of iterations is reached, the training ends and the network model and parameters are saved.

[0039] Step 5: Use the trained multi-target tracking system to perform continuous target tracking of online multi-frame images.

[0040] After specific practical experiments, the training was conducted on an NVIDIA Tesla V100 GPU with 32GB of running memory. In the experiment, the batch size was 2, the initial learning rate was 0.01, and the momentum was 0.9. The optimizer was SGD. The algorithm converged after multiple iterations of calculating the target loss function. There were a total of 150 training epochs, and the total training time was about 60 hours to complete the training of the model.

[0041] This embodiment quantitatively evaluates the results of the multi-target tracking algorithm based on the matter wave function through the multi-target tracking accuracy, specifically including: Where: m t is the number of missed detections (FP), that is, in the tth frame, there is no assumed position matching the target, f pt is FN (false positive number), that is, the assumed position given in the tth frame has no tracking target matching it, mmet is ID switch, that is, the number of ID switches of the tracking target in the tth frame, which mostly occurs in this case.

[0042] The above method is mainly used to reduce the ID switch on MOT20 and the proportion of correctly tracked trajectories in less than 20% of the frames (ML), and improve the accuracy of MOTA multi-target tracking, IDF1, and the proportion of correctly tracked trajectories in 80% of the frames (MT).

[0043] like Figure 5 As shown, it is a schematic diagram of the effect comparison of this embodiment in the frequent occlusion scene. The model is compared with five classic and high-performance tracking algorithm models on the MOT20 test set: the present invention is better than the current best tracking performance algorithm method SGT on the MOT20 test set, 4.5% higher in MOTA index, and 5.7% higher in IDF1 index. Compared with FairMOT, the MT index is 2.1% higher, and the erroneous switching of the model's identity is significantly reduced. The present invention is better than the most recent end-to-end graphics tracker (JDMOTGNN). The continuous frame prediction of the target motion position is combined with the spatial and appearance features of the previous frame of the object. The MOTA index of the present invention is 10.2% higher than that of JDMOTGNN, reflecting high robustness.

[0044] As shown in the table below, the tracker with the highest accuracy on MOT20 is compared. The MOTA index of the present invention is 77.3%, far exceeding other methods and achieving the best performance on MOT20. The MOTA index of the present invention is 21% higher than that of LPCMOT, and the IDF1 index is 13.7% higher than that of LPCMOT. In this frequently occluded MOT20 dataset, the present invention still maintains a low identity switch, reflecting a significant improvement in tracking association, and forms a stable association for the two obviously different states before and after occlusion.

[0045] MOTA↑ IDF1↑ MT↑ ML↓ IDs↓ CorrTracker 65.2 69.1 66.4 8.9 5183 JDMOTGNN 67.1 67.5 53.1 13.2 3131 FairMOT 61.8 67.3 68.8 7.6 5243 SORT 42.7 45.1 16.7 26.2 4470 LPCMOT 56.3 62.5 34.1 25.2 1562 SGT 72.8 70.5 64.3 12.8 2649 The present invention 77.3 76.2 70.9 7.2 1014

[0046] In summary, the present invention uses the information before occlusion to calculate the position of the object after occlusion and forms a continuous track position. For scenes with frequent occlusion, the present invention can better form the correct object motion trajectory to reduce identity switching and improve the MOTA index, with very high tracking performance.

[0047] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A multi-target tracking method based on matter waves in frequent occlusion scenes, characterized in that: By building and training a target detection network based on matter waves, the coordinates of the target in the image frame with occlusion are obtained from the image frame with no occlusion through quantum evolution and quantum measurement in the online stage. This is used to predict and match the coordinates of the occluded target in dense scenes, so that the ID of the same target before and after occlusion remains unchanged, thereby achieving continuous tracking of multiple targets. The matter wave feature is: as the number of image frames blocked by the target increases, the quantum evolution of the target The obtained material wave function The variance of the material wave function gradually increases, that is, the central energy of the material wave function waveform radiates to both sides. The material wave function is a vector describing the state of an object in Hilbert space. Where: matter wave coefficient represents the projection of the matter wave in the Hilbert space consisting of the basis set ψi, and the non-negative scalar j satisfies φ j represents the phase, r j Represents the phase function The weighting coefficient of is the center coordinate of the bounding box output by the detection algorithm; The material wave function is obtained by solving the Schrödinger equation, specifically: Where: φ is the material wave function, Hamiltonian in: Planck's constant, potential energy function k is the elastic coefficient and the distance from the moving point Δx to the equilibrium point, m is the mass of the substance. When the Hamiltonian at a certain moment is known, the material wave function corresponding to that moment can be calculated; The predicting and matching of the coordinates of the occluded target in a dense scene includes: Step 1) After obtaining the center coordinates of the unobstructed target on the image through the target detection network, project it into the basis set and use the relationship between the material wave functions of two adjacent frames to calculate the center coordinates of the target. After obtaining Φ(r,t+1), we define the formula according to the material wave function: Substitute the known Φ and ψi to solve for the matter wave coefficient That is, the target state is transferred from an uncertain state to a certain state; in the absence of measurement, the target state is uncertain because it is simultaneously in all possible measurements; after measurement, the target state will collapse to a certain ground state Where: Φ(r,t) is the matter wave of the current frame, Φ(r,t) multiplied by the phase motion function Get the material wave function Φ(r,t+1) of the next frame; Step 2) Use quantum measurement to predict the center coordinates of the object in the next frame: By calculating the integral expression of quantum measurement, the center coordinates of the object in the next frame (Xc, Yc) are obtained according to the center coordinates of the object in the current frame (Xc, Yc). 1 c ,Y 1 c ), specifically: X 1 c =∫(Φ(r,t+1) * X c Φ(r,t+1))dx,Y 1 c =∫(Φ(r,t+1) * Y c Φ(r,t+1))dy, where: X c Indicates the center coordinate of the object on the X axis in the current frame. 1 c Indicates the center coordinate of the object on the X-axis in the next frame; Step 3) Calculate the center coordinates of the object in the continuous frame by continuously looping quantum measurement and quantum evolution, specifically: 1 c and Y 1 c Bring in X 2 c =∫(Φ(r,t+1) * X 1 c Φ(r,t+1))dx and Y 2 c =∫(Φ(r,t+1) * Y 1 c Φ(r,t+1))dy to get X 2 c and Y 2 c , and further cyclically calculate to obtain (X 3 c ,Y 3 c ),......,(X n c ,Y n c ), realize continuous frame prediction of target trajectory; The target detection network includes: a backbone network (backbone), a feature pyramid (FPN) and a network output unit, wherein: the backbone uses pooling kernels of different sizes to pool the features of the image respectively, and then stacks and convolves to obtain 128-dimensional high-dimensional image features; the FPN fuses the features of feature layers of different sizes to enhance feature extraction, and the network output unit outputs the confidence, bounding box and category information of the target; The quantum measurement encodes the target detection result into a matter wave, and integrates the encoded matter wave with the coordinates of the target in the previous frame to obtain the two-dimensional coordinates (Xc, Yc) of the target in the current frame; The quantum evolution described above transforms the matter wave function of the current frame into the phase function Multiply them to get the matter wave function of the next frame.

2. The method for tracking multiple targets based on matter waves in frequent occlusion scenarios according to claim 1 is characterized in that: The training refers to: setting a training loss function to optimize the synthesized matter wave coefficient, specifically including: 1) Set the convergence conditions and maximum number of iterations for multi-target tracking system training, initialize the network model parameters, and set the hyperparameters of the optimization objective function; 2) Normalize and enhance the multi-target tracking data; 3) Offline training: 3.1) Input the multi-target tracking data processed in step 2 into the initialized multi-target tracking system. According to the difference between the predicted value and the true value, the multi-target tracking system calculates the gradient according to the difference, and continuously optimizes the parameters of the multi-target tracking system, so that the difference between the predicted value and the true value is continuously reduced until the difference is infinitely close to zero. In summary, the offline training of the multi-target tracking system is realized; 3.2) When the loss function converges or reaches the maximum number of iterations, the training ends and the network model and parameters are saved; The optimization described above uses the stochastic gradient descent method to update the network parameters, that is, the overall optimization goal Where: p(y=y s |x s Represents the target input category x s Predicted as category y s The probability size, N s Represents the total number of target samples, log() function is the logarithmic function with e as the base, D s Represents the target category set, and the ultimate goal is to minimize L and obtain the required model.

3. The method for tracking multiple targets based on matter waves in frequent occlusion scenarios according to claim 2 is characterized in that: The loss function includes: cross entropy loss function as confidence prediction loss And the GIOU loss function as the positioning loss, where: p(y = y s |x s ) represents the target input category x s Predicted as category y s The probability size, N s Represents the total number of target samples, log() function is the logarithmic function with e as the base, D s Represents the set of categories of the target.

4. A multi-target tracking system based on matter wave function for implementing any of the methods described in claims 1-3, characterized in that: include: A target detection network, a matter wave calculation module and a matter wave motion prediction module, wherein: the target detection network collects the target bounding box and image features of the input image; the matter wave calculation module brings the image of the target before being occluded, the Planck constant, the imaginary unit and the Hamiltonian of the target into the Schrödinger equation to solve and obtain the target initialized matter wave function; the matter wave motion prediction module obtains the coordinate position of the target when it is occluded based on the image of the target before being occluded combined with the quantum evolution prediction of the matter wave, correctly connects the target trajectory before and after occlusion with the target identity, regenerates the target from annihilation through quantum measurement and quantum evolution, and repairs the detection error to achieve continuous frame multi-target tracking.

Images