A multi-target tracking method and device based on motion equation trajectory prediction

Abstract

The present application provides a multi-target tracking method and device based on motion equation trajectory prediction, which relates to the technical field of multi-target tracking. The method includes: obtaining the detection frames of all moving targets in the current image frame output by the detector; using the state vectors of all trajectories in the previous image frame, based on the motion equations of constant angular velocity and constant linear acceleration, using extended Kalman filtering to obtain the prediction frames of all trajectories in the previous image frame in the current image frame; based on the association cost matrix between each detection frame and each prediction frame, using a matching algorithm to obtain the detection frame that matches successfully in the current image frame; using the motion information of the detection frame that matches successfully in the current image frame to update the corresponding prediction frame. The present application improves the accuracy of trajectory prediction, thereby improving the accuracy of multi-target tracking.

Description

Technical Field

[0001] The present application relates to the technical field of multi-target tracking, and in particular to a multi-target tracking method and device based on motion equation trajectory prediction. Background Art

[0002] With the advancement of computing platforms and deep learning technologies, 3D environmental perception technologies in complex electromechanical systems such as robots and self-driving cars have developed rapidly. Among them, 3D Multi Object Tracking (3D MOT) is a link between the previous and the next in environmental perception technologies, providing the motion trajectory of the same object in the dimension of time. 3D MOT usually first receives the detection results of the current frame, then locates the objects across the time dimension through the matching module, and finally uses the matched detection results to maintain the trajectory. With the emergence of a large number of high-accuracy 3D detection algorithms, the post-detection tracking paradigm based on physical models has been widely used in 3D MOT. However, since objects usually have irregular and large-scale inter-frame motion in adjacent frames, it is extremely unstable and inaccurate to directly use the physical features of the historical trajectory (usually including the center point coordinates and the detection box size) and the physical features of the current frame detection box to calculate the cost and solve the matching relationship.

[0003] In order to reduce the impact of irregular and large-scale motion between object frames on the matching results, it is necessary to predict the active trajectory, and then use the predicted state of the trajectory in the current frame to associate data with the current frame detection frame, and at the same time correct the state of the trajectory in the current frame after data association.

[0004] In order to accurately predict and correct the trajectory state, there are currently two main mainstream methods. One method is based on a neural network, that is, the motion information of the trajectory that is still active in the current frame in the past few frames is sent to the time series neural network (such as LSTM, etc.), and then the state of the active trajectory in the current frame is predicted. After obtaining the matching relationship, the trajectory state is corrected using the matching detection frame using an artificially designed function (such as taking the weighted average of prediction and detection). However, this type of method is usually sensitive to the accuracy of the input during the prediction process, and will overfit the data set, with poor portability and serious human influence during the correction process. Another method is based on linear Kalman filtering, that is, assuming that the object is based on a simple physical motion model and moves at a constant acceleration or constant speed, and the transfer state of the object at the next moment is obtained. After matching, the corresponding detection frame is used to perform a linear Kalman adaptive update on the trajectory state. Although this method has strong generalization ability in different scenarios and has a high degree of adaptive correction, it cannot describe the rotation process and nonlinear characteristics of the object, and cannot better characterize the true motion state of the object. Summary of the invention

[0005] In view of this, the present application provides a multi-target tracking method and device based on motion equation trajectory prediction to solve the above technical problems.

[0006] In a first aspect, an embodiment of the present application provides a multi-target tracking method based on motion equation trajectory prediction, the method comprising:

[0007] Get the detection frames of all moving targets in the current image frame output by the detector;

[0008] Using the state vectors of all trajectories in the previous image frame, based on the motion equations of constant angular velocity and constant linear acceleration, the extended Kalman filter is used to obtain the prediction box of all trajectories in the previous image frame in the current image frame;

[0009] Based on the correlation cost matrix between each detection frame and each prediction frame, a matching algorithm is used to obtain the detection frame that matches successfully in the current image frame;

[0010] Use the motion information of the successfully matched detection frame in the current image frame to update the corresponding prediction frame.

[0011] Furthermore, the method further includes: constructing an extended Kalman filter equation for a moving target; including:

[0012] The extended Kalman filter equation of the moving target is:

[0013] X t =(X t-1 )

[0014] Where F(·) represents the state transfer equation, X t-1 is the state vector of the trajectory of the previous image frame; X t is the state vector of the trajectory of the current image frame;

[0015] Among them, the state vector of the trajectory is X = [x, y, z, w, h, l, v, a, θ, ω], (, y, z) represents the coordinates of the center of the detection box in the global coordinate system, w, h and l represent the width, height and length of the detection box respectively, v is the total velocity of the detection box in the global coordinate system; a is the total acceleration of the detection box; ω is the angular velocity of the detection box; θ is the yaw angle of the detection box around the Z axis of the global coordinate system.

[0016] Furthermore, using the state vectors of all trajectories in the previous image frame, based on the motion equation of constant angular velocity and constant linear acceleration, the prediction box of all trajectories in the previous image frame in the current image frame is obtained by using the extended Kalman filter; including:

[0017] Construct the extended Kalman filter equation for moving targets;

[0018] According to the matching results of the detection frame and the prediction frame of the previous image frame, the final state vector of the trajectory of the previous image frame is determined And the corresponding final covariance matrix P t-1 , t is the serial number of the current image frame;

[0019] Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thus obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame.

[0020] Furthermore, according to the matching result between the detection frame and the prediction frame of the previous image frame, the final state vector of the trajectory of the previous image frame is determined. And the corresponding final covariance matrix P t-1 ;include:

[0021] For the successfully matched detection frame in the previous image frame, let the final state vector of the trajectory corresponding to the previous image frame be is the updated value of the state vector of the trajectory The final covariance matrix P t-1 is the updated value P of the trajectory covariance matrix t-1|t-1 ; Final state vector It is expressed as:

[0022]

[0023] For the prediction box that was not successfully matched in the previous image frame, let the final state vector of the trajectory corresponding to the previous image frame be is the predicted value of the state vector of the trajectory The final covariance matrix P t-1 is the predicted value P of the covariance matrix t-1|t-2 ;

[0024] For the detection frame that was not successfully matched in the previous image frame, the detection frame is initialized as a new trajectory, and the measurement vector of the detection frame is used to obtain the final state vector of the trajectory corresponding to the previous image frame. in,

[0025] Furthermore, using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1, thereby obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; including:

[0026] According to the final state vector of the trajectory of the previous image frame Calculate the state vector prediction value of the current image frame trajectory

[0027]

[0028] in

[0029] when but and The calculation formula is:

[0030]

[0031]

[0032]

[0033]

[0034]

[0035]

[0036] when but and The calculation formula is:

[0037]

[0038]

[0039]

[0040]

[0041]

[0042]

[0043]

[0044]

[0045]

[0046]

[0047] Among them, Δt is the time interval between the current frame and the previous frame;

[0048] Calculate the covariance matrix prediction value P of the current image frame trajectory t|t-1 :

[0049]

[0050] Among them, Q is the prediction noise, which is set to the unit matrix multiplied by 0.1, and A t is the Jacobian matrix of the predicted value of the state vector of the state transfer function F(·) with respect to the current image frame trajectory.

[0051] Further, the corresponding prediction frame is updated using the successfully matched detection frame in the current image frame; including:

[0052] Get the measurement vector Z of the successfully matched detection frame in the current image frame t :

[0053] Z t =[x t ,t t ,z t ,w t ,h t ,l t ,vx t ,vy t ,θ t ];

[0054] Among them, vx t and vy t Respectively represent the current speed of the detection box along the X-axis and Y-axis in the global coordinate system;

[0055] Calculate the measurement residual tmp:

[0056]

[0057] Where H(·) represents the spatial mapping equation;

[0058] Calculate the measurement residual covariance S t :

[0059]

[0060] Among them, H t is the Jacobian matrix of the predicted value of the state vector of the trajectory of the space mapping equation H(·), is the matrix H t The transposed matrix of , R is the prediction noise, which is set to the unit matrix multiplied by 0.001;

[0061] Calculate the optimal Kalman gain K t :

[0062]

[0063] Calculate the state vector update value of the trajectory corresponding to the detection box And the covariance matrix update value P t|t :

[0064]

[0065] P t|t =(IK t H t ) t|t-1 .

[0066] In a second aspect, an embodiment of the present application provides a multi-target tracking device based on motion equation trajectory prediction, the device comprising:

[0067] An acquisition unit, which acquires the detection frames of all moving targets in the current image frame output by the detector;

[0068] A prediction unit, used to obtain a prediction box of all the trajectories in the previous image frame in the current image frame by using the state vectors of all the trajectories in the previous image frame and the motion equation of constant angular velocity and constant linear acceleration by using an extended Kalman filter;

[0069] An association unit, used to obtain a target sequence number of a successfully matched detection frame in a current image frame using a matching algorithm based on an association cost matrix between each detection frame and each prediction frame;

[0070] The updating unit is used to update the corresponding prediction frame using the successfully matched detection frame in the current image frame.

[0071] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of the embodiment of the present application when executing the computer program.

[0072] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions implement the method of the embodiment of the present application when executed by a processor.

[0073] This application improves the accuracy of trajectory prediction, thereby improving the accuracy of multi-target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0075] Figure 1 A flowchart of a multi-target tracking method based on motion equation trajectory prediction provided in an embodiment of the present application;

[0076] Figure 2 A functional structure diagram of a multi-target tracking device based on motion equation trajectory prediction provided in an embodiment of the present application;

[0077] Figure 3 A functional structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0079] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0080] First, the design concept of the embodiments of the present application is briefly introduced.

[0081] Currently, traditional prediction and correction methods used in multi-target tracking have the problems of poor generalization ability and inability to characterize the nonlinear motion of targets.

[0082] To solve the above technical problems, the present application proposes a multi-target tracking method based on motion equation trajectory prediction, which can characterize the more complex physical motion of the moving targets and use nonlinear filtering to predict and correct the trajectory state, thereby improving the accuracy of trajectory prediction and thus improving the accuracy of multi-target tracking.

[0083] The overall design idea of ​​the technical solution of this application is:

[0084] Each track is initialized by a detection frame, so no matter what kind of track, the initialization process should be considered first. Let the current frame be the tth frame, taking a single detection frame as an example, let For the i-th object that did not get a match in the previous frame (t-1 frame), follow the trajectory registration rule of immediately initializing the unmatched detection box in each frame as a new trajectory. The information contained is related to the data set. Usually, each detection box contains nine types of object information [x, y, z, w, h, l, vx, vy, θ] and class is used to represent the category of the object, where x, y, z represent the coordinates of the center of the detection box in the global coordinate system, w, h, l represent the width, height, and length of the detection box respectively, vx, vy represent the speed of the detection box along the X and Y coordinate axes in the global coordinate system respectively, and θ represents the yaw angle of the object (around the z axis of the global coordinate system). Use the detection box information to initialize the trajectory state vector and measurement vector. In order to facilitate the subsequent vehicle kinematics establishment, the measurement vector Z∈R of the selected trajectory is 9×1 is [x, y, z, w, h, l, vx, vy, θ], which is consistent with the motion information of the detection box, and the state vector X∈R of the selected trajectory 10×1 =[x,y,z,w,h,l,v,a,θ,ω], where v represents the total velocity of the object, a represents the total acceleration of the object, ω represents the angular velocity of the object, and the meanings of the other physical quantities are the same as above. Therefore, except for the total velocity v of the trajectory, other state quantities can be directly assigned, where the total acceleration a and angular velocity ω are initialized to 0, and the initialization formula for v is as follows:

[0085]

[0086] set up Initialized as the jth trajectory T j In order to enable the Extended Kalman Filter (EKF) to run, T j The predicted state at frame t-1 With update status The same, are the above initialization results. Since the extended Kalman filter is not sensitive to the initial value of the covariance matrix, T j The prediction covariance matrix at the t-1th frame is and update the covariance matrix same.

[0087] Trajectory state prediction can greatly reduce the instability caused by directly using the historical physical features of the trajectory to match the relationship, and can better simulate the inter-frame motion of the trajectory. Based on the rotation characteristics and nonlinear motion state, this application establishes the motion equations of constant angular velocity and constant linear acceleration, and uses the corresponding state transfer equations to predict the trajectory physical state of each trajectory that is still active in the current frame.

[0088] After obtaining the trajectory prediction state, the predicted trajectory state is associated and matched with the current detection frame. If the current frame t has a detection frame matching it, considering that the predicted trajectory state vector and covariance matrix are not very accurate. Since the detection frame Z matching the current trajectory is obtained t (i.e., observations) can update and correct the predicted state vector and covariance matrix of the trajectory according to the update process in the EKF; otherwise, the trajectory prediction state is directly output to the result file.

[0089] After introducing the application scenarios and design concepts of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below.

[0090] like Figure 1 As shown, the present application implements a multi-target tracking method based on motion equation trajectory prediction, comprising:

[0091] Step 101: Obtain detection frames of all moving objects in the current image frame output by the detector;

[0092] Step 102: using the state vectors of all trajectories in the previous image frame, based on the motion equation of constant angular velocity and constant linear acceleration, and using extended Kalman filtering, obtain the prediction boxes of all trajectories in the previous image frame in the current image frame;

[0093] First, the extended Kalman filter equation of the moving target is:

[0094] X t =(X t-1 )

[0095] Where F(·) represents the state transfer equation, X t-1 is the state vector of the trajectory of the previous image frame; X t is the state vector of the trajectory of the current image frame;

[0096] Among them, the state vector of the trajectory is X = [x, y, z, w, h, l, v, a, θ, ω], (, y, z) represents the coordinates of the center of the detection box in the global coordinate system, w, h and l represent the width, height and length of the detection box respectively, v is the total velocity of the detection box in the global coordinate system; a is the total acceleration of the detection box; ω is the angular velocity of the detection box; θ is the yaw angle of the detection box around the Z axis of the global coordinate system.

[0097] Specifically, the step includes:

[0098] According to the matching results of the detection frame and the prediction frame of the previous image frame, the final state vector of the trajectory of the previous image frame is determined And the corresponding final covariance matrix P t-1, t is the serial number of the current image frame; specifically:

[0099] For the successfully matched detection frame in the previous image frame, let the final state vector of the trajectory corresponding to the previous image frame be is the updated value of the state vector of the trajectory The final covariance matrix P t-1 is the updated value P of the trajectory covariance matrix t-1|t-1 ; Final state vector It is expressed as:

[0100]

[0101] For the prediction box that was not successfully matched in the previous image frame, let the final state vector of the trajectory corresponding to the previous image frame be is the predicted value of the state vector of the trajectory The final covariance matrix P t-1 is the predicted value P of the covariance matrix t-1|t-2 ;

[0102] For the detection frame that was not successfully matched in the previous image frame, the detection frame is initialized as a new trajectory, and the measurement vector of the detection frame is used to obtain the final state vector of the trajectory corresponding to the previous image frame. in,

[0103] Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thus obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; specifically:

[0104] According to the final state vector of the trajectory of the previous image frame Calculate the state vector prediction value of the current image frame trajectory

[0105]

[0106] Among them but and The calculation formula is:

[0107]

[0108]

[0109]

[0110]

[0111]

[0112]

[0113] when but and The calculation formula is:

[0114]

[0115]

[0116]

[0117]

[0118]

[0119]

[0120]

[0121]

[0122]

[0123]

[0124] Among them, Δt is the time interval between the current frame and the previous frame;

[0125] Calculate the covariance matrix prediction value P of the current image frame trajectory t|t-1 :

[0126]

[0127] Among them, Q is the prediction noise, which is set to the unit matrix multiplied by 0.1, and A t is the Jacobian matrix of the predicted value of the state vector of the state transfer function F(·) with respect to the current image frame trajectory.

[0128] Step 103: Based on the association cost matrix between each detection frame and each prediction frame, a matching algorithm is used to obtain a detection frame that is successfully matched in the current image frame;

[0129] Step 104: Use the motion information of the successfully matched detection frame in the current image frame to update the corresponding prediction frame.

[0130] In this embodiment, the step specifically includes:

[0131] Get the measurement vector Z of the successfully matched detection frame in the current image frame t :

[0132] Z t =[x t ,y t ,z t ,w t ,h t ,l t ,vx t ,vy t ,θ t ];

[0133] Among them, vx t and vy t Respectively represent the current speed of the detection box along the X-axis and Y-axis in the global coordinate system;

[0134] Calculate the measurement residual tmp:

[0135]

[0136] The spatial mapping equation H(·) predicts the value Mapped to the measurement space, the detection frame center (x, y, z), the detection frame width, height, length (, h, l), and the detection frame offset angle θ can be directly assigned. Only the mapping of the total velocity v needs to be considered separately, because the measurement input is the velocity of the lower X and Y coordinate axes of the global coordinate system, so v needs to be vectorized, and the following formula is available:

[0137] vx=vcos(θ)

[0138] vy=vsin(θ)

[0139] Calculate the measurement residual covariance S t :

[0140]

[0141] Among them, H t is the Jacobian matrix of the predicted value of the state vector of the trajectory of the space mapping equation H(·), is the matrix H t The transposed matrix of , R is the prediction noise, which is set to the unit matrix multiplied by 0.001;

[0142] Calculate the optimal Kalman gain K t :

[0143]

[0144] Calculate the state vector update value of the trajectory corresponding to the detection box And the covariance matrix update value P t|t :

[0145]

[0146] P t|t =(IK t H t ) t|t-1 .

[0147] In addition, the method further comprises:

[0148] If a certain track has not been updated by the detection frame, it is necessary to consider whether to delete this track. This not only saves hardware memory space and reduces calculation time, but also reduces the unstable associations caused by invalid tracks. After the track is predicted in each frame, if there is no matching detection frame to update it, the track is judged whether to delete it. The counting method can be used to delete the track, that is, the track will be deleted if it exceeds the maximum mismatch number N.

[0149] Based on the above embodiments, the present application provides a multi-target tracking device based on motion equation trajectory prediction. Figure 2 As shown, the multi-target tracking device 200 based on motion equation trajectory prediction provided by the embodiment of the present application at least includes:

[0150] An acquisition unit 201 acquires the detection frames of all moving objects in the current image frame output by the detector;

[0151] A prediction unit 202 is used to obtain a prediction box of all the trajectories in the previous image frame in the current image frame by using the state vectors of all the trajectories in the previous image frame and the motion equation of constant angular velocity and constant linear acceleration by using an extended Kalman filter;

[0152] An association unit 203 is used to obtain a target sequence number of a successfully matched detection frame in the current image frame using a matching algorithm based on an association cost matrix between each detection frame and each prediction frame;

[0153] The updating unit 204 is used to update the corresponding prediction frame using the successfully matched detection frame in the current image frame.

[0154] It should be noted that the principle of solving the technical problem by the multi-target tracking device 200 based on motion equation trajectory prediction provided in the embodiment of the present application is similar to the method provided in the embodiment of the present application. Therefore, the implementation of the multi-target tracking device 200 based on motion equation trajectory prediction provided in the embodiment of the present application can refer to the implementation of the multi-target tracking method based on motion equation trajectory prediction provided in the embodiment of the present application, and the repeated parts will not be repeated.

[0155] Based on the above embodiments, the present application also provides an electronic device, referring to Figure 3 As shown, the electronic device 300 provided in the embodiment of the present application includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the multi-target tracking method based on motion equation trajectory prediction provided in the embodiment of the present application is implemented.

[0156] The electronic device 300 provided in the embodiment of the present application may further include a bus 303 connecting different components (including the processor 301 and the memory 302). The bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.

[0157] The memory 302 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022 , and may further include a read-only memory (ROM) 3023 .

[0158] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, the program modules 3024 including but not limited to: an operating subsystem, one or more applications, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0159] The electronic device 300 may also communicate with one or more external devices 304 (e.g., keyboards, remote controls, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 300 (e.g., mobile phones, computers, etc.), and / or communicate with any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 305. Furthermore, the electronic device 300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 306. Figure 3 As shown, the network adapter 306 communicates with other modules of the electronic device 300 via the bus 303. It should be understood that although Figure 3Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, and data backup storage subsystems.

[0160] It should be noted that Figure 3 The electronic device 300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0161] The embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions, and when the computer instructions are executed by the processor, the method provided by the embodiment of the present application is implemented. Specifically, the executable program can be built-in or installed in the electronic device 300, so that the electronic device 300 can implement the multi-target tracking method based on motion equation trajectory prediction provided by the embodiment of the present application by executing the built-in or installed executable program.

[0162] The method provided in the embodiment of the present application can also be implemented as a program product, which includes a program code. When the program product can be run on the electronic device 300, the program code is used to enable the electronic device 300 to execute the multi-target tracking method based on motion equation trajectory prediction provided in the embodiment of the present application.

[0163] The program product provided in the embodiments of the present application may adopt any combination of one or more readable media, wherein the readable medium may be a readable signal medium or a readable storage medium, and the readable storage medium may be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0164] The program product provided in the embodiment of the present application may adopt a CD-ROM and include program code, and may also be run on a computing device. However, the program product provided in the embodiment of the present application is not limited thereto. In the embodiment of the present application, the readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.

[0165] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0166] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that any modification or equivalent replacement of the technical solution of the present application does not depart from the spirit and scope of the technical solution of the present application and should be included in the scope of the claims of the present application.

Claims

1. A multi-target tracking method based on motion equation trajectory prediction, characterized in that: The method comprises: Get the detection frames of all moving targets in the current image frame output by the detector; Using the state vectors of all trajectories in the previous image frame, based on the motion equations of constant angular velocity and constant linear acceleration, the extended Kalman filter is used to obtain the prediction box of all trajectories in the previous image frame in the current image frame; Based on the correlation cost matrix between each detection frame and each prediction frame, a matching algorithm is used to obtain the detection frame that matches successfully in the current image frame; Use the motion information of the successfully matched detection frame in the current image frame to update the corresponding prediction frame; The method further includes: constructing an extended Kalman filter equation for a moving target; including: The extended Kalman filter equation of the moving target is: X t =F(X t-1 ) Where F(·) represents the state transfer equation, X t-1 is the state vector of the trajectory of the previous image frame; X t is the state vector of the trajectory of the current image frame; Among them, the state vector of the trajectory is X = [x, y, z, w, h, l, v, a, θ, ω], (x, y, z) represents the coordinates of the center of the detection frame in the global coordinate system, w, h and l represent the width, height and length of the detection frame respectively, v is the total velocity of the detection frame in the global coordinate system; a is the total acceleration of the detection frame; ω is the angular velocity of the detection frame; θ is the yaw angle of the detection frame around the Z axis of the global coordinate system; Using the state vectors of all trajectories in the previous image frame, based on the motion equations of constant angular velocity and constant linear acceleration, the extended Kalman filter is used to obtain the prediction box of all trajectories in the previous image frame in the current image frame; including: According to the matching results of the detection frame and the prediction frame of the previous image frame, the final state vector of the trajectory of the previous image frame is determined And the corresponding final covariance matrix P t-1 , t is the serial number of the current image frame; Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thereby obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; According to the matching results of the detection frame and the prediction frame of the previous image frame, the final state vector of the trajectory of the previous image frame is determined And the corresponding final covariance matrix P t-1 ;include: For the successfully matched detection frame in the previous image frame, let the final state vector of the trajectory corresponding to the previous image frame be is the updated value of the state vector of the trajectory The final covariance matrix P t-1 is the updated value P of the trajectory covariance matrix t-1|t-1 ; Final state vector It is expressed as: For the prediction box that was not successfully matched in the previous image frame, let the final state vector of the trajectory corresponding to the previous image frame be is the predicted value of the state vector of the trajectory The final covariance matrix P t-1 is the predicted value P of the covariance matrix t-1|t-2 ; For the detection frame that was not successfully matched in the previous image frame, the detection frame is initialized as a new trajectory, and the measurement vector of the detection frame is used to obtain the final state vector of the trajectory corresponding to the previous image frame. in, Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 ;include: According to the final state vector of the trajectory of the previous image frame Calculate the state vector prediction value of the current image frame trajectory when but and The calculation formula is: when but and The calculation formula is: Among them, Δt is the time interval between the current frame and the previous frame; Calculate the covariance matrix prediction value P of the current image frame trajectory t|t-1 : Among them, Q is the prediction noise, which is set to the unit matrix multiplied by 0.1, and A t is the final state vector of the state transfer function F(·) about the trajectory of the previous image frame The Jacobian matrix of .

2. The method according to claim 1, characterized in that Use the successfully matched detection frame in the current image frame to update the corresponding prediction frame; including: Get the measurement vector Z of the successfully matched detection frame in the current image frame t : Z t [x t ,y t ,z t ,w t ,h t ,l t ,vx t ,vy t ,θ t ]: Among them, vx t and vy t Respectively represent the current speed of the detection box along the X-axis and Y-axis in the global coordinate system; Calculate the measurement residual tmp: Where H(·) represents the spatial mapping equation; Calculate the measurement residual covariance S t : Among them, H t is the Jacobian matrix of the predicted value of the state vector of the space mapping equation H(·) with respect to the trajectory, is the matrix H t The transposed matrix of , R is the prediction noise, which is set to the unit matrix multiplied by 0.001; Calculate the optimal Kalman gain K t : Calculate the state vector update value of the trajectory corresponding to the detection box And the covariance matrix update value P t|t : P t|t =(I-K t H t )P t|t-1 。 3. A multi-target tracking device based on motion equation trajectory prediction, characterized in that: The device comprises: An acquisition unit, which acquires the detection frames of all moving targets in the current image frame output by the detector; A prediction unit, used to obtain a prediction box of all the trajectories in the previous image frame in the current image frame by using the state vectors of all the trajectories in the previous image frame and the motion equation of constant angular velocity and constant linear acceleration by using an extended Kalman filter; An association unit, used to obtain a successfully matched detection frame in the current image frame using a matching algorithm based on an association cost matrix between each detection frame and each prediction frame; An updating unit, used to update a corresponding prediction frame using a successfully matched detection frame in a current image frame; The device further comprises: a construction unit; specifically used for: The extended Kalman filter equation of the moving target is: X t =F(X t-1 ) Where F(·) represents the state transfer equation, X t-1 is the state vector of the trajectory of the previous image frame; X t is the state vector of the trajectory of the current image frame; Among them, the state vector of the trajectory is X = [x, y, z, w, h, l, v, a, θ, ω], (x, y, z) represents the coordinates of the center of the detection frame in the global coordinate system, w, h and l represent the width, height and length of the detection frame respectively, v is the total velocity of the detection frame in the global coordinate system; a is the total acceleration of the detection frame; ω is the angular velocity of the detection frame; θ is the yaw angle of the detection frame around the Z axis of the global coordinate system; The prediction unit is specifically used for: According to the matching results of the detection frame and the prediction frame of the previous image frame, the final state vector of the trajectory of the previous image frame is determined And the corresponding final covariance matrix P t-1 , t is the serial number of the current image frame; Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thereby obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; According to the matching results of the detection frame and the prediction frame of the previous image frame, the final state vector of the trajectory of the previous image frame is determined And the corresponding final covariance matrix P t-1 ;include: For the successfully matched detection frame in the previous image frame, let the final state vector of the trajectory corresponding to the previous image frame be is the updated value of the state vector of the trajectory Final covariance matrix P t-1 is the updated value P of the trajectory covariance matrix t-1|t-1 ; Final state vector It is expressed as: For the prediction box that was not successfully matched in the previous image frame, let the final state vector of the trajectory corresponding to the previous image frame be is the predicted value of the state vector of the trajectory The final covariance matrix P t-1 is the predicted value P of the covariance matrix t-1|t-2 ; For the detection frame that was not successfully matched in the previous image frame, the detection frame is initialized as a new trajectory, and the measurement vector of the detection frame is used to obtain the final state vector of the trajectory corresponding to the previous image frame. in, Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 ;include: According to the final state vector of the trajectory of the previous image frame Calculate the state vector prediction value of the current image frame trajectory when but and The calculation formula is: when but and The calculation formula is: Among them, Δt is the time interval between the current frame and the previous frame; Calculate the covariance matrix prediction value P of the current image frame trajectory t|t-1 : Among them, Q is the prediction noise, which is set to the unit matrix multiplied by 0.1, and A t is the final state vector of the state transfer function F(·) about the trajectory of the previous image frame The Jacobian matrix of .

4. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 2 when executing the computer program.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 2 is implemented.

Images