Highly stable multi-target tracking method based on trajectory prediction of optimal motion model

Abstract

The present application provides a highly stable multi-target tracking method based on optimal motion model trajectory prediction, which relates to the field of multi-target tracking technology, including: obtaining the detection frames of all moving targets in the current image frame output by the detector; using the motion information of all moving targets in the previous image frame, based on the optimal motion equation of the category to which the moving targets belong, obtaining the prediction frames of all trajectories in the previous image frame in the current image frame; wherein the optimal motion equation of pedestrians is the motion equation of constant speed; the optimal motion equation of bicycles is the motion equation of constant acceleration; the optimal motion equation of cars and motorcycles is the motion equation of constant turning rate and speed amplitude; the optimal motion equation of trucks is the motion equation of constant turning rate and acceleration; based on the association cost matrix between each detection frame and each prediction frame, obtaining the detection frame that matches successfully in the current image frame. The present application improves the trajectory prediction accuracy of moving targets, thereby improving the tracking accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of multi-target tracking, and in particular to a highly stable multi-target tracking method based on optimal motion model trajectory prediction. Background Art

[0002] Trajectory prediction is the core module of trajectory management in multi-target tracking. Since there are often irregular movements and large inter-frame displacements between two adjacent frames, the cost matrix construction and solution error directly using the historical frame trajectory frame and the current frame detection frame is large. Therefore, trajectory prediction is introduced to predict the active trajectory, and then the three-dimensional trajectory frame representing the trajectory in the current frame state after trajectory prediction is used to associate data with the three-dimensional detection frame of the detector in the current frame to obtain the optimal matching state.

[0003] Existing multi-target tracking trajectory prediction methods mostly use time series neural networks or linear Kalman filters for prediction. However, although time series neural networks, i.e., multi-target tracking methods based on deep learning, have good effects, they are prone to overfitting of data sets and have poor generalization performance. Although linear Kalman filters have strong generalization and fast speed in different scenarios, they are only applicable to relatively simple road conditions and linear motion models, and do not consider the influence of angular velocity, acceleration, turning rate, etc., and cannot fully meet the actual motion situation, and have poor performance. At the same time, existing multi-target tracking trajectory prediction methods generally simplify all categories of vehicles and pedestrians into the same motion model. Although it is simple, easy to implement, and has relatively strong generalization performance, the motion states of different vehicles and pedestrians are very different. For example, the motion states between trucks and pedestrians, such as speed, acceleration, turning rate, etc., are very different. Therefore, simplifying all tracking categories into the same motion model does not conform to the actual motion scene, and the prediction error is large. Summary of the invention

[0004] In view of this, the present application provides a highly stable multi-target tracking method based on optimal motion model trajectory prediction to solve the above technical problems.

[0005] In a first aspect, an embodiment of the present application provides a highly stable multi-target tracking method based on optimal motion model trajectory prediction, comprising:

[0006] Obtain the detection frames of all moving targets in the current image frame output by the detector;

[0007] Using the motion information of all moving targets in the previous image frame, based on the optimal motion equation of the category to which the moving targets belong, the prediction box of all trajectories in the previous image frame in the current image frame is obtained; wherein the categories include: pedestrians, cars, motorcycles, trucks and bicycles; the optimal motion equation of pedestrians is a constant speed motion equation; the optimal motion equation of bicycles is a constant acceleration motion equation; the optimal motion equation of cars and motorcycles is a constant turning rate and speed amplitude motion equation; the optimal motion equation of trucks is a constant turning rate and acceleration motion equation;

[0008] Based on the association cost matrix between each detection frame and each prediction frame, a matching algorithm is used to obtain the detection frame that matches successfully in the current image frame.

[0009] Furthermore, using the motion information of all moving targets in the previous image frame, based on the optimal motion equation of the category to which the moving targets belong, the prediction box of all trajectories in the previous image frame in the current image frame is obtained; including:

[0010] Get the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , t is the serial number of the current image frame;

[0011] Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thus obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame.

[0012] Furthermore, when the category of the moving target is a pedestrian, the final state vector of the trajectory of the previous image frame is It is expressed as:

[0013]

[0014] in: Represents the coordinates of the center of the detection frame of the moving target in the previous image frame in the global coordinate system, and is the width, height and length of the detection box, and They represent the speed along the X and Y axes in the global coordinate system of the detection box respectively; its state transfer function is:

[0015] X t =A1X t-1

[0016] Among them, A1 is the first state transfer matrix, Xt-1 is the state vector of the trajectory of the previous image frame; X t is the state vector of the trajectory of the current image frame;

[0017] Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thereby obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; including:

[0018] According to the final state vector of the trajectory of the previous image frame Use Kalman filtering to calculate the state vector prediction value of the current image frame trajectory

[0019]

[0020]

[0021]

[0022]

[0023]

[0024]

[0025]

[0026]

[0027]

[0028] Wherein, Δt is the time interval between the current image frame and the previous image frame;

[0029] Calculate the covariance matrix prediction value P of the current image frame trajectory t|t-1 :

[0030]

[0031] Where Q is the prediction noise.

[0032] Furthermore, when the category of the moving target is a bicycle, the final state vector of the trajectory of the previous image frame is It is expressed as:

[0033]

[0034] in: Represents the coordinates of the center of the detection frame of the moving target in the previous image frame in the global coordinate system, and is the width, height and length of the detection box, and Respectively represent the speed along the X and Y coordinate axes in the global coordinate system of the detection frame; and They represent the acceleration along the X and Y axes in the global coordinate system of the detection box respectively; their state transfer function is:

[0035] X t =A2X t-1

[0036] Among them, A2 is the second state transfer matrix, X t-1 is the state vector of the trajectory of the previous image frame; X t is the state vector of the trajectory of the current image frame;

[0037] Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thereby obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; including:

[0038] According to the final state vector of the trajectory of the previous image frame Use Kalman filtering to calculate the state vector prediction value of the current image frame trajectory

[0039]

[0040]

[0041]

[0042]

[0043]

[0044]

[0045]

[0046]

[0047]

[0048] Wherein, Δt is the time interval between the current image frame and the previous image frame;

[0049] Calculate the covariance matrix prediction value P of the current image frame trajectory t|t-1 :

[0050]

[0051] Where Q is the prediction noise.

[0052] Furthermore, when the category of the moving target is a car or a motorcycle, the final state vector of the trajectory of the previous image frame is It is expressed as:

[0053]

[0054] in: Represents the coordinates of the center of the detection frame of the moving target in the previous image frame in the global coordinate system, and is the width, height and length of the detection box of the moving target in the previous image frame, is the yaw angle of the detection frame of the moving target in the previous image frame, is the speed of the detection frame of the moving target in the previous image frame, is the angular velocity of the detection frame of the moving target in the previous image frame; its state transfer function is:

[0055] X t =F(X t-1 )

[0056] Where F(·) represents the nonlinear state transfer equation, X t-1 is the state vector of the trajectory of the previous image frame; X t is the state vector of the trajectory of the current image frame;

[0057] Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thereby obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; including:

[0058] According to the final state vector of the trajectory of the previous image frame Use the extended Kalman filter to calculate the state vector prediction value of the current image frame trajectory

[0059]

[0060] when but and The calculation formula is:

[0061]

[0062]

[0063]

[0064] when but and The calculation formula is:

[0065]

[0066]

[0067]

[0068]

[0069]

[0070]

[0071]

[0072]

[0073]

[0074] Wherein, Δt is the time interval between the current image frame and the previous image frame;

[0075] Calculate the covariance matrix prediction value P of the current image frame trajectory t|t-1 :

[0076]

[0077] Where Q is the prediction noise, A t is the final state vector of the state transfer function F(·) about the trajectory of the previous image frame The Jacobian matrix of .

[0078] In a second aspect, an embodiment of the present application provides a highly stable multi-target tracking device based on optimal motion model trajectory prediction, comprising:

[0079] An acquisition unit, used for acquiring the detection frames of all moving targets in the current image frame output by the detector;

[0080] A prediction unit, used to obtain a prediction frame of all trajectories in the previous image frame in the current image frame by using the motion information of all moving targets in the previous image frame and based on the optimal motion equation of the category to which the moving targets belong; wherein the categories include: pedestrians, cars, motorcycles, trucks and bicycles; the optimal motion equation of pedestrians is a motion equation of constant speed; the optimal motion equation of bicycles is a motion equation of constant acceleration; the optimal motion equation of cars and motorcycles is a motion equation of constant turning rate and speed amplitude; the optimal motion equation of trucks is a motion equation of constant turning rate and acceleration;

[0081] The matching unit is used to obtain the successfully matched detection frame in the current image frame using a matching algorithm based on the association cost matrix between each detection frame and each prediction frame.

[0082] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method as described in an embodiment of the present application when executing the computer program.

[0083] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions, and the computer instructions implement the method of the embodiment of the present application when executed by a processor.

[0084] The present application improves the trajectory prediction accuracy of a moving target, thereby improving the tracking accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0086] Figure 1 A flowchart of a highly stable multi-target tracking method based on optimal motion model trajectory prediction provided by an embodiment of the present application;

[0087] Figure 2 A functional structure diagram of a highly stable multi-target tracking device based on optimal motion model trajectory prediction provided by an embodiment of the present application;

[0088] Figure 3 A functional structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0089] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0090] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0091] First, the design concept of the embodiments of the present application is briefly introduced.

[0092] Existing multi-target tracking trajectory prediction methods mostly use time series neural networks or linear Kalman filters for prediction. However, although time series neural networks, i.e., multi-target tracking methods based on deep learning, have good effects, they are prone to overfitting of data sets and have poor generalization performance. Although linear Kalman filters have strong generalization and fast speed in different scenarios, they are only applicable to relatively simple road conditions and linear motion models, and do not consider the influence of angular velocity, acceleration, turning rate, etc., and cannot fully meet the actual motion situation, and have poor performance. At the same time, existing multi-target tracking trajectory prediction methods generally simplify all categories of vehicles and pedestrians into the same motion model. Although it is simple, easy to implement, and has relatively strong generalization performance, the motion states of different vehicles and pedestrians are very different. For example, the motion states between trucks and pedestrians, such as speed, acceleration, turning rate, etc., are very different. Therefore, simplifying all tracking categories into the same motion model does not conform to the actual motion scene, and the prediction error is large.

[0093] In order to achieve the optimal state of trajectory prediction and improve tracking accuracy, the present application proposes a highly stable multi-target tracking method based on optimal motion model trajectory prediction. This method adopts targeted motion models for various target categories to improve their compatibility with the actual motion state, thereby achieving optimal prediction performance and improving target tracking performance.

[0094] After introducing the application scenarios and design concepts of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below.

[0095] like Figure 1 As shown, the present application implements a highly stable multi-target tracking method based on optimal motion model trajectory prediction, comprising:

[0096] Step 101: Obtain detection frames of all moving objects in the current image frame output by the detector;

[0097] Step 102: using the motion information of all moving objects in the previous image frame and based on the optimal motion equation of the category to which the moving objects belong, obtain the prediction box of all trajectories in the previous image frame in the current image frame;

[0098] In this embodiment, considering that the motion states of targets such as vehicles, pedestrians, trucks, bicycles and cars are different, different motion models are used for different categories of targets according to their motion characteristics to model the motion states of targets of different categories.

[0099] For tracking target categories with simple actual road conditions and sparse data, linear motion models can be used to simplify them, effectively improving the inference speed and prediction speed, such as the constant velocity model CV (Constant Velocity) and the constant acceleration model CA (Constant Acceleration). For some tracking target categories with complex actual road conditions and motion conditions, when the linear model error is large, nonlinear models can be used for simplification, such as the constant turn rate and velocity amplitude model CTRV (Constant Turn Rate and Velocity), the constant turn rate and acceleration model CTRA (Constant Turn Rate and Acceleration), etc.

[0100] The categories include: pedestrians, cars, motorcycles, trucks and bicycles; the optimal motion equation for pedestrians is the motion equation for constant speed; the optimal motion equation for bicycles is the motion equation for constant acceleration; the optimal motion equation for cars and motorcycles is the motion equation for constant turning rate and speed amplitude; the optimal motion equation for trucks is the motion equation for constant turning rate and acceleration;

[0101] In this embodiment, the step specifically includes:

[0102] Get the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , t is the serial number of the current image frame;

[0103] Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thus obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame.

[0104] When the category of the moving target is a pedestrian, the final state vector of the trajectory of the previous image frame is It is expressed as:

[0105]

[0106] in: Represents the coordinates of the center of the detection frame of the moving target in the previous image frame in the global coordinate system, and is the width, height and length of the detection box, and They represent the speed along the X and Y axes in the global coordinate system of the detection box respectively; its state transfer function is:

[0107] X t =A1X t-1

[0108] Among them, A1 is the first state transfer matrix, X t-1 is the state vector of the trajectory of the previous image frame; X t is the state vector of the trajectory of the current image frame;

[0109] Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thereby obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; including:

[0110] According to the final state vector of the trajectory of the previous image frame Use Kalman filtering to calculate the state vector prediction value of the current image frame trajectory

[0111]

[0112]

[0113]

[0114]

[0115]

[0116]

[0117]

[0118]

[0119]

[0120] Wherein, Δt is the time interval between the current image frame and the previous image frame;

[0121] Calculate the covariance matrix prediction value P of the current image frame trajectory t|t-1 :

[0122]

[0123] Where Q is the prediction noise.

[0124] When the category of the moving target is a bicycle, the final state vector of the trajectory of the previous image frame is It is expressed as:

[0125]

[0126] in: Represents the coordinates of the center of the detection frame of the moving target in the previous image frame in the global coordinate system, and is the width, height and length of the detection box, and Respectively represent the speed along the X and Y coordinate axes in the global coordinate system of the detection frame; Know They represent the acceleration along the X and Y axes in the global coordinate system of the detection box respectively; their state transfer function is:

[0127] X t =A2X t-1

[0128] Among them, A2 is the second state transfer matrix, X t-1 is the state vector of the trajectory of the previous image frame; X t is the state vector of the trajectory of the current image frame;

[0129] Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thereby obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; including:

[0130] According to the final state vector of the trajectory of the previous image frame Use Kalman filtering to calculate the state vector prediction value of the current image frame trajectory

[0131]

[0132]

[0133]

[0134]

[0135]

[0136]

[0137]

[0138]

[0139]

[0140] Wherein, Δt is the time interval between the current image frame and the previous image frame;

[0141] Calculate the covariance matrix prediction value P of the current image frame trajectory t|t-1 :

[0142]

[0143] Where Q is the prediction noise.

[0144] When the category of the moving target is a car or a motorcycle, the final state vector of the trajectory of the previous image frame is It is expressed as:

[0145]

[0146] in: Represents the coordinates of the center of the detection frame of the moving target in the previous image frame in the global coordinate system, and is the width, height and length of the detection box of the moving target in the previous image frame, is the yaw angle of the detection frame of the moving target in the previous image frame, is the speed of the detection frame of the moving target in the previous image frame, is the angular velocity of the detection frame of the moving target in the previous image frame; its state transfer function is:

[0147] X t =F1(X t-1 )

[0148] Where F1(·) represents the first nonlinear state transfer equation, X t-1 is the state vector of the trajectory of the previous image frame; X t is the state vector of the trajectory of the current image frame;

[0149] Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thereby obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; including:

[0150] According to the final state vector of the trajectory of the previous image frame Use the extended Kalman filter to calculate the state vector prediction value of the current image frame trajectory

[0151]

[0152] when but and The calculation formula is:

[0153]

[0154]

[0155]

[0156] when but and The calculation formula is:

[0157]

[0158]

[0159]

[0160]

[0161]

[0162]

[0163]

[0164]

[0165]

[0166] Wherein, Δt is the time interval between the current image frame and the previous image frame;

[0167] Calculate the covariance matrix prediction value P of the current image frame trajectory t|t-1 :

[0168]

[0169] Where Q is the prediction noise, A 1t is the final state vector of the state transfer function F1(·) with respect to the trajectory of the previous image frame The Jacobian matrix of .

[0170] When the category of the moving target is a truck, the final state vector of the trajectory of the previous image frame is It is expressed as:

[0171]

[0172] in: Represents the coordinates of the center of the detection frame of the moving target in the previous image frame in the global coordinate system, and is the width, height and length of the detection box of the moving target in the previous image frame, is the yaw angle of the detection frame of the moving target in the previous image frame, is the speed of the detection frame of the moving target in the previous image frame, is the angular velocity of the detection frame of the moving target in the previous image frame, is the acceleration of the detection frame of the moving target in the previous image frame; its state transfer function is:

[0173] X t =F2(X t-1 )

[0174] Where F2(·) represents the second nonlinear state transfer equation, X t-1 is the state vector of the trajectory of the previous image frame; X t is the state vector of the trajectory of the current image frame;

[0175] Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thereby obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; including:

[0176] According to the final state vector of the trajectory of the previous image frame Calculate the state vector prediction value of the current image frame trajectory

[0177]

[0178] when but and The calculation formula is:

[0179]

[0180]

[0181] when but and The calculation formula is:

[0182]

[0183]

[0184]

[0185]

[0186]

[0187]

[0188]

[0189]

[0190]

[0191]

[0192] Among them, Δt is the time interval between the current frame and the previous frame;

[0193] Calculate the covariance matrix prediction value P of the current image frame trajectory t|t-1 :

[0194]

[0195] Where Q is the prediction noise, A 2t is the final state vector of the state transfer function F2(·) about the trajectory of the previous image frame The Jacobian matrix of .

[0196] Step 103: Based on the association cost matrix between each detection frame and each prediction frame, a matching algorithm is used to obtain a detection frame that has been successfully matched in the current image frame.

[0197] Based on the above embodiments, the present application provides a highly stable multi-target tracking device based on optimal motion model trajectory prediction. Figure 2 As shown, the highly stable multi-target tracking device 200 based on optimal motion model trajectory prediction provided by the embodiment of the present application at least includes:

[0198] An acquisition unit 201 is used to acquire the detection frames of all moving objects in the current image frame output by the detector;

[0199] The prediction unit 202 is used to obtain the prediction box of all the trajectories in the previous image frame in the current image frame based on the optimal motion equation of the category to which the moving target belongs, using the motion information of all the moving targets in the previous image frame; wherein the categories include: pedestrians, cars, motorcycles, trucks and bicycles; the optimal motion equation of pedestrians is a motion equation of constant speed; the optimal motion equation of bicycles is a motion equation of constant acceleration; the optimal motion equation of cars and motorcycles is a motion equation of constant turning rate and speed amplitude; the optimal motion equation of trucks is a motion equation of constant turning rate and acceleration;

[0200] The matching unit 203 is used to obtain a successfully matched detection frame in the current image frame using a matching algorithm based on the association cost matrix between each detection frame and each prediction frame.

[0201] It should be noted that the principle of solving the technical problem by the highly stable multi-target tracking device 200 based on optimal motion model trajectory prediction provided in the embodiment of the present application is similar to the method provided in the embodiment of the present application. Therefore, the implementation of the highly stable multi-target tracking device 200 based on optimal motion model trajectory prediction provided in the embodiment of the present application can refer to the implementation of the method provided in the embodiment of the present application, and the repeated parts will not be repeated.

[0202] Based on the above embodiments, the present application also provides an electronic device, referring to Figure 3 As shown, the electronic device 300 provided in the embodiment of the present application includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the highly stable multi-target tracking method based on optimal motion model trajectory prediction provided in the embodiment of the present application is implemented.

[0203] The electronic device 300 provided in the embodiment of the present application may further include a bus 303 connecting different components (including the processor 301 and the memory 302). The bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.

[0204] The memory 302 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022 , and may further include a read-only memory (ROM) 3023 .

[0205] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, the program modules 3024 including but not limited to: an operating subsystem, one or more applications, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0206] The electronic device 300 may also communicate with one or more external devices 304 (e.g., keyboards, remote controls, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 300 (e.g., mobile phones, computers, etc.), and / or communicate with any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 305. Furthermore, the electronic device 300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 306. Figure 3 As shown, the network adapter 306 communicates with other modules of the electronic device 300 via the bus 303. It should be understood that although Figure 3 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, and data backup storage subsystems.

[0207] It should be noted that Figure 3 The electronic device 300 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0208] The embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions, and when the computer instructions are executed by the processor, the vehicle re-identification method provided in the embodiment of the present application is implemented. Specifically, the executable program can be built-in or installed in the electronic device 300, so that the electronic device 300 can implement the highly stable multi-target tracking method based on optimal motion model trajectory prediction provided in the embodiment of the present application by executing the built-in or installed executable program.

[0209] The method provided in the embodiment of the present application can also be implemented as a program product, which includes a program code. When the program product can be run on an electronic device 300, the program code is used to enable the electronic device 300 to execute the highly stable multi-target tracking method based on optimal motion model trajectory prediction provided in the embodiment of the present application.

[0210] The program product provided in the embodiments of the present application may adopt any combination of one or more readable media, wherein the readable medium may be a readable signal medium or a readable storage medium, and the readable storage medium may be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0211] The program product provided in the embodiment of the present application may adopt a CD-ROM and include program code, and may also be run on a computing device. However, the program product provided in the embodiment of the present application is not limited thereto. In the embodiment of the present application, the readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.

[0212] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0213] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0214] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application and are not intended to limit it. Although the present application is described in detail with reference to the embodiments, a person skilled in the art should understand that any modification or equivalent replacement of the technical solution of the present application does not depart from the spirit and scope of the technical solution of the present application and should be included in the scope of the claims of the present application.

Claims

1. A highly stable multi-target tracking method based on optimal motion model trajectory prediction, characterized in that: include: Obtain the detection frames of all moving targets in the current image frame output by the detector; Using the motion information of all moving targets in the previous image frame, based on the optimal motion equation of the category to which the moving targets belong, the prediction box of all trajectories in the previous image frame in the current image frame is obtained; wherein the categories include: pedestrians, cars, motorcycles, trucks and bicycles; the optimal motion equation of pedestrians is a constant speed motion equation; the optimal motion equation of bicycles is a constant acceleration motion equation; the optimal motion equation of cars and motorcycles is a constant turning rate and speed amplitude motion equation; the optimal motion equation of trucks is a constant turning rate and acceleration motion equation; Based on the correlation cost matrix between each detection frame and each prediction frame, a matching algorithm is used to obtain the detection frame that matches successfully in the current image frame; Using the motion information of all moving targets in the previous image frame, based on the optimal motion equation of the category to which the moving targets belong, the prediction box of all trajectories in the previous image frame in the current image frame is obtained; including: Get the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , t is the serial number of the current image frame; Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thereby obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; When the category of the moving target is a pedestrian, the final state vector of the trajectory of the previous image frame is It is expressed as: in: Represents the coordinates of the center of the detection frame of the moving target in the previous image frame in the global coordinate system, and is the width, height and length of the detection box, and They represent the speed along the X and Y axes in the global coordinate system of the detection box respectively; its state transfer function is: X t =A1X t-1 Among them, A1 is the first state transfer matrix, X t-1 is the state vector of the trajectory of the previous image frame; X t is the state vector of the trajectory of the current image frame; Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thereby obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; including: According to the final state vector of the trajectory of the previous image frame Use Kalman filtering to calculate the state vector prediction value of the current image frame trajectory Wherein, Δt is the time interval between the current image frame and the previous image frame; Calculate the covariance matrix prediction value P of the current image frame trajectory t|t-1 : Where Q is the prediction noise; When the category of the moving target is a bicycle, the final state vector of the trajectory of the previous image frame is It is expressed as: in: Represents the coordinates of the center of the detection frame of the moving target in the previous image frame in the global coordinate system, and is the width, height and length of the detection box, and Respectively represent the speed along the X and Y coordinate axes in the global coordinate system of the detection frame; and They represent the acceleration along the X and Y axes in the global coordinate system of the detection box respectively; their state transfer function is: X t =A2X t-1 Among them, A2 is the second state transfer matrix, X t-1 is the state vector of the trajectory of the previous image frame; X t is the state vector of the trajectory of the current image frame; Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thereby obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; including: According to the final state vector of the trajectory of the previous image frame Use Kalman filtering to calculate the state vector prediction value of the current image frame trajectory Wherein, Δt is the time interval between the current image frame and the previous image frame; Calculate the covariance matrix prediction value P of the current image frame trajectory t|t-1 : Where Q is the prediction noise; When the category of the moving target is a car or a motorcycle, the final state vector of the trajectory of the previous image frame is It is expressed as: in: Represents the coordinates of the center of the detection frame of the moving target in the previous image frame in the global coordinate system, and is the width, height and length of the detection box of the moving target in the previous image frame, is the yaw angle of the detection frame of the moving target in the previous image frame, is the speed of the detection frame of the moving target in the previous image frame, is the angular velocity of the detection frame of the moving target in the previous image frame; its state transfer function is: X t =F(X t-1 ) Where F(·) represents the nonlinear state transfer equation, X t-1 is the state vector of the trajectory of the previous image frame; X t is the state vector of the trajectory of the current image frame; Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thereby obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; including: According to the final state vector of the trajectory of the previous image frame Use the extended Kalman filter to calculate the state vector prediction value of the current image frame trajectory when but and The calculation formula is: when but and The calculation formula is: Wherein, Δt is the time interval between the current image frame and the previous image frame; Calculate the covariance matrix prediction value P of the current image frame trajectory t|t-1 : Where Q is the prediction noise, A t is the final state vector of the state transfer function F(·) about the trajectory of the previous image frame The Jacobian matrix of .

2. A highly stable multi-target tracking device based on optimal motion model trajectory prediction, characterized in that: include: An acquisition unit, used for acquiring the detection frames of all moving targets in the current image frame output by the detector; A prediction unit, used to obtain a prediction frame of all trajectories in the previous image frame in the current image frame by using the motion information of all moving targets in the previous image frame and based on the optimal motion equation of the category to which the moving targets belong; wherein the categories include: pedestrians, cars, motorcycles, trucks and bicycles; the optimal motion equation of pedestrians is a motion equation of constant speed; the optimal motion equation of bicycles is a motion equation of constant acceleration; the optimal motion equation of cars and motorcycles is a motion equation of constant turning rate and speed amplitude; the optimal motion equation of trucks is a motion equation of constant turning rate and acceleration; A matching unit, used to obtain a successfully matched detection frame in the current image frame using a matching algorithm based on a correlation cost matrix between each detection frame and each prediction frame; The prediction unit is specifically used for: Get the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , t is the serial number of the current image frame; Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thereby obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; When the category of the moving target is a pedestrian, the final state vector of the trajectory of the previous image frame is It is expressed as: in: Represents the coordinates of the center of the detection frame of the moving target in the previous image frame in the global coordinate system, and is the width, height and length of the detection box, and They represent the speed along the X and Y axes in the global coordinate system of the detection box respectively; its state transfer function is: X t =A1X t-1 Among them, A1 is the first state transfer matrix, X t-1 is the state vector of the trajectory of the previous image frame; X t is the state vector of the trajectory of the current image frame; Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thereby obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; including: According to the final state vector of the trajectory of the previous image frame Use Kalman filtering to calculate the state vector prediction value of the current image frame trajectory Wherein, Δt is the time interval between the current image frame and the previous image frame; Calculate the covariance matrix prediction value P of the current image frame trajectory t|t-1 : Where Q is the prediction noise; When the category of the moving target is a bicycle, the final state vector of the trajectory of the previous image frame is It is expressed as: in: Represents the coordinates of the center of the detection frame of the moving target in the previous image frame in the global coordinate system, and is the width, height and length of the detection box, and Respectively represent the speed along the X and Y coordinate axes in the global coordinate system of the detection frame; and They represent the acceleration along the X and Y axes in the global coordinate system of the detection box respectively; their state transfer function is: X t =A2X t-1 Among them, A2 is the second state transfer matrix, X t-1 is the state vector of the trajectory of the previous image frame; X t is the state vector of the trajectory of the current image frame; Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thereby obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; including: According to the final state vector of the trajectory of the previous image frame Use Kalman filtering to calculate the state vector prediction value of the current image frame trajectory Wherein, Δt is the time interval between the current image frame and the previous image frame; Calculate the covariance matrix prediction value P of the current image frame trajectory t|t-1 : Where Q is the prediction noise; When the category of the moving target is a car or a motorcycle, the final state vector of the trajectory of the previous image frame is It is expressed as: in: Represents the coordinates of the center of the detection frame of the moving target in the previous image frame in the global coordinate system, and is the width, height and length of the detection box of the moving target in the previous image frame, is the yaw angle of the detection frame of the moving target in the previous image frame, is the speed of the detection frame of the moving target in the previous image frame, is the angular velocity of the detection frame of the moving target in the previous image frame; its state transfer function is: X t =F(X t-1 ) Where F(·) represents the nonlinear state transfer equation, X t-1 is the state vector of the trajectory of the previous image frame; X t is the state vector of the trajectory of the current image frame; Using the final state vector of the trajectory of the previous image frame And the corresponding final covariance matrix P t-1 , calculate the state vector prediction value of the current image frame trajectory And the corresponding covariance matrix predicted value P t|t-1 , thereby obtaining the predicted boxes of all trajectories in the previous image frame in the current image frame; including: According to the final state vector of the trajectory of the previous image frame Use the extended Kalman filter to calculate the state vector prediction value of the current image frame trajectory when but and The calculation formula is: when but and The calculation formula is: Wherein, Δt is the time interval between the current image frame and the previous image frame; Calculate the covariance matrix prediction value P of the current image frame trajectory t|t-1 : Where Q is the prediction noise, A t is the final state vector of the state transfer function F(·) about the trajectory of the previous image frame The Jacobian matrix of .

3. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to claim 1 when executing the computer program.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to claim 1 is implemented.

Images