Target tracking method and device, electronic equipment and storage medium

Abstract

The invention discloses a target tracking method and device, electronic equipment and a storage medium. The invention relates to the technical field of computer vision. The method comprises the following steps: acquiring a detection target frame of a to-be-tracked target in a current image frame; a prediction target frame of a to-be-tracked target in the current image frame is obtained, the prediction target frame is obtained through prediction based on a Kalman filter, and a state vector of the Kalman filter comprises a position change second derivative; performing generalized intersection-to-union ratio correlation matching on the detection target frame of the to-be-tracked target in the current image frame and the prediction target frame of the to-be-tracked target in the current image frame to obtain a matching result; and under the condition that the matching result meets a target tracking success condition, performing target tracking on the next image frame. According to the technical scheme, by introducing the position change second derivative, utilization of the change rate of the target movement speed is achieved, the accuracy of the prediction target frame can be improved, the prediction target frame is closer to a real target, and therefore the target tracking precision is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a target tracking method, device, electronic device and storage medium. Background Art

[0002] Target tracking technology can continuously identify and locate one or more moving targets over a period of time. It is widely used in video surveillance, robot navigation, and autonomous driving.

[0003] Existing target tracking technologies often use target tracking algorithms based on Kalman filters, such as the SORT multi-target tracking algorithm or the DeepSORT algorithm.

[0004] In the process of implementing the present disclosure, it is found that there are at least the following technical problems in the prior art: the above-mentioned existing target tracking solution has the problem of low target tracking accuracy. Summary of the invention

[0005] The present disclosure provides a target tracking method, device, electronic device and storage medium. By introducing the second-order derivative of position change, the rate of change of target motion speed is utilized, which can improve the accuracy of the predicted target frame and make the predicted target frame closer to the real target, thereby improving the target tracking accuracy.

[0006] According to one aspect of the present disclosure, a target tracking method is provided, comprising:

[0007] Obtaining a detection target frame of a target to be tracked in a current image frame, wherein the detection target frame is detected based on a target detection algorithm;

[0008] Acquire a predicted target frame of a target to be tracked in a current image frame, wherein the predicted target frame is obtained based on a Kalman filter prediction, and a state vector of the Kalman filter includes a second-order derivative of position change;

[0009] Performing generalized intersection-union-correlation matching on a detection target frame of the target to be tracked in the current image frame and a prediction target frame of the target to be tracked in the current image frame to obtain a matching result between the detection target frame and the prediction target frame;

[0010] When the matching result between the detected target frame and the predicted target frame satisfies the target tracking success condition, target tracking is performed on the next image frame.

[0011] According to another aspect of the present disclosure, there is provided a target tracking device, comprising:

[0012] A detection target frame acquisition module is used to acquire a detection target frame of a target to be tracked in a current image frame, wherein the detection target frame is detected based on a target detection algorithm;

[0013] A predicted target frame acquisition module is used to acquire a predicted target frame of a target to be tracked in a current image frame, wherein the predicted target frame is obtained based on a Kalman filter prediction, and a state vector of the Kalman filter includes a second-order derivative of a position change;

[0014] A generalized intersection-and-union ratio association matching module is used to perform generalized intersection-and-union ratio association matching on a detection target frame of a target to be tracked in the current image frame and a predicted target frame of a target to be tracked in the current image frame to obtain a matching result between the detection target frame and the predicted target frame;

[0015] The target tracking success judgment module is used to perform target tracking on the next image frame when the matching result between the detected target frame and the predicted target frame meets the target tracking success condition.

[0016] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0017] at least one processor;

[0018] and a memory communicatively coupled to the at least one processor;

[0019] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the target tracking method described in any embodiment of the present disclosure.

[0020] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the target tracking method described in any embodiment of the present disclosure when executed.

[0021] The technical solution of the disclosed embodiment is to obtain the detection target frame of the target to be tracked in the current image frame, wherein the detection target frame is detected based on the target detection algorithm, and then obtain the predicted target frame of the target to be tracked in the current image frame, wherein the predicted target frame is predicted based on the Kalman filter, and the state vector of the Kalman filter includes the second-order derivative of the position change, and then perform generalized intersection and union correlation matching on the detection target frame of the target to be tracked in the current image frame and the predicted target frame of the target to be tracked in the current image frame to obtain the matching result of the detection target frame and the predicted target frame; when the matching result of the detection target frame and the predicted target frame meets the target tracking success condition, the target is tracked in the next image frame. The above technical solution, by introducing the second-order derivative of the position change, realizes the utilization of the rate of change of the target motion speed, can improve the accuracy of the predicted target frame, make the predicted target frame closer to the real target, and thus improve the target tracking accuracy.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 is a flow chart of a target tracking method provided according to the first embodiment of the present disclosure;

[0025] Figure 2 is a flow chart of a target tracking method provided according to the second embodiment of the present disclosure;

[0026] Figure 3 is a flow chart of a target tracking method provided according to Embodiment 3 of the present disclosure;

[0027] Figure 4 is a flow chart of a target tracking method provided according to a fourth embodiment of the present disclosure;

[0028] Figure 5 is a structural schematic diagram of a target tracking device provided according to Embodiment 5 of the present disclosure;

[0029] Figure 6 It is a structural diagram of an electronic device for implementing the target tracking method of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the scheme of the present disclosure, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present disclosure.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0032] The standard Kalman filter assumes that the dynamic model and observation model of the system are linear, which requires that the target motion is a straight-line motion. However, in the real world, the motion patterns of many targets and the observation process of the sensor are nonlinear. For these situations, the standard Kalman filter may not be able to provide accurate state estimation, resulting in reduced target tracking accuracy. To this end, the embodiments of the present disclosure provide a target tracking method, device, electronic device and storage medium, which can effectively solve this problem. The following is a further detailed description of a target tracking method, device, electronic device and storage medium provided by the embodiments of the present disclosure.

[0033] Embodiment 1

[0034] Figure 1 This is a flow chart of a target tracking method provided in the first embodiment of the present disclosure. This embodiment is applicable to the case where a robot tracks a moving target. The method can be executed by a target tracking device. The target tracking device can be implemented in the form of hardware and / or software. The target tracking device can be configured in an electronic device such as a robot. Figure 1 As shown, the method includes:

[0035] S110, obtaining a detection target frame of a target to be tracked in a current image frame, wherein the detection target frame is detected based on a target detection algorithm.

[0036] The current image frame refers to the image frame being processed by the electronic device at the current moment, which can be acquired in real time by the robot's camera or read from a preset storage path of the electronic device. The target to be tracked can be a moving target such as a person or an animal. The target detection algorithm can be a single-stage detection algorithm or a two-stage detection algorithm, etc., which is not specifically limited here.

[0037] Exemplarily, target detection can be performed on image frames captured in real time by the robot camera to identify and lock the target to be tracked, and obtain the position information of the target to be tracked. The position information of the target to be tracked may include a detection target frame, that is, the detection target frame is a target frame obtained by a target detection algorithm.

[0038] S120, obtaining a predicted target frame of the target to be tracked in the current image frame, wherein the predicted target frame is obtained based on a Kalman filter prediction, and a state vector of the Kalman filter includes a second-order derivative of position change.

[0039] In the disclosed embodiment, the predicted target frame is obtained by predicting the target frame through a Kalman filter. It should be noted that the state vector of the Kalman filter designed in this embodiment includes not only the target position and the first-order derivative of the position change, but also the second-order derivative of the position change. The second-order derivative of the position change refers to the rate of change of the target motion speed, that is, the acceleration. By introducing the second-order derivative of the position change, the rate of change of the target motion speed is utilized, so that the Kalman filter can adapt to nonlinear variable speed motion, thereby improving the accuracy of the predicted target frame, making the predicted target frame closer to the real target, thereby improving the target tracking accuracy.

[0040] The target position may include the horizontal coordinate of the center point of the target frame, the vertical coordinate of the center point of the target frame, the area of ​​the target frame, and the aspect ratio of the target frame. The first-order derivative of the position change may include the first-order derivative of the horizontal coordinate of the center point of the target frame with respect to time, the first-order derivative of the vertical coordinate of the center point of the target frame with respect to time, and the first-order derivative of the area of ​​the target frame with respect to time. The second-order derivative of the position change may include the second-order derivative of the horizontal coordinate of the center point of the target frame with respect to time, the second-order derivative of the vertical coordinate of the center point of the target frame with respect to time, and the second-order derivative of the area of ​​the target frame with respect to time.

[0041] Exemplarily, the state vector of the Kalman filter designed in the embodiment of the present disclosure can be expressed as follows:

[0042] ;

[0043] in, Indicates the horizontal coordinate of the center point of the target frame, Indicates the ordinate of the center point of the target frame, represents the area of ​​the target box, Indicates the aspect ratio of the target box, Represents the first-order derivative of the horizontal coordinate of the center point of the target frame with respect to time, Represents the first-order derivative of the ordinate of the center point of the target frame with respect to time, represents the first-order derivative of the target box area with respect to time, Represents the second-order derivative of the horizontal coordinate of the center point of the target frame with respect to time, Represents the second-order derivative of the ordinate of the center point of the target frame with respect to time, Represents the second-order derivative of the target box’s area with respect to time.

[0044] S130, performing generalized intersection-union-correlation matching on the detected target frame of the target to be tracked in the current image frame and the predicted target frame of the target to be tracked in the current image frame to obtain a matching result of the detected target frame and the predicted target frame.

[0045] S140: When the matching result between the detected target frame and the predicted target frame satisfies a target tracking success condition, target tracking is performed on the next image frame.

[0046] The matching result of the detected target frame and the predicted target frame can be the generalized intersection over union (GIoU) ​​of the two. The generalized intersection over union (GIoU) ​​can be used to measure the degree of overlap between the detected target frame and the predicted target frame. The larger the value, the higher the overlap between the two, and vice versa.

[0047] Specifically, the target tracking success condition may be that the matching result between the detected target frame and the predicted target frame is greater than a preset threshold. For example, the preset threshold may be 0.4. If the matching result between the detected target frame and the predicted target frame is 0.8, it indicates that the target tracking is successful and the target tracking can be continued for the next image frame; if the matching result between the detected target frame and the predicted target frame is 0.2, it indicates that the target tracking fails.

[0048] The technical solution of the disclosed embodiment is to obtain the detection target frame of the target to be tracked in the current image frame, wherein the detection target frame is detected based on the target detection algorithm, and then obtain the predicted target frame of the target to be tracked in the current image frame, wherein the predicted target frame is predicted based on the Kalman filter, and the state vector of the Kalman filter includes the second-order derivative of the position change, and then perform generalized intersection and union correlation matching on the detection target frame of the target to be tracked in the current image frame and the predicted target frame of the target to be tracked in the current image frame to obtain the matching result of the detection target frame and the predicted target frame; when the matching result of the detection target frame and the predicted target frame meets the target tracking success condition, the target is tracked in the next image frame. The above technical solution, by introducing the second-order derivative of the position change, realizes the utilization of the rate of change of the target motion speed, can improve the accuracy of the predicted target frame, make the predicted target frame closer to the real target, and thus improve the target tracking accuracy.

[0049] Embodiment 2

[0050] Figure 2 A flow chart of a target tracking method provided for the second embodiment of the present disclosure, the method of this embodiment can be combined with the various optional schemes in the target tracking method provided in the above embodiments. The target tracking method provided in this embodiment is further optimized. Optionally, obtaining the predicted target frame of the target to be tracked in the current image frame includes: obtaining the estimated state vector of the previous image frame of the current image frame; performing Kalman filter prediction based on the estimated state vector of the previous image frame of the current image frame to obtain the predicted state vector of the current image frame; determining the predicted target frame of the target to be tracked in the current image frame based on the predicted state vector of the current image frame.

[0051] like Figure 2 As shown, the method includes:

[0052] S210, obtaining a detection target frame of a target to be tracked in a current image frame, wherein the detection target frame is detected based on a target detection algorithm.

[0053] S220: Obtain an estimated state vector of an image frame previous to the current image frame.

[0054] The previous image frame refers to the image frame processed at the previous moment before the current moment. The estimated state vector refers to the state vector predicted by the Kalman filter.

[0055] S230 , performing Kalman filter prediction based on the estimated state vector of the previous image frame of the current image frame to obtain a predicted state vector of the current image frame.

[0056] Specifically, the state vector of the estimated state of the previous image frame of the current image frame may be predicted by using the first preset state transfer function to obtain the predicted state vector of the current image frame.

[0057] Optionally, the first preset state transfer function is:

[0058] ;

[0059] ;

[0060] ;

[0061] ;

[0062] in, Indicates the actual horizontal coordinate of the center point of the target frame in the previous image frame. Indicates the actual vertical coordinate of the center point of the target frame in the previous image frame. Indicates the actual area of ​​the target box in the previous image frame. Indicates the actual aspect ratio of the target frame of the previous image frame. Represents the first-order derivative of the horizontal coordinate of the actual center point of the target frame of the previous image frame with respect to time, Represents the first-order derivative of the ordinate of the actual center point of the target frame of the previous image frame with respect to time, Represents the first-order derivative of the actual area of ​​the target box in the previous image frame with respect to time, Represents the second-order derivative of the horizontal coordinate of the actual center point of the target frame in the previous image frame with respect to time, Represents the second-order derivative of the ordinate of the actual center point of the target frame in the previous image frame with respect to time, Represents the second-order derivative of the actual area of ​​the target box with respect to time, represents the time step, , , and is random noise; Indicates the horizontal coordinate of the predicted center point of the target box in the current image frame, Indicates the ordinate of the predicted center point of the target box in the current image frame. Represents the predicted area of ​​the target box in the current image frame, Indicates the predicted aspect ratio of the target box in the current image frame.

[0063] S240: Determine a predicted target frame of the target to be tracked in the current image frame based on the predicted state vector of the current image frame.

[0064] In the disclosed embodiment, the predicted state vector includes the horizontal and vertical coordinates, area and aspect ratio of the center point of the target frame, and the predicted target frame can be calculated by the horizontal and vertical coordinates, area and aspect ratio of the center point of the target frame.

[0065] S250, performing generalized intersection-union-correlation matching on the detected target frame of the target to be tracked in the current image frame and the predicted target frame of the target to be tracked in the current image frame to obtain a matching result of the detected target frame and the predicted target frame.

[0066] S260: When the matching result between the detected target frame and the predicted target frame satisfies a target tracking success condition, perform target tracking on the next image frame.

[0067] On the basis of the above embodiment, optionally, after performing Kalman filtering prediction based on the state vector of the previous image frame of the current image frame, it also includes: obtaining the measured position information of the target to be tracked in the current image frame; based on the measured position information of the target to be tracked in the current image frame, performing Kalman filtering to update the predicted state vector of the current image frame to obtain the estimated state vector of the current image frame, and updating the predicted state vector of the current image frame through a second preset state transfer function.

[0068] The measured position information is the position information of the target obtained by the target detection algorithm.

[0069] Exemplarily, the Kalman gain may be calculated:

[0070] ;

[0071] in, represents the Kalman gain of the current image frame, represents the observation matrix, represents the measurement noise covariance matrix, Represents the prediction error covariance of the current image frame.

[0072] Furthermore, the Kalman gain is used to perform Kalman filtering update on the predicted state vector of the current image frame to obtain the estimated state vector of the current image frame. The formula is as follows:

[0073] ;

[0074] in, represents the estimated state vector of the current image frame, represents the predicted state vector of the current image frame, Indicates the measured position information of the target to be tracked in the current image frame.

[0075] Furthermore, the error covariance can be updated as follows:

[0076] ;

[0077] in, represents the estimated error covariance of the current image frame, Represents the identity matrix.

[0078] In the embodiment of the present disclosure, the second preset state transfer function is:

[0079] ;

[0080] ;

[0081] ;

[0082] ;

[0083] ;

[0084] ;

[0085] in, Indicates the actual horizontal coordinate of the center point of the target frame of the current image frame. Indicates the actual center point ordinate of the target frame of the current image frame. The actual area of ​​the target box of the current image frame, Represents the first-order derivative of the horizontal coordinate of the predicted center point of the target box in the current image frame with respect to time, Represents the first-order derivative of the ordinate of the predicted center point of the target frame of the current image frame with respect to time, Represents the first-order derivative of the predicted area of ​​the target box in the current image frame with respect to time; Represents the second-order derivative of the horizontal coordinate of the predicted center point of the target frame of the current image frame with respect to time, Represents the second-order derivative of the ordinate of the predicted center point of the target frame of the current image frame with respect to time, Represents the second-order derivative of the predicted area of ​​the target box in the current image frame with respect to time; , , , , and is random noise.

[0086] It should be noted that the first preset state transfer function is used to update The second preset state transfer function is used to update In other words, the prediction of the values ​​of all elements in the state vector of the present disclosure is completed in two stages, namely the Kalman filter prediction stage and the Kalman filter update stage.

[0087] The technical solution of the disclosed embodiment obtains the estimated state vector of the previous image frame of the current image frame, and then performs Kalman filter prediction based on the estimated state vector of the previous image frame of the current image frame to obtain the predicted state vector of the current image frame, and then determines the predicted target frame of the target to be tracked in the current image frame based on the predicted state vector of the current image frame, thereby realizing the automatic construction of the predicted target frame.

[0088] Embodiment 3

[0089] Figure 3 A flow chart of a target tracking method provided for the third embodiment of the present disclosure, the method of this embodiment can be combined with the various optional schemes in the target tracking method provided in the above embodiments. The target tracking method provided in this embodiment has been further optimized. Optionally, after obtaining the matching result of the detection target frame and the prediction target frame, it also includes: when the matching result of the detection target frame and the prediction target frame meets the target tracking failure condition, performing appearance feature extraction on the current image frame to obtain the appearance features of all targets in the current image frame; performing similarity matching on the appearance features of all targets in a historical appearance feature library to obtain tracking targets, wherein the historical appearance feature library includes the appearance features of the tracking targets in the historical target tracking process.

[0090] like Figure 3 As shown, the method includes:

[0091] S310: Acquire a detection target frame of a target to be tracked in a current image frame, wherein the detection target frame is detected based on a target detection algorithm.

[0092] S320, obtaining a predicted target frame of the target to be tracked in the current image frame, wherein the predicted target frame is obtained based on a Kalman filter prediction, and a state vector of the Kalman filter includes a second-order derivative of position change.

[0093] S330, performing generalized intersection-union-correlation matching on the detected target frame of the target to be tracked in the current image frame and the predicted target frame of the target to be tracked in the current image frame to obtain a matching result of the detected target frame and the predicted target frame.

[0094] S340: When the matching result between the detected target frame and the predicted target frame satisfies a target tracking success condition, target tracking is performed on the next image frame.

[0095] S350. When the matching result between the detected target frame and the predicted target frame satisfies the target tracking failure condition, extract the appearance features of the current image frame to obtain the appearance features of all targets in the current image frame.

[0096] S360: Perform similarity matching on the appearance features of all the targets in a historical appearance feature library to obtain a tracking target, wherein the historical appearance feature library includes the appearance features of the tracking targets in the historical target tracking process.

[0097] When the target is temporarily blocked by other objects, there is no new measured position information to update the estimated state vector. In addition, in actual scenarios, noise accumulates over time, resulting in huge deviations in the predicted state vector after the target reappears, often leading to tracking failure or loss.

[0098] To this end, in the event of target tracking failure, the embodiment of the present disclosure extracts appearance features of the current image frame to obtain the appearance features of all targets in the current image frame, and then performs similarity matching on the appearance features of all targets in a historical appearance feature library to obtain the tracking target, that is, to retrieve the target and continue tracking, thereby effectively improving the tracking effect when the target is obscured.

[0099] The appearance feature refers to a feature visible outside the target, such as a physical appearance feature or a clothing feature, etc. In the disclosed embodiment, the appearance feature can be extracted by an appearance feature extractor, and the appearance feature extractor can be a deep neural network, such as a MobileNet or other trained convolutional neural networks, etc., which are not limited here.

[0100] Exemplarily, the target tracking failure condition may be that the matching result between the detected target frame and the predicted target frame is less than a preset threshold. Specifically, in the case of target tracking failure, multiple targets in the current image frame are obtained through the target detection algorithm, and the appearance features of each target are obtained through the appearance feature extractor, and the appearance features of all targets are similarly matched with the appearance features of the tracked targets in the historical target tracking process to retrieve the tracked target. Similarity matching can be achieved by calculating the Euclidean distance or cosine similarity between the appearance features, etc., which is not specifically limited here. After retrieving the tracked target, the previous trajectory can be cleared, and the state vector of the Kalman filter can be initialized to re-track the target.

[0101] The technical solution of the disclosed embodiment is to extract the appearance features of the current image frame when the matching result of the detected target frame and the predicted target frame meets the target tracking failure condition, so as to obtain the appearance features of all targets in the current image frame, and then perform similarity matching on the appearance features of all targets in the historical appearance feature library, so as to retrieve the target and continue tracking, thereby effectively improving the tracking effect when the target is obscured.

[0102] Embodiment 4

[0103] Figure 4This is a flow chart of a target tracking method provided in the fourth embodiment of the present disclosure. The method in this embodiment is a preferred example of the above embodiment. Figure 4 As shown, the method includes:

[0104] The first step is to initialize the state vector of the Kalman filter.

[0105] Specifically, the state vector of the Kalman filter can be initialized according to the measured position information of the target. For example, the state vector of the Kalman filter can be obtained from the measured position information. . The initial value of can be 0.

[0106] The second step is to perform target detection on the image frame.

[0107] Specifically, the robot in the indoor scene can collect image frames through a camera and perform target detection on the image frames to obtain a detection target frame of the target to be tracked. For example, the robot in the indoor scene can be a robot for guiding services or a security monitoring robot, etc., which is not limited here.

[0108] The third step is to crop the image frame and extract the appearance features of the target.

[0109] Specifically, the image frame can be cropped according to the detection target frame of the target to be tracked, and the cropped image is input into the appearance feature extractor to obtain the appearance features of the target and save them in the historical appearance feature library.

[0110] The fourth step is Kalman filter prediction.

[0111] Specifically, Kalman filter prediction is performed based on the estimated state vector of the previous image frame of the current image frame to obtain the predicted state vector of the current image frame, and then the predicted target frame of the target to be tracked in the current image frame is determined based on the predicted state vector of the current image frame.

[0112] Step 5: GIoU association matching.

[0113] Specifically, a generalized intersection-union-correlation matching is performed on the detection target frame of the target to be tracked in the current image frame and the predicted target frame of the target to be tracked in the current image frame to obtain a matching result of the detection target frame and the predicted target frame.

[0114] Step 6: Kalman filter update.

[0115] Specifically, the measured position information of the target to be tracked in the current image frame is obtained; based on the measured position information of the target to be tracked in the current image frame, the predicted state vector of the current image frame is updated by Kalman filtering to obtain the estimated state vector of the current image frame.

[0116] Step 7. If target tracking fails, execute step 8. If target tracking succeeds, return to step 2.

[0117] Step 8: Feature similarity matching.

[0118] Specifically, in the case of target tracking failure, multiple targets in the current image frame are obtained through the target detection algorithm, and the appearance features of each target are obtained through the appearance feature extractor. The appearance features of all targets are similarly matched with the appearance features of the tracked targets in the historical target tracking process to retrieve the tracked target.

[0119] Step 9. Clear the existing tracks and return to step 1.

[0120] Specifically, after the tracking target is retrieved, the previous track can be cleared, and the state vector of the Kalman filter can be initialized to re-track the target.

[0121] The technical solution of the disclosed embodiment, by introducing the second-order derivative of position change, realizes the utilization of the rate of change of the target motion speed, which can improve the accuracy of the predicted target frame, make the predicted target frame closer to the real target, and thus improve the target tracking accuracy. In addition, in the case of target tracking failure, the appearance feature similarity matching is used to retrieve the tracking target and automatically clear the tracking record, and restart new tracking, which improves the continuity and accuracy of tracking.

[0122] Embodiment 5

[0123] Figure 5 This is a schematic diagram of the structure of a target tracking device provided in Embodiment 5 of the present disclosure. Figure 5 As shown, the device comprises:

[0124] A detection target frame acquisition module 510 is used to acquire a detection target frame of a target to be tracked in a current image frame, wherein the detection target frame is detected based on a target detection algorithm;

[0125] A predicted target frame acquisition module 520 is used to acquire a predicted target frame of a target to be tracked in a current image frame, wherein the predicted target frame is obtained based on a Kalman filter prediction, and a state vector of the Kalman filter includes a second-order derivative of a position change;

[0126] A generalized intersection-and-union (IOR) correlation matching module 530 is used to perform generalized intersection-and-union (IOR) correlation matching on a detected target frame of the target to be tracked in the current image frame and a predicted target frame of the target to be tracked in the current image frame to obtain a matching result between the detected target frame and the predicted target frame;

[0127] The target tracking success judgment module 540 is used to perform target tracking on the next image frame when the matching result between the detected target frame and the predicted target frame meets the target tracking success condition.

[0128] The technical solution of the disclosed embodiment is to obtain the detection target frame of the target to be tracked in the current image frame, wherein the detection target frame is detected based on the target detection algorithm, and then obtain the predicted target frame of the target to be tracked in the current image frame, wherein the predicted target frame is predicted based on the Kalman filter, and the state vector of the Kalman filter includes the second-order derivative of the position change, and then perform generalized intersection and union correlation matching on the detection target frame of the target to be tracked in the current image frame and the predicted target frame of the target to be tracked in the current image frame to obtain the matching result of the detection target frame and the predicted target frame; when the matching result of the detection target frame and the predicted target frame meets the target tracking success condition, the target is tracked in the next image frame. The above technical solution, by introducing the second-order derivative of the position change, realizes the utilization of the rate of change of the target motion speed, can improve the accuracy of the predicted target frame, make the predicted target frame closer to the real target, and thus improve the target tracking accuracy.

[0129] In some optional embodiments, the second-order derivative of the position change includes the second-order derivative of the horizontal coordinate of the center point of the target frame with respect to time, the second-order derivative of the vertical coordinate of the center point of the target frame with respect to time, and the second-order derivative of the area of ​​the target frame with respect to time.

[0130] In some optional implementations, the predicted target frame acquisition module 520 includes:

[0131] The estimated state vector acquisition unit of the previous image frame is used to acquire the estimated state vector of the previous image frame of the current image frame;

[0132] A Kalman filter prediction unit, configured to perform Kalman filter prediction based on an estimated state vector of an image frame previous to the current image frame, to obtain a predicted state vector of the current image frame;

[0133] A predicted target frame determining unit is used to determine a predicted target frame of a target to be tracked in a current image frame based on the predicted state vector of the current image frame.

[0134] In some optional implementations, the Kalman filter prediction unit may be specifically used for:

[0135] The estimated state vector of the previous image frame of the current image frame is predicted by using a first preset state transfer function to obtain a predicted state vector of the current image frame.

[0136] In some optional implementations, the first preset state transfer function is:

[0137] ;

[0138] ;

[0139] ;

[0140] ;

[0141] in, Indicates the actual horizontal coordinate of the center point of the target frame in the previous image frame. Indicates the actual vertical coordinate of the center point of the target frame in the previous image frame. Indicates the actual area of ​​the target box in the previous image frame. Indicates the actual aspect ratio of the target frame of the previous image frame. Represents the first-order derivative of the horizontal coordinate of the actual center point of the target frame of the previous image frame with respect to time, Represents the first-order derivative of the ordinate of the actual center point of the target frame of the previous image frame with respect to time, Represents the first-order derivative of the actual area of ​​the target box in the previous image frame with respect to time, Represents the second-order derivative of the horizontal coordinate of the actual center point of the target frame in the previous image frame with respect to time, Represents the second-order derivative of the ordinate of the actual center point of the target frame in the previous image frame with respect to time, Represents the second-order derivative of the actual area of ​​the target box with respect to time, represents the time step, , , and is random noise; Indicates the horizontal coordinate of the predicted center point of the target box in the current image frame, Indicates the ordinate of the predicted center point of the target box in the current image frame. Represents the predicted area of ​​the target box in the current image frame, Indicates the predicted aspect ratio of the target box in the current image frame.

[0142] In some optional implementations, the target tracking device further includes:

[0143] A Kalman filter update module is used to obtain the measured position information of the target to be tracked in the current image frame; based on the measured position information of the target to be tracked in the current image frame, perform Kalman filter update on the predicted state vector of the current image frame to obtain the estimated state vector of the current image frame; and update the predicted state vector of the current image frame by a second preset state transfer function, wherein the second preset state transfer function is:

[0144] ;

[0145] ;

[0146] ;

[0147] ;

[0148] ;

[0149] ;

[0150] in, Indicates the actual horizontal coordinate of the center point of the target frame of the current image frame. Indicates the actual center point ordinate of the target frame of the current image frame. The actual area of ​​the target box of the current image frame, Represents the first-order derivative of the horizontal coordinate of the predicted center point of the target box in the current image frame with respect to time, Represents the first-order derivative of the ordinate of the predicted center point of the target frame of the current image frame with respect to time, Represents the first-order derivative of the predicted area of ​​the target box in the current image frame with respect to time; Represents the second-order derivative of the horizontal coordinate of the predicted center point of the target frame of the current image frame with respect to time, Represents the second-order derivative of the ordinate of the predicted center point of the target frame of the current image frame with respect to time, Represents the second-order derivative of the predicted area of ​​the target box in the current image frame with respect to time; , , , , and is random noise.

[0151] In some optional implementations, the target tracking device further includes:

[0152] The tracking target matching module is used to extract the appearance features of the current image frame when the matching result between the detected target frame and the predicted target frame meets the target tracking failure condition, so as to obtain the appearance features of all targets in the current image frame; and perform similarity matching on the appearance features of all targets in a historical appearance feature library to obtain the tracking target, wherein the historical appearance feature library includes the appearance features of the tracking targets in the historical target tracking process.

[0153] The target tracking device provided in the embodiments of the present disclosure can execute the target tracking method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0154] Embodiment 6

[0155] Figure 6 A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0156] like Figure 6 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The I / O interface 15 is also connected to the bus 14.

[0157] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0158] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a target tracking method, which includes:

[0159] Obtaining a detection target frame of a target to be tracked in a current image frame, wherein the detection target frame is detected based on a target detection algorithm;

[0160] Acquire a predicted target frame of a target to be tracked in a current image frame, wherein the predicted target frame is obtained based on a Kalman filter prediction, and a state vector of the Kalman filter includes a second-order derivative of position change;

[0161] Performing generalized intersection-union-correlation matching on a detection target frame of the target to be tracked in the current image frame and a prediction target frame of the target to be tracked in the current image frame to obtain a matching result between the detection target frame and the prediction target frame;

[0162] When the matching result between the detected target frame and the predicted target frame satisfies the target tracking success condition, target tracking is performed on the next image frame.

[0163] In some embodiments, the target tracking method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the target tracking method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the target tracking method in any other appropriate manner (e.g., by means of firmware).

[0164] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0165] Computer programs for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0166] In the context of the present disclosure, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0167] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0168] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0169] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0170] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of this disclosure can be achieved, and this document is not limited here.

[0171] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A target tracking method, characterized in that: include: Obtaining a detection target frame of a target to be tracked in a current image frame, wherein the detection target frame is detected based on a target detection algorithm; Acquire a predicted target frame of a target to be tracked in a current image frame, wherein the predicted target frame is obtained based on a Kalman filter prediction, and a state vector of the Kalman filter includes a second-order derivative of position change; Performing generalized intersection-union-correlation matching on a detection target frame of the target to be tracked in the current image frame and a prediction target frame of the target to be tracked in the current image frame to obtain a matching result between the detection target frame and the prediction target frame; When the matching result between the detected target frame and the predicted target frame satisfies the target tracking success condition, target tracking is performed on the next image frame.

2. The target tracking method according to claim 1, characterized in that: The second-order derivative of the position change includes the second-order derivative of the horizontal coordinate of the center point of the target frame with respect to time, the second-order derivative of the vertical coordinate of the center point of the target frame with respect to time, and the second-order derivative of the area of ​​the target frame with respect to time.

3. The target tracking method according to claim 1, characterized in that: The step of obtaining a predicted target frame of a target to be tracked in a current image frame includes: Obtaining an estimated state vector of the previous image frame of the current image frame; Performing Kalman filter prediction based on the estimated state vector of the previous image frame of the current image frame to obtain a predicted state vector of the current image frame; A predicted target frame of a target to be tracked in the current image frame is determined based on the predicted state vector of the current image frame.

4. The target tracking method according to claim 3, characterized in that: The performing Kalman filter prediction based on the estimated state vector of the previous image frame of the current image frame to obtain the predicted state vector of the current image frame includes: The estimated state vector of the previous image frame of the current image frame is predicted by using a first preset state transfer function to obtain a predicted state vector of the current image frame.

5. The target tracking method according to claim 4, characterized in that: The first preset state transfer function is: ； ； ； ； in, Indicates the actual horizontal coordinate of the center point of the target frame in the previous image frame. Indicates the actual vertical coordinate of the center point of the target frame in the previous image frame. Indicates the actual area of ​​the target box in the previous image frame. Indicates the actual aspect ratio of the target frame of the previous image frame. Represents the first-order derivative of the horizontal coordinate of the actual center point of the target frame of the previous image frame with respect to time, Represents the first-order derivative of the ordinate of the actual center point of the target frame of the previous image frame with respect to time, Represents the first-order derivative of the actual area of ​​the target box in the previous image frame with respect to time, Represents the second-order derivative of the horizontal coordinate of the actual center point of the target frame in the previous image frame with respect to time, Represents the second-order derivative of the ordinate of the actual center point of the target frame in the previous image frame with respect to time, Represents the second-order derivative of the actual area of ​​the target box with respect to time, represents the time step, , , and is random noise; Indicates the horizontal coordinate of the predicted center point of the target box in the current image frame, Indicates the ordinate of the predicted center point of the target box in the current image frame. Represents the predicted area of ​​the target box in the current image frame, Indicates the predicted aspect ratio of the target box in the current image frame.

6. The target tracking method according to claim 5, characterized in that: After performing Kalman filter prediction based on the state vector of the previous image frame of the current image frame, the method further includes: Obtaining the measured position information of the target to be tracked in the current image frame; Based on the measured position information of the target to be tracked in the current image frame, performing Kalman filtering update on the predicted state vector of the current image frame to obtain an estimated state vector of the current image frame; The predicted state vector of the current image frame is updated by a second preset state transfer function, wherein the second preset state transfer function is: ； ； ； ； ； ； in, Indicates the actual horizontal coordinate of the center point of the target frame of the current image frame. Indicates the actual center point ordinate of the target frame of the current image frame. The actual area of ​​the target box of the current image frame, Represents the first-order derivative of the horizontal coordinate of the predicted center point of the target box in the current image frame with respect to time, Represents the first-order derivative of the ordinate of the predicted center point of the target frame of the current image frame with respect to time, Represents the first-order derivative of the predicted area of ​​the target box in the current image frame with respect to time; Represents the second-order derivative of the horizontal coordinate of the predicted center point of the target frame of the current image frame with respect to time, Represents the second-order derivative of the ordinate of the predicted center point of the target frame of the current image frame with respect to time, Represents the second-order derivative of the predicted area of ​​the target box in the current image frame with respect to time; , , , , and is random noise.

7. The target tracking method according to any one of claims 1 to 6, characterized in that: After obtaining the matching result between the detected target frame and the predicted target frame, it also includes: When the matching result between the detected target frame and the predicted target frame satisfies the target tracking failure condition, performing appearance feature extraction on the current image frame to obtain appearance features of all targets in the current image frame; The appearance features of all the targets are matched for similarity in a historical appearance feature library to obtain a tracking target, wherein the historical appearance feature library includes the appearance features of the tracking target in the historical target tracking process.

8. A target tracking device, characterized in that: include: A detection target frame acquisition module is used to acquire a detection target frame of a target to be tracked in a current image frame, wherein the detection target frame is detected based on a target detection algorithm; A predicted target frame acquisition module is used to acquire a predicted target frame of a target to be tracked in a current image frame, wherein the predicted target frame is obtained based on a Kalman filter prediction, and a state vector of the Kalman filter includes a second-order derivative of a position change; A generalized intersection-and-union ratio association matching module is used to perform generalized intersection-and-union ratio association matching on a detection target frame of a target to be tracked in the current image frame and a predicted target frame of a target to be tracked in the current image frame to obtain a matching result between the detection target frame and the predicted target frame; The target tracking success judgment module is used to perform target tracking on the next image frame when the matching result between the detected target frame and the predicted target frame meets the target tracking success condition.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the target tracking method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the target tracking method according to any one of claims 1 to 7 when executed.

Images