Target tracking method, device and storage medium

Abstract

The present disclosure relates to a target tracking method, device and storage medium, the method comprising: acquiring an image acquisition sequence; performing object detection on the first image in the image acquisition sequence, obtaining a 3D detection frame of the object on the first image in the target three-dimensional space and a 2D detection frame on the first image, the first image being any image in the image acquisition sequence except the first image; predicting the 3D prediction frame of the target object in the target three-dimensional space and the 2D prediction frame on the first image according to the tracking result of tracking the target object for the second image, the second image being the previous image of the first image in the image acquisition sequence; determining the tracking result of tracking the target object for the first image according to the 3D detection frame, the 2D detection frame, the 3D prediction frame and the 2D prediction frame. The target tracking method disclosed in the present disclosure can improve the tracking accuracy of the target object.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a target tracking method, device and storage medium. Background Art

[0002] Object tracking is to find the object in a given image sequence, identify the same object in different frames, and assign an ID to the same object in different frames. In related technologies, object tracking is usually performed based on 2D information. However, it is difficult to accurately estimate the motion of the object using only 2D information, resulting in incorrect tracking. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a target tracking method, device and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a target tracking method is provided, including:

[0005] Acquire an image acquisition sequence, wherein the image acquisition sequence is obtained based on images acquired by an image acquisition device at multiple acquisition moments;

[0006] Performing object detection on a first image in the image acquisition sequence to obtain a 3D detection frame of the object on the first image in a target three-dimensional space and a 2D detection frame on the first image, wherein the first image is any image in the image acquisition sequence except the first image;

[0007] Predicting a 3D prediction box of the target object in the target three-dimensional space and a 2D prediction box of the target object on the first image according to a tracking result of tracking the target object in a second image, wherein the second image is a previous image of the first image in the image acquisition sequence;

[0008] A tracking result of tracking the target object for the first image is determined according to the 3D detection frame, the 2D detection frame, the 3D prediction frame, and the 2D prediction frame.

[0009] In some embodiments, determining a tracking result of tracking the target object for the first image according to the 3D detection frame, the 2D detection frame, the 3D prediction frame, and the 2D prediction frame includes:

[0010] Determining, from each of the 3D detection frames, a target 3D detection frame that matches the 3D prediction frame according to a first intersection-over-union ratio and / or a distance value between the 3D prediction frame and each of the 3D detection frames;

[0011] An object corresponding to the target 3D detection frame in the target three-dimensional space is used as the target object, and position information of the target 3D detection frame in the target three-dimensional space is used as 3D tracking position information of the target object, and the tracking result includes the 3D tracking position information.

[0012] In some embodiments, determining a tracking result of tracking the target object for the first image according to the 3D detection frame, the 2D detection frame, the 3D prediction frame, and the 2D prediction frame further includes:

[0013] For a 3D detection frame in each of the 3D detection frames that is not matched to the 3D prediction frame, determining a 2D detection frame corresponding to the same object as the 3D detection frame, and determining a target 2D detection frame that matches the 2D prediction frame according to a second intersection-and-union ratio between the 2D detection frame and the 2D prediction frame;

[0014] An object corresponding to the target 2D detection frame on the first image is used as the target object, and position information of the target 2D detection frame on the first image is used as 2D tracking position information of the target object, and the tracking result includes the 2D tracking position information.

[0015] In some embodiments, the tracking result includes 3D tracking position information of the target object corresponding to the second image, 2D tracking position information, and motion data of the target object;

[0016] The step of predicting a 3D prediction box of the target object in the three-dimensional space and a 2D prediction box of the target object on the first image according to a tracking result of tracking the target object on the second image includes:

[0017] updating a tracker according to the motion data;

[0018] The 3D tracking position information and the 2D tracking position information are input into the updated tracker to obtain the 3D prediction frame and the 2D prediction frame output by the tracker.

[0019] In some embodiments, the motion data includes a rate of change of position of the target object on the image, and a speed and acceleration of the target object in the target three-dimensional space;

[0020] The tracker can output the 2D prediction frame based on the position change rate and the 2D tracking position information, and output the 3D prediction frame based on the speed, the acceleration, and the 3D tracking position information.

[0021] In some embodiments, the first image includes a plurality of acquired images, and the plurality of acquired images are images acquired by a plurality of image acquisition devices at the same acquisition time;

[0022] The performing object detection on the first image in the image acquisition sequence to obtain a 3D detection frame of the object on the first image in the target three-dimensional space and a 2D detection frame on the first image includes:

[0023] Performing object detection on the multiple acquired images to obtain a 3D detection frame of the object on each acquired image in the three-dimensional space of each image acquisition device and a 2D detection frame on the acquired image;

[0024] Mapping 3D detection frames located in the three-dimensional space of different image acquisition devices to the same target coordinate system according to the external parameters of each of the image acquisition devices, wherein the target three-dimensional space is a space defined by the target coordinate system;

[0025] The multiple acquired images are stitched together to obtain a stitched image, and the 2D detection frames on the multiple acquired images are mapped to the stitched image, and the 2D detection frame on the first image is the 2D detection frame on the stitched image.

[0026] In some embodiments, before determining a tracking result of tracking the target object for the first image according to the 3D detection frame, the 2D detection frame, the 3D prediction frame, and the 2D prediction frame, the method further includes:

[0027] Non-maximum suppression processing is performed on the 3D detection frame, the 2D detection frame, the 3D prediction frame and the 2D prediction frame.

[0028] According to a second aspect of an embodiment of the present disclosure, there is provided a target tracking device, comprising:

[0029] An acquisition module is configured to acquire an image acquisition sequence, wherein the image acquisition sequence is obtained based on images acquired by an image acquisition device at multiple acquisition moments;

[0030] a detection module configured to perform object detection on a first image in the image acquisition sequence to obtain a 3D detection frame of the object on the first image in a target three-dimensional space and a 2D detection frame on the first image, wherein the first image is any image in the image acquisition sequence except the first image;

[0031] a prediction module configured to predict a 3D prediction box of the target object in the target three-dimensional space and a 2D prediction box on the first image according to a tracking result of tracking the target object in a second image, wherein the second image is a previous image of the first image in the image acquisition sequence;

[0032] A determination module is configured to determine a tracking result of tracking the target object for the first image according to the 3D detection frame, the 2D detection frame, the 3D prediction frame and the 2D prediction frame.

[0033] In some embodiments, the determining module is further configured to:

[0034] Determining, from each of the 3D detection frames, a target 3D detection frame that matches the 3D prediction frame according to a first intersection-over-union ratio and / or a distance value between the 3D prediction frame and each of the 3D detection frames;

[0035] An object corresponding to the target 3D detection frame in the target three-dimensional space is used as the target object, and position information of the target 3D detection frame in the target three-dimensional space is used as 3D tracking position information of the target object, and the tracking result includes the 3D tracking position information.

[0036] In some embodiments, the determining module is further configured to:

[0037] For a 3D detection frame in each of the 3D detection frames that is not matched to the 3D prediction frame, determining a 2D detection frame corresponding to the same object as the 3D detection frame, and determining a target 2D detection frame that matches the 2D prediction frame according to a second intersection-and-union ratio between the 2D detection frame and the 2D prediction frame;

[0038] An object corresponding to the target 2D detection frame on the first image is used as the target object, and position information of the target 2D detection frame on the first image is used as 2D tracking position information of the target object, and the tracking result includes the 2D tracking position information.

[0039] In some embodiments, the tracking result includes 3D tracking position information of the target object corresponding to the second image, 2D tracking position information, and motion data of the target object;

[0040] The prediction module is further configured to:

[0041] updating a tracker according to the motion data;

[0042] The 3D tracking position information and the 2D tracking position information are input into the updated tracker to obtain the 3D prediction frame and the 2D prediction frame output by the tracker.

[0043] In some embodiments, the motion data includes a rate of change of position of the target object on the image, and a speed and acceleration of the target object in the target three-dimensional space;

[0044] The tracker can output the 2D prediction frame based on the position change rate and the 2D tracking position information, and output the 3D prediction frame based on the speed, the acceleration, and the 3D tracking position information.

[0045] In some embodiments, the first image includes a plurality of acquired images, and the plurality of acquired images are images acquired by a plurality of image acquisition devices at the same acquisition time;

[0046] The detection module is further configured to:

[0047] Performing object detection on the multiple acquired images to obtain a 3D detection frame of the object on each acquired image in the three-dimensional space of each image acquisition device and a 2D detection frame on the acquired image;

[0048] Mapping 3D detection frames located in three-dimensional spaces of different image acquisition devices to the same target coordinate system according to the external parameters of each of the image acquisition devices, wherein the target three-dimensional space is a space defined by the target coordinate system;

[0049] The multiple acquired images are stitched together to obtain a stitched image, and the 2D detection frames on the multiple acquired images are mapped to the stitched image, and the 2D detection frame on the first image is the 2D detection frame on the stitched image.

[0050] In some embodiments, the determining module is further configured to:

[0051] Non-maximum suppression processing is performed on the 3D detection frame, the 2D detection frame, the 3D prediction frame and the 2D prediction frame.

[0052] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the target tracking method provided in the first aspect of the present disclosure are implemented.

[0053] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: determining the tracking result of the target object through 3D frame information (e.g., 3D detection frame and 3D prediction frame) and 2D frame information (e.g., 2D detection frame and 2D prediction frame), introducing 3D frame information, so that the target tracking method of the present disclosure can continue to estimate the motion of the target object in the target three-dimensional space for a period of time after the target object is lost, thereby improving the probability of successful matching after the target object reappears, reducing ID switching caused by missed detection or out-of-view of the target, that is, reducing the situation of erroneous tracking of the target. At the same time, combining 3D frame information and 2D frame information to determine the tracking result of the target object can improve the tracking accuracy of the target object.

[0054] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0056] Figure 1 The present invention is a flowchart of a target tracking method according to an exemplary embodiment.

[0057] Figure 2 The figure is a flowchart of determining a 3D detection frame and a 2D detection frame according to an exemplary embodiment.

[0058] Figure 3 The invention is a block diagram of a target tracking device according to an exemplary embodiment.

[0059] Figure 4 It is a block diagram of a device for target tracking according to an exemplary embodiment.

[0060] Figure 5 It is a block diagram of a device for target tracking according to an exemplary embodiment. DETAILED DESCRIPTION

[0061] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0062] In some embodiments, the target tracking method of the present disclosure can be applied to different scenarios. For example, it can be applied to an autonomous driving scenario to track targets in images captured by an image acquisition device on the vehicle. For another example, it can be applied to a traffic monitoring scenario to track targets in images captured by an image acquisition device in a traffic monitoring system. It should be understood that the application scenarios of the target tracking method mentioned in the present disclosure are only some examples or embodiments of the present disclosure. For ordinary technicians in this field, the target tracking method can also be applied to other similar scenarios without creative work. For example, it can also be applied to target tracking of mobile robots, and the present disclosure does not limit this.

[0063] In the related art, 2D information is usually used to track targets in images captured by one or more image acquisition devices. However, due to the affine transformation when the target is projected onto the 2D image, it is difficult to accurately estimate the motion of the target on the 2D image. As a result, the target cannot be accurately tracked using 2D information, resulting in the wrong ID being matched to the target. When using 2D information to track a target, once the target is lost, it is difficult to find it back. In addition, in the related art, images captured by multiple image acquisition devices are usually tracked and processed separately, which is not only inefficient, but also unable to process overlapping targets in images of adjacent image acquisition devices.

[0064] Figure 1 is a flow chart of a target tracking method according to an exemplary embodiment. Figure 1 As shown, the method includes the following steps.

[0065] Step 110, acquiring an image acquisition sequence, where the image acquisition sequence is obtained based on images acquired by an image acquisition device at multiple acquisition moments.

[0066] In some embodiments, the image acquisition sequence may be obtained based on the images acquired by one or more image acquisition devices at multiple acquisition moments. For one image acquisition device, the acquired image at each acquisition moment in the image acquisition sequence may be the acquired image of the image acquisition device at that acquisition moment; for multiple image acquisition devices, the acquired image at each acquisition moment in the image acquisition sequence may be the acquired images of the multiple image acquisition devices at that acquisition moment.

[0067] For example, if an image acquisition device is image acquisition device 1, and the images acquired by the image acquisition device 1 at multiple acquisition times t1, t2, and t3 are P1, P2, and P3, then the image acquisition sequence 1 may be (P1, P2, P3). 11 , P 12, P 13 , the captured images of the image acquisition device 2 at multiple capture times t1, t2 and t3 are P 21 , P 22 , P 23 , the captured images of the image acquisition device 3 at multiple acquisition times t1, t2 and t3 are P 31 , P 32 , P 33 For example, the image acquisition sequence 2 may be (P 11 P 21 P 11 , P 12 P 22 P 32 , P 13 P 23 P 33 ).

[0068] In some embodiments, the image acquisition device may include but is not limited to a video camera and a camera. The image acquisition device may be set at a preset fixed position or in a mobile device, and the preset fixed position and the mobile device may be specifically set according to actual needs. For example, the mobile device may be an autonomous driving vehicle. In some embodiments, the image acquisition device may be one or more cameras included in the autonomous driving vehicle.

[0069] In some embodiments, when there are multiple image acquisition devices, the acquisition directions of the multiple image acquisition devices may be different. For example, still taking the multiple image acquisition devices as image acquisition devices 1-3 as an example, the acquisition directions of image acquisition devices 1-3 may be left, forward, right, etc. It is worth noting that the acquisition directions of the multiple image acquisition devices may be specifically set according to actual conditions, and the present disclosure does not impose any restrictions on this.

[0070] In some embodiments, an image acquisition sequence may be acquired based on a video acquired by one or more image acquisition devices. For example, the acquired images in the image acquisition sequence may be image frames included in the video.

[0071] Step 120, performing object detection on the first image in the image acquisition sequence to obtain a 3D detection frame of the object on the first image in the target three-dimensional space and a 2D detection frame on the first image, wherein the first image is any image in the image acquisition sequence except the first image.

[0072] In some embodiments, the object on the first image may refer to one or more targets included in the first image, and the object may include targets of different categories. For example, taking the first image as a road condition image, the object on the first image may include targets of pedestrian category and targets of vehicle category, etc.

[0073] In some embodiments, object detection may be performed on the first image according to a monocular 3D detection algorithm. In some embodiments, the monocular 3D detection algorithm may include but is not limited to a fully convolutional one-stage 3D monocular target detection method (FullyConvolutional One-Stage Monocular 3D Object Detection, FCOS 3D) and a real-time monocular 3D target detection algorithm (Real-time Monocular 3D Object Detection, RTM 3D).

[0074] The monocular 3D detection algorithm can simultaneously obtain the 3D detection frame of the object included in the image in the three-dimensional space of the image acquisition device (for example, the camera coordinate system) and the 2D detection frame in the image coordinate system of the image. In some embodiments, the 3D detection frame in the three-dimensional space of the image acquisition device obtained by the detection algorithm can be represented by (x, y, z, rot, w, h, l), where (x, y, z) can represent the coordinates of the center point of the 3D detection frame in the three-dimensional space of the image acquisition device, rot can represent the heading angle of the 3D detection frame, and (w, h, l) can represent the width, height and length of the 3D detection frame respectively. The 2D detection frame obtained by the detection algorithm can be represented by (x1, y1, x2, y2), where (x1, y1) can represent the coordinates of the upper left corner of the 2D detection frame in the image coordinate system, and (x2, y2) can represent the coordinates of the lower right corner of the 2D detection frame in the image coordinate system.

[0075] As mentioned above, the image acquisition device may include one or more. For an image acquisition device, the first image may be an acquisition image. For example, still taking the above-mentioned image acquisition sequence 1 as an example, the first image may be P2 or P3. In some embodiments, when the first image is an acquisition image, object detection may be performed on the acquisition image to obtain a 3D detection frame of the object on the acquisition image in the three-dimensional space of the image acquisition device, as well as a 2D detection frame on the acquisition image. The 2D detection frame on the acquisition image is the 2D detection frame in the image coordinate system of the acquisition image. In some embodiments, when the first image is an acquisition image, the 3D detection frame in the three-dimensional space of the image acquisition device may be determined as the 3D detection frame in the target three-dimensional space, or the 3D detection frame in the three-dimensional space of the image acquisition device may be mapped to the target coordinate system, and the 3D detection frame in the target coordinate system may be determined as the 3D detection frame in the target three-dimensional space. For specific details about the target coordinate system, see. Figure 2 And the related description will not be repeated here.

[0076] For multiple image acquisition devices, the first image may include multiple acquired images, and the multiple acquired images may be images acquired by multiple image acquisition devices at the same acquisition time. For example, still taking the above-mentioned image acquisition sequence 2 as an example, the first image may include P 12 , P 22 and P 32 , or including P 13 , P 23 and P 33 . In some embodiments, when the first image is a plurality of acquired images, object detection may be performed on the plurality of acquired images to obtain a 3D detection frame of the object on each acquired image in the three-dimensional space of each image acquisition device and a 2D detection frame on the acquired image. In some embodiments, the 3D detection frame in the three-dimensional space of each image acquisition device and the 2D detection frame on each acquired image may be processed separately to obtain a 3D detection frame in the target three-dimensional space and a 2D detection frame on the first image. For specific details on obtaining a 3D detection frame and a 2D detection frame when the first image is a plurality of acquired images, see Figure 2 And the related description will not be repeated here.

[0077] Step 130, predicting a 3D prediction box of the target object in the target three-dimensional space and a 2D prediction box on the first image according to the tracking result of the target object in the second image, wherein the second image is the previous image of the first image in the image acquisition sequence.

[0078] In some embodiments, the first image and the second image may be images captured by an image capture device at different capture times, wherein the second image is an image obtained by the image capture device at a previous capture time of the first image, that is, the second image is the previous image of the first image in the image capture sequence.

[0079] In some embodiments, the target object may be one or more targets included in the second image, and the target object may include targets of different categories. For example, targets of pedestrian category and targets of vehicle category, etc. In some embodiments, the tracking result of tracking the target object for the second image may include 3D tracking position information of the target object corresponding to the second image, 2D tracking position information, and motion data of the target object. For the determination method of 3D tracking position information and 2D tracking position information, please refer to the following step 140 and its related description, which will not be repeated here.

[0080] In some embodiments, the motion data may include the rate of change of the position of the target object on the image and the speed and acceleration of the target object in the target three-dimensional space. The rate of change of the position of the target object on the image may refer to the rate of change of the position between the 2D tracking information of the target object on the second image and the 2D tracking information of the target object on the first image. For details on determining the rate of change of the position, please refer to the relevant description of the tracker below, which will not be repeated here.

[0081] In some embodiments, the speed and acceleration of the target object in the target three-dimensional space may refer to the speed and acceleration of the target object at the acquisition time corresponding to the second image. For example, taking the target object as a pedestrian and the acquisition time t2 corresponding to the second image as an example, the speed and acceleration of the target object in the target three-dimensional space may be the speed and acceleration of the pedestrian at time t2. It can be understood that the speed and acceleration are the walking speed and acceleration of the pedestrian in the real space. In some embodiments, when the image acquisition device is set in a mobile device, the speed and acceleration of the mobile device at the acquisition time of the second image can be determined as the speed and acceleration of the target object in the target three-dimensional space. For example, taking the mobile device as an autonomous driving vehicle as an example, the motion data may be the speed and acceleration of the vehicle at time t2. It can be understood that the speed and acceleration are the driving speed and acceleration of the vehicle in the real space.

[0082] In some embodiments, the 3D prediction box of the target object in the target three-dimensional space and the 2D prediction box on the first image can be predicted based on the tracking result of the tracker tracking the target object for the second image. In some embodiments, the 3D prediction box of the target object in the target three-dimensional space and the 2D prediction box on the first image are predicted based on the tracking result of the tracker tracking the target object for the second image, including: updating the tracker according to the motion data; inputting the 3D tracking position information and the 2D tracking position information into the updated tracker to obtain the 3D prediction box and the 2D prediction box output by the tracker.

[0083] In some embodiments, the tracker can output a 2D prediction frame based on the position change rate and the 2D tracking position information, and output a 3D prediction frame based on the speed, acceleration and the 3D tracking position. In some embodiments, the value of the rotation angle of the 3D prediction frame set by the tracker remains unchanged.

[0084] In some embodiments, the 2D tracking information of the target object may be a 2D detection box corresponding to the 2D tracking information. In some embodiments, the 2D tracking information may be represented by (cx, cy, w, h), where (cx, cy) represents the coordinates of the center point of the 2D detection box corresponding to the 2D tracking information in the image coordinate system of the first image, and (w, h) may represent the width and height of the 2D detection box. In some embodiments, the 3D tracking information of the target object may be a 3D detection box corresponding to the 3D tracking information. In some embodiments, the 3D tracking information may be represented by (x, y, rot), where (x, y) may represent the coordinates of the center point of the 3D detection box corresponding to the 3D tracking information in the target three-dimensional space, and rot represents the rotation angle of the 3D detection box.

[0085] In some embodiments, a 3D prediction box of each target object in the target three-dimensional space and a 2D prediction box on the first image may be predicted based on the tracking results of multiple trackers tracking each target object for the second image. In some embodiments, for any target object, the tracker corresponding to the target object may include a state transfer function of the target object, and based on the state transfer function of the target object and the tracking results of tracking the target object for the second image, a 3D prediction box of the target object in the target three-dimensional space and a 2D prediction box on the first image are predicted.

[0086] In some embodiments, for any target object, the state transfer function of the target object included in the tracker corresponding to the target object is the following formula (1):

[0087]

[0088] The meanings of (cx, cy, w, h) and (x, y, rot) can be found in the above descriptions and will not be repeated here. CX ,V Cy ,V w ,V h ) represents the rate of change of the position of the target object on the image, corresponding to (cx, cy, w, h), V x and V y Respectively represent the speed of the target object along the X-axis and Y-axis directions of the target three-dimensional space, a x and a y Respectively represent the acceleration of the target object along the X-axis and Y-axis directions of the target three-dimensional space.

[0089] In some embodiments, when the second image is the first image in the image acquisition sequence, the position change rate of the target object on the image is an initial value of 0. By way of example, still taking the above-mentioned image acquisition sequence 1 "(P1, P2, P3)" as an example, assuming that the first image is P2, which includes target objects 1-5, and the second image is P1, which includes objects 1-5, the following will describe the process of predicting the 3D prediction box and the 2D prediction box of the target object 1 for each acquired image in the image acquisition sequence 1 by the tracker 1 corresponding to the target object 1 in combination with the example. First, by performing object detection on the second image P1, a 2D detection frame and a 3D detection frame of the target object 1 corresponding to the second image P1 can be obtained. Since the second image P1 is the first image, at this time, the position change rate 1 corresponding to the target object 1 is 0, and the 3D tracking position information and the 2D tracking position information of the target object 1 corresponding to the second image P1 are the 3D detection frame and the 2D detection frame obtained by its object detection (i.e., the 3D detection frame and the 2D detection frame obtained by the above step 120). At the same time, the speed 1 and the acceleration 1 of the target object 1 at the acquisition time corresponding to the second image P1 can be detected. The tracker 1 is updated according to the position change rate 1, the speed 1 and the acceleration 1, that is, the above formula (1) is updated. The 3D tracking position information and the 2D tracking position information are input into the updated tracker 1 to obtain the 3D prediction frame and the 2D prediction frame of the target object 1 corresponding to the image at the next moment (i.e., the first image P2). Further, the 3D detection frame and 2D detection frame of the object on the first image P2 can be obtained according to the object detection. By means of the following step 140, an object belonging to the same target as the target object 1 in the first image P2 can be obtained, assuming it is object 1. Further, the position change rate 2 of the target object 1 on the second image P1 and the first image P2 can be obtained through the 2D detection frame corresponding to object 1 and the 2D detection frame of the target object 1. The 3D prediction frame and 2D prediction frame of the target object 1 corresponding to the third image P3 can be obtained through the position change rate 2 and the speed and acceleration of the target object 1 at the time of acquisition of the first image P2. It can be seen that the position information of the target object 1 on each acquired image in the image acquisition sequence can be obtained through the tracker 1 corresponding to the target object 1, so as to realize the tracking of the target object 1 on the image sequence.

[0090] In the embodiments of the present specification, by outputting a 2D prediction frame based on the position change rate and 2D tracking position information according to the tracker, that is, the tracker uses a uniform speed model to predict or estimate the motion of the 2D frame, the amount of calculation for predicting the 2D frame can be reduced. By outputting a 3D prediction frame based on the speed, acceleration and 3D tracking position information according to the tracker, that is, the tracker uses a uniform acceleration model to predict or estimate the motion of the 3D frame, the prediction accuracy of the 3D frame can be improved. The angle uncertainty of the 3D detection frame obtained by object detection is large. By keeping the rotation angle of the 3D prediction frame in the tracker unchanged and using a static model to smooth the rotation angle, the prediction accuracy of the 3D frame can be further improved.

[0091] Step 140: Determine a tracking result of tracking the target object in the first image according to the 3D detection frame, the 2D detection frame, the 3D prediction frame, and the 2D prediction frame.

[0092] In some embodiments, before determining the tracking result of tracking the target object for the first image based on the 3D detection frame, the 2D detection frame, the 3D prediction frame, and the 2D prediction frame, the method further includes: performing non-maximum suppression processing on the 3D detection frame, the 2D detection frame, the 3D prediction frame, and the 2D prediction frame. By performing non-maximum suppression processing, overlapping 3D frames (i.e., 3D detection frames or 3D prediction frames) and 2D frames (i.e., 2D detection frames or 2D prediction frames) can be filtered out to avoid overlapping frames affecting subsequent matching between 3D frames and matching between 2D frames, thereby improving the accuracy of subsequent matching and thus improving the accuracy of target tracking.

[0093] In some embodiments, a tracking result of tracking a target object for a first image is determined based on a 3D detection frame, a 2D detection frame, a 3D prediction frame, and a 2D prediction frame, including: determining a target 3D detection frame that matches the 3D prediction frame from each 3D detection frame based on a first intersection-and-union ratio and / or a distance value between the 3D prediction frame and each 3D detection frame; taking an object corresponding to the target 3D detection frame in the target three-dimensional space as the target object, and taking the position information of the target 3D detection frame in the target three-dimensional space as the 3D tracking position information of the target object, and the tracking result includes the 3D tracking position information.

[0094] In some embodiments, the first intersection-and-union ratio may refer to the overlap rate between the 3D prediction frame and the 3D detection frame, that is, the ratio of the intersection and union of the 3D prediction frame and the 3D detection frame. In some embodiments, the target object may be one or more, and correspondingly, the 3D prediction frame of the target object in the target three-dimensional space may also include one or more. In some embodiments, for multiple 3D prediction frames, a first intersection-and-union ratio matrix may be determined based on the first intersection-and-union ratio between each 3D prediction frame and each 3D detection frame. In some embodiments, the distance value may be the distance between the center points of the 3D prediction frame and the 3D detection frame in the target three-dimensional space. The distance may include, but is not limited to, Manhattan distance or Euclidean distance, etc. In some embodiments, for multiple 3D prediction frames, a distance matrix may be determined based on the distance value between each 3D prediction frame and each 3D detection frame. In an embodiment of the present specification, the first intersection-and-union ratio matrix or the distance matrix may be solved according to the Hungarian algorithm to determine the matching result between the 3D prediction frame and the 3D detection frame, and determine the target 3D detection frame matching each 3D prediction frame from each 3D detection frame.

[0095] As mentioned above, the target object or objects may be targets of different categories. In some embodiments, for the target objects and objects of the vehicle category, a first intersection-and-union matrix may be determined based on the first intersection-and-union value between each 3D prediction frame and each 3D detection frame. For the target objects and objects of the pedestrian category, a distance matrix may be determined based on the distance value between each 3D prediction frame and each 3D detection frame. In the embodiments of this specification, the first intersection-and-union matrix and the distance matrix may be solved respectively according to the Hungarian algorithm to determine the matching result between the 3D prediction frame and the 3D detection frame, and the target 3D detection frame matching each 3D prediction frame may be determined from each 3D detection frame.

[0096] For example, the second image includes target objects 1-5, and the 3D prediction boxes of the target objects 1-5 are N 3D-1 、N 3D-2 、N 3D-3 、n 3D-4 、n 3D-5 The first image includes objects 1-5, and the 3D detection boxes of objects 1-5 are M 3D-1 、M 3D-2 、M 3D-3 、M 3D-4 、M 3D-5 For example, if the first intersection-over-union matrix and the distance matrix are solved according to the Hungarian algorithm, the matching result is: N 3D-1 -M 3D-2 , N 3D-2 -M 3D-1 , N 3D-3 -M 3D-3 , then with the 3D prediction box N 3D-1The matched target 3D detection frame is 3D detection frame M 3D-2 , and the 3D prediction box N 3D-2 The matched target 3D detection frame is 3D detection frame M 3D-1 , and the 3D prediction box N 3D-3 The matched target 3D detection frame is 3D detection frame M 3D-3 .

[0097] In some embodiments, determining a tracking result of tracking a target object for a first image based on a 3D detection frame, a 2D detection frame, a 3D prediction frame, and a 2D prediction frame also includes: for a 3D detection frame in each 3D detection frame that is not matched to a 3D prediction frame, determining a 2D detection frame corresponding to the same object as the 3D detection frame, and determining a target 2D detection frame that matches the 2D prediction frame based on a second intersection-and-union ratio between the 2D detection frame and the 2D prediction frame; taking an object on the first image corresponding to the target 2D detection frame as the target object, and taking the position information of the target 2D detection frame on the first image as the 2D tracking position information of the target object, and the tracking result includes the 2D tracking position information.

[0098] The method for determining the second intersection-and-union ratio between the 2D detection frame and the 2D prediction frame is similar to the first intersection-and-union ratio, and will not be repeated here. In some embodiments, a second intersection-and-union matrix can be determined based on the second intersection-and-union ratio between the 2D detection frame and the 2D prediction frame, and the second intersection-and-union matrix can be solved according to the Hungarian algorithm to determine the matching result between the 2D detection frame and the 2D prediction frame, thereby determining the target 2D detection frame that matches the 2D prediction frame.

[0099] For example, still taking the above example, the 3D detection frames that are not matched to the 3D prediction frames in the 3D detection frames include M 3D-4 and M 3D-5 , determine the 3D detection box M 3D-4 and M 3D-5 2D detection box M corresponding to the same object 2D-4 and M 2D-5 , and the 2D detection box M 2D-4 and M 2D-5 and 2D prediction box N 2D-1 、N 2D-2 、N 2D-3 、N 2D-4 、N 2D-5 The second intersection-and-union matrix between the 2D detection boxes and the 2D prediction boxes is obtained by solving the second intersection-and-union matrix according to the Hungarian algorithm. 2D-4 -N 2D-4 , M 2D-5 -N 2D-5 .

[0100] The 3D prediction frame and the 2D prediction frame are the 3D frame and the 2D frame corresponding to the target object in the second image, and the 3D detection frame and the 3D detection frame are the 3D frame and the 2D frame corresponding to the object in the first image. By matching the prediction frame and the detection frame, the target object and the object belonging to the same target in the second image and the first image can be determined to achieve target tracking. For example, with the above matching result M 2D-4 -N 2D-4 For example, it can be determined that the object 4 in the second image at the previous moment and the target object 4 in the first image at the next moment belong to the same target. In some embodiments, the target object and the object belonging to the same target can be given the same ID.

[0101] In the embodiments of the present specification, the 3D detection frame and the 3D prediction frame are first matched, and then the 2D detection frame and the 2D prediction frame are matched, using a two-stage matching. The matching accuracy of the 3D frame (i.e., the 3D detection frame and the 3D prediction frame) is high, and the probability of wrong matching is low. The matching of the 2D frame (i.e., the 2D detection frame and the 2D prediction frame) makes the probability of missed matching low. Therefore, through the two-stage matching, high-precision matching can be achieved while reducing the situation of missed matching, that is, the matching accuracy of the target object in the image at the previous moment and the object in the image at the next moment is improved, and the situation of missed matching between the target object and the object is reduced, thereby improving the tracking accuracy of the same target in images at different moments.

[0102] In some embodiments, a new tracker may be created for a target object for which both the 3D prediction box and the 2D prediction box have not been successfully matched. In some embodiments, a tracker corresponding to a target object for which both the 3D prediction box and the 2D prediction box have not been successfully matched for a preset number of times may be discarded. The preset number of times may be specifically set according to actual conditions. As can be seen from the above, the 3D prediction box and the 2D prediction box correspond to two-stage matching. If there is no matching result for the target object in the two-stage matching for the preset number of times, it can be considered that the tracker has failed to track the target object, and the tracker of the target object is discarded.

[0103] In the embodiment of the present disclosure, the tracking result of the target object is determined by using 3D frame and 2D frame information, and 3D information is introduced, so that the target tracking method of the present disclosure can continue to estimate the motion of the target object in the target three-dimensional space for a period of time after the target object is lost, thereby improving the probability of successful matching after the target object reappears, reducing ID switching caused by missed detection or out-of-field situations, that is, reducing the situation of erroneous tracking of the target.

[0104] Figure 2 is a flowchart of determining a 3D detection frame and a 2D detection frame according to an exemplary embodiment. Figure 2 As shown, the method includes the following steps.

[0105] Step 210: Perform object detection on the multiple acquired images to obtain a 3D detection frame of the object on each acquired image in the three-dimensional space of each image acquisition device and a 2D detection frame on the acquired image.

[0106] The specific details of step 210 are similar to those of step 120. For details, please refer to the above step 210 and its related description, which will not be repeated here.

[0107] For example, still taking the aforementioned multiple acquired images as P1, P2, and P3 as an example, object detection can be performed on the acquired image P1 to obtain a 3D detection box of the object on the acquired image P1 in the three-dimensional space of the image acquisition device 1 and a 2D detection box in the image coordinate system of the acquired image P1; object detection for the acquired images P2 and P3 is similar to that for the acquired image P1, and will not be repeated here.

[0108] Step 220 , mapping the 3D detection boxes in the three-dimensional spaces of different image acquisition devices to the same target coordinate system according to the external parameters of each of the image acquisition devices, wherein the target three-dimensional space is the space defined by the target coordinate system.

[0109] In some embodiments, the target coordinate system may be determined based on the position where the image acquisition device is set. For example, if the image acquisition device is set in an autonomous driving vehicle, the target coordinate system may be the vehicle coordinate system corresponding to the first image in the image acquisition sequence. For another example, if the image acquisition device is set at a preset fixed position, the target coordinate system may be a coordinate system determined based on the preset fixed position, and the origin, X-axis, Y-axis, and Z-axis of the coordinate system may be specifically set according to actual conditions.

[0110] In some embodiments, the external parameters of each image acquisition coordinate system can reflect the posture relationship between the image acquisition coordinate system and the target coordinate system. The external parameters may include translation parameters and rotation parameters. In some embodiments, the external parameters of each image acquisition coordinate system can be obtained by calibrating the image acquisition device. For the calibration of the image acquisition device, please refer to the relevant technology, which will not be repeated here.

[0111] Step 230, stitching the multiple acquired images to obtain a stitched image, and mapping the 2D detection frames on the multiple acquired images to the stitched image, the 2D detection frame on the first image being the 2D detection frame on the stitched image.

[0112] In some embodiments, the 2D detection frame on the stitched image may refer to a 2D detection frame in the image coordinate system of the stitched image. By mapping the 2D detection frames on multiple acquired images to the stitched image, the 2D detection frame in the image coordinate system of each acquired image may be converted to the image coordinate system of the stitched image, that is, the 2D detection frames corresponding to different image coordinate systems may be converted to the same image coordinate system.

[0113] In the embodiments of the present specification, by mapping 3D detection frames located in the three-dimensional space of different image acquisition devices to the same target coordinate system, and mapping 2D detection frames on multiple acquired images to a stitched image, the detection results of multiple acquired images of multiple image acquisition devices are fused, and the targets in the multiple acquired images can be tracked simultaneously, so that the tracking algorithm only needs to be executed once to achieve the tracking of the same target in different image acquisition images, avoiding the inefficiency caused by separately tracking the target in the acquired image of each image acquisition device, and reducing the ID switching problem of the same target in different image acquisition devices.

[0114] Figure 3 FIG. 3 is a block diagram of a target tracking device 300 according to an exemplary embodiment. Figure 3 The device includes an acquisition module 310, a detection module 320, a prediction module 330 and a determination module 340.

[0115] An acquisition module 310 is configured to acquire an image acquisition sequence, wherein the image acquisition sequence is obtained based on images acquired by an image acquisition device at multiple acquisition moments;

[0116] The detection module 320 is configured to perform object detection on a first image in the image acquisition sequence to obtain a 3D detection frame of the object on the first image in a target three-dimensional space and a 2D detection frame on the first image, wherein the first image is any image in the image acquisition sequence except the first image;

[0117] A prediction module 330 is configured to predict a 3D prediction box of the target object in the target three-dimensional space and a 2D prediction box on the first image according to a tracking result of tracking the target object in a second image, wherein the second image is a previous image of the first image in the image acquisition sequence;

[0118] The determination module 340 is configured to determine a tracking result of tracking the target object for the first image according to the 3D detection frame, the 2D detection frame, the 3D prediction frame and the 2D prediction frame.

[0119] In some embodiments, the determination module 340 is further configured to:

[0120] Determining, from each of the 3D detection frames, a target 3D detection frame that matches the 3D prediction frame according to a first intersection-over-union ratio and / or a distance value between the 3D prediction frame and each of the 3D detection frames;

[0121] An object corresponding to the target 3D detection frame in the target three-dimensional space is used as the target object, and position information of the target 3D detection frame in the target three-dimensional space is used as 3D tracking position information of the target object, and the tracking result includes the 3D tracking position information.

[0122] In some embodiments, the determination module 340 is further configured to:

[0123] For a 3D detection frame in each of the 3D detection frames that is not matched to the 3D prediction frame, determining a 2D detection frame corresponding to the same object as the 3D detection frame, and determining a target 2D detection frame that matches the 2D prediction frame according to a second intersection-and-union ratio between the 2D detection frame and the 2D prediction frame;

[0124] An object corresponding to the target 2D detection frame on the first image is used as the target object, and position information of the target 2D detection frame on the first image is used as 2D tracking position information of the target object, and the tracking result includes the 2D tracking position information.

[0125] In some embodiments, the tracking result includes 3D tracking position information of the target object corresponding to the second image, 2D tracking position information, and motion data of the target object;

[0126] The prediction module 330 is further configured to:

[0127] updating a tracker according to the motion data;

[0128] The 3D tracking position information and the 2D tracking position information are input into the updated tracker to obtain the 3D prediction frame and the 2D prediction frame output by the tracker.

[0129] In some embodiments, the motion data includes a rate of change of position of the target object on the image, and a speed and acceleration of the target object in the target three-dimensional space;

[0130] The tracker can output the 2D prediction frame based on the position change rate and the 2D tracking position information, and output the 3D prediction frame based on the speed, the acceleration, and the 3D tracking position information.

[0131] In some embodiments, the first image includes a plurality of acquired images, and the plurality of acquired images are images acquired by a plurality of image acquisition devices at the same acquisition time;

[0132] The detection module 320 is further configured to:

[0133] Performing object detection on the multiple acquired images to obtain a 3D detection frame of the object on each acquired image in the three-dimensional space of each image acquisition device and a 2D detection frame on the acquired image;

[0134] Mapping 3D detection frames located in the three-dimensional space of different image acquisition devices to the same target coordinate system according to the external parameters of each of the image acquisition devices, wherein the target three-dimensional space is a space defined by the target coordinate system;

[0135] The multiple acquired images are stitched together to obtain a stitched image, and the 2D detection frames on the multiple acquired images are mapped to the stitched image, and the 2D detection frame on the first image is the 2D detection frame on the stitched image.

[0136] In some embodiments, the determination module 340 is further configured to:

[0137] Non-maximum suppression processing is performed on the 3D detection frame, the 2D detection frame, the 3D prediction frame and the 2D prediction frame.

[0138] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0139] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, and when the program instructions are executed by a processor, the steps of the target tracking method provided by the present disclosure are implemented.

[0140] Figure 4 4 is a block diagram of a device 400 for target tracking according to an exemplary embodiment. For example, the device 400 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0141] Reference Figure 4 , the device 400 may include one or more of the following components: a processing component 402 , a memory 404 , a power component 406 , a multimedia component 408 , an audio component 410 , an input / output (I / O) interface 412 , a sensor component 414 , and a communication component 416 .

[0142] The processing component 402 generally controls the overall operation of the device 400, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the above-mentioned target tracking method. In addition, the processing component 402 may include one or more modules to facilitate the interaction between the processing component 402 and other components. For example, the processing component 402 may include a multimedia module to facilitate the interaction between the multimedia component 408 and the processing component 402.

[0143] The memory 404 is configured to store various types of data to support operations on the device 400. Examples of such data include instructions for any application or method operating on the device 400, contact data, phone book data, messages, pictures, videos, etc. The memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0144] The power component 406 provides power to the various components of the device 400. The power component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 400.

[0145] The multimedia component 408 includes a screen that provides an output interface between the device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 408 includes a front camera and / or a rear camera. When the device 400 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0146] The audio component 410 is configured to output and / or input audio signals. For example, the audio component 410 includes a microphone (MIC), and when the device 400 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 404 or sent via the communication component 416. In some embodiments, the audio component 410 also includes a speaker for outputting audio signals.

[0147] I / O interface 412 provides an interface between processing component 402 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0148] The sensor assembly 414 includes one or more sensors for providing various aspects of the status assessment of the device 400. For example, the sensor assembly 414 can detect the open / closed state of the device 400, the relative positioning of components, such as the display and keypad of the device 400, and the sensor assembly 414 can also detect the position change of the device 400 or a component of the device 400, the presence or absence of user contact with the device 400, the orientation or acceleration / deceleration of the device 400, and the temperature change of the device 400. The sensor assembly 414 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 414 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 414 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0149] The communication component 416 is configured to facilitate wired or wireless communication between the device 400 and other devices. The device 400 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 416 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0150] In an exemplary embodiment, the device 400 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above-mentioned target tracking method.

[0151] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, and the instructions can be executed by a processor 420 of the device 400 to perform the target tracking method described above. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0152] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program executable by a programmable device. The computer program has a code portion for executing the above target tracking method when executed by the programmable device.

[0153] Figure 5 is a block diagram of a device 500 for target tracking according to an exemplary embodiment. For example, the device 500 may be provided as a server. Figure 5 , the apparatus 500 includes a processing component 522, which further includes one or more processors, and a memory resource represented by a memory 532 for storing instructions executable by the processing component 522, such as an application. The application stored in the memory 532 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 522 is configured to execute instructions to perform the above-mentioned target tracking method.

[0154] The device 500 may also include a power supply component 526 configured to perform power management of the device 500, a wired or wireless network interface 550 configured to connect the device 500 to a network, and an input / output (I / O) interface 558. The device 500 may operate based on an operating system stored in the memory 532, such as Windows Server 2008. TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM or similar.

[0155] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the present disclosure. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0156] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A target tracking method, characterized in that: include: Acquire an image acquisition sequence, wherein the image acquisition sequence is obtained based on images acquired by an image acquisition device at multiple acquisition moments; Performing object detection on a first image in the image acquisition sequence to obtain a 3D detection frame of the object on the first image in a target three-dimensional space and a 2D detection frame on the first image, wherein the first image is any image in the image acquisition sequence except the first image; Predicting a 3D prediction box of the target object in the target three-dimensional space and a 2D prediction box of the target object on the first image according to a tracking result of tracking the target object in a second image, wherein the second image is a previous image of the first image in the image acquisition sequence; Determine a tracking result of tracking the target object with respect to the first image according to the 3D detection frame, the 2D detection frame, the 3D prediction frame, and the 2D prediction frame; The determining, according to the 3D detection frame, the 2D detection frame, the 3D prediction frame, and the 2D prediction frame, a tracking result of tracking the target object for the first image includes: Determining, from each of the 3D detection frames, a target 3D detection frame that matches the 3D prediction frame according to a first intersection-over-union ratio and / or a distance value between the 3D prediction frame and each of the 3D detection frames; taking an object in the target three-dimensional space corresponding to the target 3D detection frame as the target object, and taking position information of the target 3D detection frame in the target three-dimensional space as 3D tracking position information of the target object, wherein the tracking result includes the 3D tracking position information; For a 3D detection frame in each of the 3D detection frames that is not matched to the 3D prediction frame, determining a 2D detection frame corresponding to the same object as the 3D detection frame, and determining a target 2D detection frame that matches the 2D prediction frame according to a second intersection-and-union ratio between the 2D detection frame and the 2D prediction frame; An object corresponding to the target 2D detection frame on the first image is used as the target object, and position information of the target 2D detection frame on the first image is used as 2D tracking position information of the target object, and the tracking result includes the 2D tracking position information.

2. The target tracking method according to claim 1, characterized in that: The tracking result includes 3D tracking position information of the target object corresponding to the second image, 2D tracking position information and motion data of the target object; The step of predicting a 3D prediction box of the target object in the three-dimensional space and a 2D prediction box of the target object on the first image according to a tracking result of tracking the target object on the second image includes: updating a tracker according to the motion data; The 3D tracking position information and the 2D tracking position information are input into the updated tracker to obtain the 3D prediction frame and the 2D prediction frame output by the tracker.

3. The target tracking method according to claim 2, characterized in that: The motion data includes the rate of change of the position of the target object on the image, and the speed and acceleration of the target object in the target three-dimensional space; The tracker can output the 2D prediction frame based on the position change rate and the 2D tracking position information, and output the 3D prediction frame based on the speed, the acceleration, and the 3D tracking position information.

4. The target tracking method according to claim 1, characterized in that: The first image includes a plurality of acquired images, and the plurality of acquired images are images acquired by a plurality of image acquisition devices at the same acquisition time; The performing object detection on the first image in the image acquisition sequence to obtain a 3D detection frame of the object on the first image in the target three-dimensional space and a 2D detection frame on the first image includes: Performing object detection on the multiple acquired images to obtain a 3D detection frame of the object on each acquired image in the three-dimensional space of each image acquisition device and a 2D detection frame on the acquired image; Mapping 3D detection frames located in three-dimensional spaces of different image acquisition devices to the same target coordinate system according to the external parameters of each of the image acquisition devices, wherein the target three-dimensional space is a space defined by the target coordinate system; The multiple acquired images are stitched together to obtain a stitched image, and the 2D detection frames on the multiple acquired images are mapped to the stitched image, and the 2D detection frame on the first image is the 2D detection frame on the stitched image.

5. The target tracking method according to claim 1, before determining a tracking result of tracking the target object for the first image according to the 3D detection frame, the 2D detection frame, the 3D prediction frame and the 2D prediction frame, the method further comprises: Non-maximum suppression processing is performed on the 3D detection frame, the 2D detection frame, the 3D prediction frame, and the 2D prediction frame.

6. A target tracking device, characterized in that: include: An acquisition module is configured to acquire an image acquisition sequence, wherein the image acquisition sequence is obtained based on images acquired by an image acquisition device at multiple acquisition moments; a detection module configured to perform object detection on a first image in the image acquisition sequence to obtain a 3D detection frame of the object on the first image in a target three-dimensional space and a 2D detection frame on the first image, wherein the first image is any image in the image acquisition sequence except the first image; a prediction module configured to predict a 3D prediction box of the target object in the target three-dimensional space and a 2D prediction box on the first image according to a tracking result of tracking the target object in a second image, wherein the second image is a previous image of the first image in the image acquisition sequence; a determination module, configured to determine a tracking result of tracking the target object for the first image according to the 3D detection frame, the 2D detection frame, the 3D prediction frame, and the 2D prediction frame; The determination module is further configured to: Determining, from each of the 3D detection frames, a target 3D detection frame that matches the 3D prediction frame according to a first intersection-over-union ratio and / or a distance value between the 3D prediction frame and each of the 3D detection frames; taking an object in the target three-dimensional space corresponding to the target 3D detection frame as the target object, and taking position information of the target 3D detection frame in the target three-dimensional space as 3D tracking position information of the target object, wherein the tracking result includes the 3D tracking position information; For a 3D detection frame in each of the 3D detection frames that is not matched to the 3D prediction frame, determining a 2D detection frame corresponding to the same object as the 3D detection frame, and determining a target 2D detection frame that matches the 2D prediction frame according to a second intersection-and-union ratio between the 2D detection frame and the 2D prediction frame; An object corresponding to the target 2D detection frame on the first image is used as the target object, and position information of the target 2D detection frame on the first image is used as 2D tracking position information of the target object, and the tracking result includes the 2D tracking position information.

7. A target tracking device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: Acquire an image acquisition sequence, wherein the image acquisition sequence is obtained based on images acquired by an image acquisition device at multiple acquisition moments; Performing object detection on a first image in the image acquisition sequence to obtain a 3D detection frame of the object on the first image in a target three-dimensional space and a 2D detection frame on the first image, wherein the first image is any image in the image acquisition sequence except the first image; Predicting a 3D prediction box of the target object in the target three-dimensional space and a 2D prediction box of the target object on the first image according to a tracking result of tracking the target object in a second image, wherein the second image is a previous image of the first image in the image acquisition sequence; Determine a tracking result of tracking the target object with respect to the first image according to the 3D detection frame, the 2D detection frame, the 3D prediction frame, and the 2D prediction frame; The processor is further configured to: Determining, from each of the 3D detection frames, a target 3D detection frame that matches the 3D prediction frame according to a first intersection-over-union ratio and / or a distance value between the 3D prediction frame and each of the 3D detection frames; taking an object in the target three-dimensional space corresponding to the target 3D detection frame as the target object, and taking position information of the target 3D detection frame in the target three-dimensional space as 3D tracking position information of the target object, wherein the tracking result includes the 3D tracking position information; For a 3D detection frame in each of the 3D detection frames that is not matched to the 3D prediction frame, determining a 2D detection frame corresponding to the same object as the 3D detection frame, and determining a target 2D detection frame that matches the 2D prediction frame according to a second intersection-and-union ratio between the 2D detection frame and the 2D prediction frame; An object corresponding to the target 2D detection frame on the first image is used as the target object, and position information of the target 2D detection frame on the first image is used as 2D tracking position information of the target object, and the tracking result includes the 2D tracking position information.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 5 are implemented.

Images