Target tracking method and device, electronic equipment and medium

Abstract

The embodiment of the invention provides a target tracking method and device, electronic equipment and a medium, and relates to the field of image processing. The method comprises the following steps: detecting images collected by each monocular camera, and determining an obstacle target and detection information of the obstacle target; according to the detection information of the obstacle target, tracking the obstacle target in the current frame image collected by each monocular camera to obtain a first tracking result corresponding to the monocular camera; fusing the first tracking results corresponding to the two adjacent monocular cameras to obtain a second tracking result; or fusing the detection information of the obstacle target of the current frame image collected by the plurality of monocular cameras to obtain a current frame fused obstacle target; and tracking the current frame fusion obstacle target according to the detection information of the current frame fusion obstacle target to obtain a third tracking result. According to the method, vehicle surrounding information and obstacle tracks are obtained in real time in an omnibearing manner, and the accuracy and stability of target tracking identification are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a target tracking method, device, electronic equipment and medium. Background Art

[0002] Single-camera target tracking limits the target tracking range to the field of view of a single camera, which is difficult to meet the needs of complex application scenarios. Cross-camera target tracking integrates the information of multiple cameras and uses technologies such as trajectory association to jointly track multiple targets between multiple cameras. It is of great significance for real-time monitoring of complex scenes such as smart security, autonomous driving, and smart cities. For example, in the field of autonomous driving, in order to make correct decisions, it is necessary to integrate the information of multiple cameras, radars and other sensors to fully obtain 360-degree environmental perception. The current tracking method mainly follows the method of fusion before detection, that is, sensor information is integrated before target detection, and model reasoning directly outputs the 3D information of the target without the need for cross-sensor matching later. However, the tracking effect of this method mainly depends on the reasoning ability of the model, which usually requires a large amount of labeled data for training. Data labeling is difficult, and the model structure is more complex, the reasoning speed is slow, and it is difficult to deploy and run in resource-constrained environments or environments with high real-time requirements. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, an embodiment of the present invention provides a target tracking method, device, electronic device and medium.

[0004] In a first aspect, an embodiment of the present invention provides a target tracking method, the method comprising a plurality of monocular cameras deployed on a same vehicle; the method further comprising:

[0005] Detecting the images captured by each monocular camera to determine obstacle targets in the images and detection information of the obstacle targets;

[0006] For each monocular camera, according to a preset target tracking method and detection information of the obstacle target in the current frame image collected by the monocular camera, the obstacle target in the current frame image is tracked to obtain a first tracking result corresponding to the monocular camera; according to a preset fusion method and the tracking information of the obstacle target in the first tracking result, the first tracking results corresponding to two adjacent monocular cameras are fused to obtain a second tracking result;

[0007] Or, according to the preset fusion method and the detection information of the obstacle target in the current frame image captured by the multiple monocular cameras, the obstacle target in the current frame image captured by the multiple monocular cameras is fused to obtain the current frame fused obstacle target; according to the preset target tracking method and the detection information of the current frame fused obstacle target, the current frame fused obstacle target is tracked to obtain a third tracking result.

[0008] In a second aspect, an embodiment of the present invention provides a target tracking device, the device comprising a plurality of monocular cameras deployed on the same vehicle; the device further comprising:

[0009] A detection module, used to detect the images captured by each monocular camera, determine the obstacle targets in the images and the detection information of the obstacle targets;

[0010] A tracking module, configured to track the obstacle target in the current frame image for each monocular camera according to a preset target tracking method and detection information of the obstacle target in the current frame image collected by the monocular camera, and obtain a first tracking result corresponding to the monocular camera;

[0011] The fusion module is used to fuse the first tracking results corresponding to two adjacent monocular cameras according to a preset fusion method and the tracking information of the obstacle target in the first tracking result to obtain a second tracking result.

[0012] or,

[0013] The fusion module is used to fuse the obstacle targets in the current frame images collected by the multiple monocular cameras according to the preset fusion method and the detection information of the obstacle targets in the current frame images collected by the multiple monocular cameras to obtain the current frame fused obstacle targets;

[0014] The tracking module is used to track the obstacle target fused in the current frame according to the preset target tracking method and the detection information of the obstacle target fused in the current frame to obtain a third tracking result.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor is used to implement the target tracking method provided by any embodiment of the present invention when executing the program stored in the memory.

[0016] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a target tracking method provided by any embodiment of the present invention.

[0017] The technical solution provided by the embodiment of the present invention brings at least the following beneficial effects:

[0018] The target tracking method provided by the embodiment of the present invention detects the image captured by each monocular camera, determines the obstacle target in the image and the detection information of the obstacle target, tracks the obstacle target in the current frame image according to the preset target tracking method and the detection information of the obstacle target in the current frame image captured by the monocular camera, and obtains the first tracking result corresponding to the monocular camera; according to the preset fusion method, fuses the first tracking results corresponding to two adjacent monocular cameras to obtain the second tracking result; or, according to the detection information of the obstacle target in the current frame image captured by multiple monocular cameras, fuses the current frame images captured by multiple monocular cameras to obtain the current frame fused obstacle target; according to the preset target tracking method and the detection information of the obstacle target in the current frame fused image, tracks the obstacle target in the current frame fused image to obtain the third tracking result. This method performs obstacle detection on a single monocular camera, and then tracks / matches the detection results of each monocular camera to obtain the final obstacle detection information. It can obtain all-round and real-time information about the vehicle's surroundings and track its trajectory, which can effectively improve the accuracy and stability of target tracking and recognition. In addition, this method can use a lightweight model, reduce computing pressure, and reduce resource requirements and costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0020] Figure 1 A schematic diagram showing a flow chart of a target tracking method according to an embodiment of the present invention is shown;

[0021] Figure 2A A schematic diagram showing obstacle target filtering by a target tracking method according to an embodiment of the present invention is shown;

[0022] Figure 2B A schematic diagram showing obstacle target filtering by a target tracking method according to another embodiment of the present invention is shown;

[0023] Figure 3 A schematic diagram showing a sub-process of a target tracking method according to an embodiment of the present invention;

[0024] Figure 4 A schematic diagram showing a tracking queue of a target tracking method according to an embodiment of the present invention is shown;

[0025] Figure 5 A schematic diagram showing a target tracking method for matching an obstacle target according to an embodiment of the present invention is shown;

[0026] Figure 6 A schematic diagram showing a target tracking method matching a tracking object according to an embodiment of the present invention is shown;

[0027] Figure 7 A schematic diagram showing the structure of a target tracking device according to an embodiment of the present invention is shown;

[0028] Figure 8 A schematic structural diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0030] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0031] Figure 1 A flow chart of a target tracking method according to an embodiment of the present invention is shown. The method continuously collects image information through multiple monocular cameras deployed on the same vehicle, and performs obstacle detection on the image sequences taken by the multiple monocular cameras to track / match the obstacle targets in the image to obtain the final obstacle 3D information, that is, to obtain real-time, omnidirectional obstacle trajectory information. Optionally, in an embodiment of the present invention, the obstacle targets in the image collected by each monocular camera can be tracked according to a preset target tracking method, and then the obstacle target tracking results corresponding to each monocular camera are fused according to a preset fusion method to obtain the final obstacle tracking result (such as Figure 1Still optionally, the embodiment of the present invention may also first fuse the obstacle targets in the images captured by the multiple monocular cameras according to the preset fusion method, and then track the fused obstacle targets according to the preset tracking method to obtain the final obstacle tracking result (such as Figure 1 Steps S101, S102b, and S103b) are shown.

[0032] like Figure 1 As shown, the method includes:

[0033] Step S101: Detecting each image captured by the monocular camera to determine obstacle targets in the image and detection information of the obstacle targets.

[0034] In this step, obstacle detection is performed on the image captured by each monocular camera to identify obstacle targets in the image. For example, in a parking lot scene, obstacle targets may be vehicles parked in the parking lot, wall pillars in the parking lot, and indicator devices set up in the parking lot.

[0035] In an optional embodiment, the same detection model can be used to detect images collected by multiple monocular cameras respectively to determine the obstacle targets in each image. In other optional embodiments, the number of images included in a single detection batch of the detection model can be set according to the number of monocular cameras (for example, the number of monocular cameras is 4, and the number of images included in a single detection batch of the detection model is also set to 4), and according to the number of images included in the single detection batch, multiple images collected by multiple monocular cameras are simultaneously input into a preset detection model to detect the multiple images at the same time, thereby improving the obstacle detection efficiency. Among them, the detection model can adopt a YOLO (You Only Look Once) model or a DETR (DETEction TRansformer) model based on a Transformer architecture.

[0036] The detection information of the obstacle target may include, for example, but is not limited to: the size information of the detection frame, the position information of the detection frame, the orientation information of the obstacle target, the category information of the obstacle target and the detection confidence information. Among them, if the preset detection model is a 2D detection model, the detection frame obtained by detection is a 2D detection frame. If the preset detection model is a 3D detection model, the detection frame obtained by detection is a 3D detection frame. The size information of the 2D detection frame includes the length and width of the detection frame. The size information of the 3D detection frame includes the length, width and height of the detection frame. The position information of the detection frame includes, for example, the coordinates of the center point of the detection frame in the image coordinate system. The orientation information of the obstacle target includes, for example, the orientation angle of the detection frame. The orientation angle of the detection frame can be obtained by the rotation angle of the detection frame around a certain coordinate axis. The category information of the obstacle target is used to indicate the category of the obstacle target. The detection confidence information is used to indicate the reliability of the detection model in determining the category of the obstacle target, and is used to identify the reliability of the model for the detection result.

[0037] In an optional embodiment, if the image captured by the monocular camera is distorted, for example, the monocular camera uses a lens with a larger viewing angle such as a pinhole camera or a fisheye camera with a heavier distortion, the influence of the distortion needs to be considered. In an optional embodiment, the image can be dedistorted before being detected. For example, the image is dedistorted according to the intrinsic parameters and distortion parameters of the monocular camera. In other optional embodiments, after the obstacle targets in the image are detected, only the obstacle targets detected in a specific area of ​​the image are retained, and the obstacle targets detected in other areas are discarded. As an optional example, the obstacle targets can be filtered based on ROI (Region of Interest). For example Figure 2A As shown in FIG. 2 , if the detection model is a 2D detection model, the middle area of ​​the image is selected as the region of interest ROI, and only the obstacle targets detected in the middle area of ​​the image are retained (such as the current detection 1 in FIG. 2 ), and the obstacle targets detected in other areas are discarded (such as the current detection 2 in FIG. 2 ).

[0038] In an optional embodiment, if the detection model is a 3D detection model, such as Figure 2B As shown in the top view perspective of the 3D obstacle, the camera position can be approximated to the origin of the vehicle coordinate system. The obstacle shown in the solid line frame is the 3D target obtained by the right-view camera, and the dotted line frame is the 3D target obtained by the front-view camera. Since the angle of obstacle 1 in the right-view camera is large (about 90 degrees), it can be deleted.

[0039] Step S102a: for each monocular camera, according to a preset target tracking method and the detection information of the obstacle target in the current frame image collected by the monocular camera, track the obstacle target in the current frame image to obtain a first tracking result corresponding to the monocular camera.

[0040] In this step, for each camera, a tracking list corresponding to the monocular camera is created, and the tracking list is used to record the obstacle targets captured by the camera. Tracking of obstacle targets is achieved by matching obstacle targets and updating the tracking list. For example, the obstacle target detected in the current frame image (recorded as image 1) (the obstacle target is recorded in the tracking list as a tracking object) is predicted to predict the position of the obstacle target in the next frame image. Then the next frame image (recorded as image 2) captured by the monocular camera (the next frame image and the current frame image are two adjacent frames captured by the same monocular camera) is detected to detect the obstacle target in image 2. The detected obstacle target is matched with the predicted obstacle target to determine whether the detected obstacle target is the same obstacle as the obstacle target detected in the current frame image. If so, the tracking list is updated based on the obstacle target in image 2. If not, it can be determined that the obstacle target in image 2 is a new obstacle captured by the monocular camera, and the new obstacle target is recorded in the tracking list as a new tracking object. The first tracking result corresponding to each monocular camera is the tracking list corresponding to each monocular camera. The tracking information of the obstacle target in the first tracking result includes: the position, size, trajectory, kinematic parameters (such as speed, acceleration, etc.) and category of each tracking object in the tracking list. As an optional example, the obstacle target detected in image 1 can be predicted by the Kalman filter algorithm, and the detected obstacle target is matched with the predicted obstacle target according to, for example, a greedy algorithm or a Hungarian algorithm. Among them, the Kalman filter algorithm establishes a dynamic model of the system, and when a frame of data is measured, it updates the estimated value of the next frame (that is, the prediction of the next frame) by weighted averaging. The greedy algorithm (also known as the greedy algorithm) directly selects the match with the smallest loss in a single match, which can reach a local optimum. The Hungarian algorithm selects the one with the smallest total loss among all matching schemes to achieve the overall optimum.

[0041] Figure 3 FIG. 1 is a schematic diagram of a tracking sub-process of a target tracking method provided by an embodiment of the present invention. Figure 3 The process of tracking the obstacle target in step S102 includes:

[0042] Step S301: for each monocular camera, create a first tracking list corresponding to the monocular camera, and record the obstacle target in the current frame image captured by the monocular camera as a tracking object in the first tracking list.

[0043] Step S302: predicting the state of the obstacle target in the current frame image according to the Kalman filtering method and the detection information of the obstacle target in the current frame image collected by the monocular camera, and determining the prediction information of the obstacle target in the current frame image in the next frame image.

[0044] Step S303: Detect the next frame of image captured by the monocular camera to determine the obstacle target in the next frame of image and the detection information of the obstacle target.

[0045] The detection information of the obstacle target includes, for example, but is not limited to, the size information of the detection frame, the position information of the detection frame, the orientation information of the obstacle target (for example, the orientation angle of the obstacle target), the category information of the obstacle target and the detection confidence information. The prediction information of the obstacle target in the current frame image on the next frame image includes, for example, the category prediction information, orientation prediction information, predicted size information of the detection frame, predicted position information of the detection frame, etc. of the obstacle target in the current frame image on the next frame image. Therefore, the category information, orientation information, size information of the detection frame, and position information of the detection frame of the obstacle target in the next frame image can be compared with the category prediction information, orientation prediction information, predicted size information of the detection frame, and predicted position information of the detection frame of the obstacle target in the current frame image on the next frame image to determine whether the obstacle target in the next frame image is the same obstacle as the obstacle target in the current frame image. As an optional example, if the detection frame is a 2D detection frame, matching can be performed according to the following formula (1):

[0046] L 1 =m 1 *ΔW+m 2 *ΔH+m 3 *ΔP x +m 4 *ΔP y +m 5 *Δori (1)

[0047] Among them, L 1 Represents the loss value between the obstacle target in the next frame image and the obstacle target in the current frame image. m 1 、m 2 、m 3 、m 4 、m 5 represents the preset weight coefficient, ΔW represents the difference between the length of the obstacle detection frame in the next frame image and the predicted length of the obstacle detection frame in the current frame image on the next frame image, ΔH represents the difference between the width of the obstacle detection frame in the next frame image and the predicted width of the obstacle detection frame in the current frame image on the next frame image, (ΔP x , ΔP y) represents the difference between the coordinates of the detection box of the obstacle target in the next frame image and the predicted coordinates of the detection box of the obstacle target in the current frame image on the next frame image, and Δori represents the difference between the orientation angle of the obstacle target in the next frame image and the predicted orientation angle of the obstacle target in the current frame image on the next frame image.

[0048] If L 1 If L is less than or equal to the specified threshold, it is determined that the obstacle target in the next frame image is the same obstacle as the obstacle target in the current frame image. 1 If the loss value between the obstacle target in the next frame image and the obstacle target in the current frame image is greater than the specified threshold, it is determined that the obstacle target in the next frame image is not the same obstacle as the obstacle target in the current frame image. In an optional embodiment, if the loss values ​​between multiple, for example, two obstacle targets in the next frame image and the same obstacle target in the current frame image are all less than the specified threshold, the obstacle target with the smallest loss value is the same obstacle as the obstacle target in the current frame image.

[0049] Referring to formula (1), if the detection frame is a 3D detection frame, the obstacle target detected in the next frame image is matched with the obstacle target in the current frame image according to the length, width, height, position information of the 3D detection frame and the orientation information of the obstacle target.

[0050] Step S304: when the obstacle target in the next frame image is the same obstacle as the obstacle target in the current frame image, update the detection information of the tracking object in the first tracking list based on the detection information of the obstacle target in the next frame image.

[0051] Step S305: When the obstacle target in the next frame image is not the same obstacle as the obstacle target in the current frame image, the obstacle target in the next frame image is recorded in the first tracking list as a new tracking object.

[0052] Step S306: Count the number of times that the tracking object recorded in the first tracking list fails to be matched.

[0053] Step S307: When the number of unsuccessful matches is greater than or equal to the specified threshold, the tracking object is deleted from the first tracking list. If the number of unsuccessful matches corresponding to a tracking object in the tracking list is greater than or equal to the specified threshold, it means that the tracking object has not been detected in multiple frames of images, and it can be determined that the tracking object is no longer within the shooting angle of the monocular camera, and the vehicle has moved away from the tracking object, and the tracking object can be deleted from the tracking list.

[0054] Figure 4 FIG. 1 is a schematic diagram showing a tracking list of a target tracking method according to an embodiment of the present invention. Figure 4As shown, List 1 records the obstacle targets detected from the image captured by the monocular camera, List 2 records the prediction information of the obstacle targets in the current frame image in List 1 on the next frame image, and List 3 records the obstacle targets detected from the next frame image. According to the prediction information of the obstacle targets in the current frame image in List 2 on the next frame image and the detection information of the obstacle targets in List 3, the obstacle targets in List 2 are matched with the obstacle targets in List 3 to determine whether the obstacle targets in List 3 match the obstacle targets in List 1.

[0055] Figure 5 FIG. 2 shows a tracking and matching schematic diagram of a target tracking method according to an embodiment of the present invention. Figure 5 As shown in the figure, the prediction A and prediction B marked by the solid line frame are the prediction information of the obstacle target in the current frame image on the next frame image, and the current detection 1 and current detection 2 marked by the dotted line frame are the detection frames corresponding to the obstacle target detected in the next frame image. According to the above formula (1), the current detection 1 is matched with the prediction A, the current detection 1 is matched with the prediction B, the current detection 2 is matched with the prediction A, and the current detection 2 is matched with the prediction B. The tracking list corresponding to the tracking object is updated according to the current detection 1. If the current detection 2 fails to match with both the prediction A and the prediction B, the current detection 2 is recorded in the tracking list as a new tracking object. If the prediction B has reached the upper limit of M frames in the matching of the past N frames and has not been matched, it means that the tracking object has disappeared, and the tracking object can be deleted from the tracking list.

[0056] In an optional embodiment, the detection information of the obstacle target includes the category information of the obstacle target. When matching in step S303, only obstacle targets of the same category can be matched according to the category information of the obstacle target. In other optional embodiments, for some obstacles that may be confused during detection, such as two-wheeled vehicles and motorcycles, SUVs (Sport-Utility-Vehicles) and MPVs (Multi-Purpose Vehicles), even if the obstacle targets in the front and rear frame images are identified as different types, the possibility that they are the same obstacle still needs to be considered. Therefore, a confusing category group can be set in advance, and when the category information of the obstacle target belongs to a confusing category, it is matched with other obstacle targets in the confusing category group according to the confusing category group.

[0057] In an optional embodiment, when tracking and matching is performed according to the detection information and the prediction information in step S303, the model re-identification information can also be combined, that is, the model is used to identify whether the obstacle targets in the previous and next frame images are the same obstacle. For example, the next frame image captured by the monocular camera is matched with the obstacle target in the current frame image using a preset recognition model, and the similarity value between the obstacle target on the next frame image and the obstacle target in the current frame image is determined. Then, according to the similarity value between the obstacle target on the next frame image and the obstacle target in the current frame image, the detection information of the obstacle target in the next frame image, and the prediction information of the obstacle target in the current frame image on the next frame image, it is determined whether the obstacle target in the next frame image and the obstacle target in the current frame image are the same obstacle. For example, if the similarity value between the obstacle target on the next frame image and the obstacle target in the current frame image is less than or equal to a preset threshold, and the loss value L1 between the obstacle target on the next frame image and the obstacle target in the current frame image is less than or equal to a specified threshold, it is determined that the obstacle target in the next frame image and the obstacle target in the current frame image are the same obstacle. On the contrary, the obstacle target in the next frame image is not the same obstacle as the obstacle target in the current frame image. For another example, matching can be performed according to the following formula (2):

[0058] L 2 =m 1 *ΔW+m 2 *ΔH+m 3 *ΔP x +m 4 *ΔP y +m 5 *Δori+m 6 *L_reid (2)

[0059] Among them, L 2 Represents the loss value between the obstacle target in the next frame image and the obstacle target in the current frame image. m 6 represents the preset weight coefficient, and L_reid represents the similarity value between the obstacle target in the next frame image and the obstacle target in the current frame image.

[0060] In an optional embodiment, in order to improve the matching efficiency and reduce the number of times the matching loss value is calculated and the resource occupation, when matching, filtering can be performed according to the category of the detection frame and / or the intersection over union (IoU). For example, an embodiment of the present invention also includes: comparing the category information of the obstacle target in the next frame image with the category information of the obstacle target in the current frame image, and / or comparing the size information of the detection frame of the obstacle target in the next frame image with the size information of the detection frame of the obstacle target in the current frame image; when it is determined that the obstacle target in the next frame image is similar to the obstacle target in the current frame image according to the category information and / or the size information of the detection frame, comparing the detection information of the obstacle target in the next frame image with the prediction information of the obstacle target in the current frame image on the next frame image. For example, if the category information of the obstacle target in the next frame image is the same as the category information of the obstacle target in the current frame image, it is determined that the obstacle target in the next frame image is similar to the tracked object. For another example, based on the detection frame size information of the obstacle target in the next frame image and the size information of the detection frame of the tracked object, the intersection and union ratio of the two detection frames is calculated. If the value of the intersection and union ratio of the two detection frames is greater than or equal to the specified threshold, it is determined that the obstacle target in the next frame image is similar to the obstacle target in the current frame image.

[0061] Step S103a: According to a preset fusion method and the tracking information of the obstacle target in the first tracking result, the first tracking results corresponding to two adjacent monocular cameras are fused to obtain a second tracking result.

[0062] If the detection model used in step S101 is a 2D detection model (i.e., the detection frame is a 2D detection frame), the tracked object can be tracked based on the 2D detection frame first, so that the movement of the 2D detection frame of the tracked object in the picture is more stable, which is conducive to the fusion between subsequent tracked objects. Before fusion, the 2D tracked object needs to be projected into 3D space. Projecting the 2D tracked object into 3D space requires using the camera calibration information to calculate the transformation matrix between each coordinate system. The transformation formula is shown in the following formula (3):

[0063] P 3D =M c2v ·M i2c ·P img (3)

[0064] P img Represents the position coordinates of the obstacle target in the image coordinate system, M c2v Represents the transformation matrix from the camera coordinate system to the current vehicle coordinate system, M i2c Represents the transformation matrix from the image coordinate system to the camera coordinate system, P 3DRepresents the position coordinates of the obstacle coordinate system in the vehicle coordinate system, that is, the position in 3D space.

[0065] In this embodiment, 3D tracking can be performed for each monocular camera, and the corresponding tracking queues can be updated respectively, and then the adjacent cameras can be fused. The fusion between adjacent cameras is to fuse the tracking lists corresponding to the adjacent cameras, and at this time, the tracking objects in the tracking lists need to be matched. The two successfully matched tracking objects are the tracking results of a single obstacle existing in the overlapping fields of view of adjacent cameras. When fusion is performed, the more credible one can be selected according to the confidence of the tracking trajectories of the two cameras, or the two trajectories can be weighted averaged.

[0066] In an optional embodiment, the tracking information of the tracked object includes: the position, size, trajectory, kinematic parameters and category of the tracked object. The tracking information corresponding to two adjacent cameras can be fused according to the following process:

[0067] (1) The tracking objects in the two tracking lists corresponding to the two adjacent monocular cameras are respectively taken as the first tracking object and the second tracking object, and the matching loss value between the first tracking object and the second tracking object is calculated according to the position difference, size difference, trajectory difference, kinematic parameter difference and category difference between the first tracking object and the second tracking object.

[0068] The position difference between the first tracked object and the second tracked object can be represented by the difference between the center point of the detection frame corresponding to the first tracked object and the center point of the detection frame corresponding to the second tracked object. The size difference can be the difference between the size of the detection frame corresponding to the first tracked object and the size of the detection frame corresponding to the second tracked object. The trajectory difference is the difference between the trajectory of the first tracked object and the trajectory of the second tracked object. The kinematic parameters may include, for example, motion speed, acceleration, etc. The kinematic parameter difference is the difference between the motion speed of the first tracked object and the motion speed of the second tracked object.

[0069] As an alternative example, Figure 6 As shown, A and B represent the tracking objects of adjacent cameras. When fusing the tracking queues corresponding to the adjacent cameras, it is necessary to determine whether the two are the same obstacle. Optionally, the two can be matched according to the following formula (4):

[0070] L 3 =w 1 *L position +w 2 *L size +w 3 *L trajectory +w 4 *L movement +w 5*L category (4)

[0071] Among them, L 3 represents the matching loss value between the first tracked object and the second tracked object. 1 、w 2 、w 3 、w 4 and w 5 They represent weight coefficients respectively. position represents the position difference between the first tracked object and the second tracked object, L size represents the size difference between the first tracked object and the second tracked object, L trajectory represents the trajectory difference between the first tracked object and the second tracked object, L movement represents the kinematic parameter difference between the first tracked object and the second tracked object, L category Indicates the column difference between the first tracked object and the second tracked object.

[0072] (2) For each first tracked object, if there is a second tracked object whose matching loss value with the first tracked object is less than or equal to a preset third threshold, determine that the first tracked object and the second tracked object are the same obstacle, and use the second tracked object as the object to be fused corresponding to the first tracked object.

[0073] (3) When there are at least two second tracked objects whose matching loss values ​​with the first tracked object are less than or equal to a preset third threshold, determine the second tracked object that matches the first tracked object from the at least two second tracked objects according to a preset matching method, determine that the first tracked object and the second tracked object that matches it are the same obstacle, and use the second tracked object that matches the first tracked object as the object to be fused corresponding to the first tracked object. The second tracked object that matches the first tracked object can be determined from the at least two second tracked objects according to formula (1) or (2).

[0074] (4) According to the tracking confidences of the first tracked object and the corresponding object to be fused, the first tracked object and the corresponding obstacle to be fused are fused to obtain a second tracking result.

[0075] refer to Figure 6, if it is determined through matching that A and B are the same obstacle, one of them is retained according to the tracking confidence. If the tracking confidence of the first tracking object is greater than or equal to the tracking confidence of the second tracking object, the tracking information of the first tracking object is retained as the fused tracking information. If the tracking confidence of the first tracking object is less than the tracking confidence of the second tracking object, the tracking information of the second tracking object is retained as the fused tracking information. Optionally, the tracking information of the first obstacle and the second obstacle can be weighted averaged according to the tracking confidence of the first tracking object and the tracking confidence of the second tracking object, and the tracking information after the weighted average is used as the fused tracking information.

[0076] Optionally, the tracking confidence of the tracking object (obstacle target) can be determined according to one or more of the following confidence functions: a first confidence function related to the number of tracking retention frames, a second confidence function related to detection confidence information, a third confidence function related to the obstacle distance, a fourth confidence function related to camera parameters, and a fifth confidence function related to kinematic parameters. As an optional example, the tracking confidence of the tracking object can be calculated according to the following formula (5):

[0077] C=w 5 *C age +w 6 *C det_confidence +w 7 *C distance +w 8 *C sensor +w 9 *

[0078] C motivation (5)

[0079] C represents the tracking confidence of the tracked object. 5 、w 6 、w 7 、w 8 and w 9 Represent the weight coefficients respectively. age represents the first confidence function related to the number of tracking retention frames, C det_confidence represents the second confidence function related to the detection confidence information, C distance represents the third confidence function related to the obstacle distance, C sensor represents the fourth confidence function related to the camera parameters, C motivation The first confidence function, the second confidence function, the third confidence function and the fourth confidence function can be designed according to the application scenario requirements, and the present invention does not limit this.

[0080] Step S102b: According to a preset fusion method and the detection information of the obstacle targets in the current frame images captured by the multiple monocular cameras, the obstacle targets in the current frame images captured by the multiple monocular cameras are fused to obtain a current frame fused obstacle target.

[0081] Step S102c: Track the obstacle target fused in the current frame according to a preset target tracking method and the detection information of the obstacle target fused in the current frame to obtain a third tracking result.

[0082] The fusion method of step S102b is the same as the fusion method of step S103a. Referring to the implementation process of step S103a, step S102b includes:

[0083] Compare the category information, orientation information, size information of the detection frame, and position information of the detection frame of the obstacle targets in the current frame images captured by the multiple monocular cameras to determine whether the obstacle targets in the current frame images captured by the multiple monocular cameras are the same obstacle;

[0084] If so, according to the tracking confidence of the obstacle target in the current frame image captured by each monocular camera, the obstacle targets corresponding to the same obstacle are fused to obtain the current frame fused obstacle target.

[0085] The fusion method of step S103b is the same as the fusion method of step S102a. Referring to the implementation process of step S102a, step S103b includes:

[0086] Creating a second tracking list, and recording the current frame fused obstacle target as a tracking object in the second tracking list;

[0087] According to the Kalman filtering method and the detection information of the obstacle target fused in the current frame, predicting the state of the obstacle target fused in the current frame, and determining the prediction information of the obstacle target fused in the current frame on the next frame image;

[0088] Detecting multiple next frame images captured by the multiple monocular cameras, determining obstacle targets on the multiple next frame images and detection information of the obstacle targets, and fusing the obstacle targets on the multiple next frame images to obtain a next frame fused obstacle target;

[0089] Determining whether the obstacle target fused in the next frame and the obstacle target fused in the current frame are the same obstacle according to the detection information of the obstacle target fused in the next frame and the prediction information of the obstacle target fused in the current frame on the next frame image;

[0090] If so, updating the current frame fused obstacle target in the second tracking list based on the detection information of the next frame fused obstacle target;

[0091] If not, the next frame fused obstacle target is recorded as a new tracking object in the second tracking list;

[0092] The number of unsuccessful matches of the tracking object recorded in the second tracking list is counted; when the number of unsuccessful matches is greater than or equal to a preset second threshold, the tracking object is deleted from the second tracking list.

[0093] The prediction information of the obstacle target fused in the current frame on the next frame image includes category prediction information, orientation prediction information, prediction size information of the detection frame, and prediction position information of the obstacle target fused in the current frame on the next frame image.

[0094] Optionally, determining whether the next frame fused obstacle target and the current frame fused obstacle target are the same obstacle according to the detection information of the next frame fused obstacle target and the prediction information of the current frame fused obstacle target on the next frame image includes:

[0095] The category information, orientation information, size information of the detection frame, and position information of the detection frame of the obstacle target in the next frame fused image are respectively compared with the category prediction information, orientation prediction information, predicted size information of the detection frame, and predicted position information of the detection frame of the obstacle target in the current frame fused image in the next frame image to determine whether the obstacle target in the next frame fused image and the obstacle target in the current frame fused image are the same obstacle.

[0096] Optionally, determining whether the next frame fused obstacle target and the current frame fused obstacle target are the same obstacle according to the detection information of the next frame fused obstacle target and the prediction information of the current frame fused obstacle target on the next frame image includes:

[0097] Using the preset recognition model, matching the next frame fused obstacle target with the current frame obstacle target to determine a similarity value between the next frame obstacle target and the current frame obstacle target;

[0098] According to the similarity value, the detection information of the next frame fused obstacle target and the prediction information of the current frame fused obstacle target on the next frame image, it is determined whether the next frame fused obstacle target and the current frame fused obstacle target are the same obstacle.

[0099] The target tracking method provided by the embodiment of the present invention detects the image captured by each monocular camera, determines the obstacle target in the image and the detection information of the obstacle target, tracks the obstacle target in the current frame image according to the preset target tracking method and the detection information of the obstacle target in the current frame image captured by the monocular camera, and obtains the first tracking result corresponding to the monocular camera; according to the preset fusion method, fuses the first tracking results corresponding to two adjacent monocular cameras to obtain the second tracking result; or, according to the detection information of the obstacle target in the current frame image captured by multiple monocular cameras, fuses the current frame images captured by multiple monocular cameras to obtain the current frame fused obstacle target; according to the preset target tracking method and the detection information of the obstacle target in the current frame fused image, tracks the obstacle target in the current frame fused image to obtain the third tracking result. This method performs obstacle detection on a single monocular camera, and then tracks / matches the detection results of each monocular camera to obtain the final obstacle detection information. It can obtain all-round and real-time information about the vehicle's surroundings and track its trajectory, which can effectively improve the accuracy and stability of target tracking and recognition. In addition, this method can use a lightweight model, reduce computing pressure, and reduce resource requirements and costs.

[0100] Figure 7 FIG. 2 shows a schematic diagram of the structure of a target tracking device provided by an embodiment of the present invention. Figure 7 As shown, the target tracking device 700 includes:

[0101] The detection module 701 detects each image captured by the monocular camera to determine the obstacle target in the image and the detection information of the obstacle target;

[0102] The tracking module 702 tracks the obstacle target in the current frame image for each monocular camera according to a preset target tracking method and the detection information of the obstacle target in the current frame image collected by the monocular camera, and obtains a first tracking result corresponding to the monocular camera;

[0103] A fusion module 703, which fuses the first tracking results corresponding to two adjacent monocular cameras according to a preset fusion method and the tracking information of the obstacle target in the first tracking result to obtain a second tracking result;

[0104] or,

[0105] The fusion module 702 is used to fuse the obstacle targets in the current frame images collected by the multiple monocular cameras according to a preset fusion method and the detection information of the obstacle targets in the current frame images collected by the multiple monocular cameras to obtain a current frame fused obstacle target;

[0106] The tracking module 03 is used to track the obstacle target fused in the current frame according to a preset target tracking method and the detection information of the obstacle target fused in the current frame to obtain a third tracking result.

[0107] Optionally, the detection module is used to: for each image captured by a monocular camera, input the image as input data into a preset first detection model to detect the image and determine the obstacle target in the image and the detection information of the obstacle target; or, according to the number of the multiple monocular cameras, set the number of images included in a single detection batch; according to the number of images included in the single detection batch, simultaneously input the multiple images captured by the multiple monocular cameras into a preset second detection model to simultaneously detect the multiple images and determine the obstacle target in each image and the detection information of the obstacle target.

[0108] Optionally, the tracking module is used to: for each monocular camera, create a first tracking list corresponding to the monocular camera, and record the obstacle target in the current frame image captured by the monocular camera as a tracking object in the first tracking list; predict the state of the obstacle target in the current frame image according to the Kalman filtering method and the detection information of the obstacle target in the current frame image captured by the monocular camera, and determine the prediction information of the obstacle target in the current frame image in the next frame image; detect the next frame image captured by the monocular camera, determine the obstacle target in the next frame image and the detection information of the obstacle target; and detect the next frame image according to the next frame The detection information of the obstacle target in the image and the prediction information are used to determine whether the obstacle target in the next frame image and the obstacle target in the current frame image are the same obstacle; if the obstacle target in the next frame image and the obstacle target in the current frame image are the same obstacle, the tracking information of the tracking object in the first tracking list is updated based on the detection information of the obstacle target in the next frame image; if the obstacle target in the next frame image and the obstacle target in the current frame image are not the same obstacle, the obstacle target in the next frame image is recorded as a new tracking object in the first tracking list.

[0109] Optionally, the tracking module is used to: count the number of unsuccessful matches of the tracking object recorded in the first tracking list; and delete the tracking object from the first tracking list when the number of unsuccessful matches is greater than or equal to a specified threshold.

[0110] Optionally, the prediction information of the obstacle target in the current frame image on the next frame image includes category prediction information, orientation prediction information, prediction size information of the detection frame, and prediction position information of the detection frame of the obstacle target in the current frame image on the next frame image;

[0111] The tracking module is used to respectively compare the category information, orientation information, detection frame size information, and detection frame position information of the obstacle target in the next frame image with the category prediction information, orientation prediction information, detection frame prediction size information, and detection frame prediction position information of the obstacle target in the current frame image in the next frame image to determine whether the obstacle target in the next frame image is the same obstacle as the obstacle target in the current frame image.

[0112] Optionally, the tracking module is used to: use a preset recognition model to match the obstacle target in the next frame image with the obstacle target in the current frame image, and determine the similarity value between the obstacle target in the next frame image and the obstacle target in the current frame image; and determine whether the obstacle target in the next frame image and the obstacle target in the current frame image are the same obstacle based on the similarity value, the detection information of the obstacle target in the next frame image and the prediction information.

[0113] Optionally, the first tracking result corresponding to the monocular camera is a tracking list corresponding to the monocular camera, and the tracking information of the obstacle target in the first tracking result includes: the position, size, trajectory, kinematic parameters and category of the tracking object in the tracking list; the fusion module is used to: respectively use the tracking objects in the two tracking lists corresponding to two adjacent monocular cameras as the first tracking object and the second tracking object, and calculate the matching loss value between the first tracking object and the second tracking object according to the position difference, size difference, trajectory difference, kinematic parameter difference and category difference between the first tracking object and the second tracking object; for each first tracking object, if there is a second tracking object whose matching loss value with the first tracking object is less than or equal to a preset third threshold, determine The first tracking object and the second tracking object are determined to be the same obstacle, and the second tracking object is used as the object to be fused corresponding to the first tracking object; when there are at least two second tracking objects whose matching loss values ​​with the first tracking object are less than or equal to a preset third threshold, a second tracking object matching the first tracking object is determined from the at least two second tracking objects according to a preset matching method, and the first tracking object and the second tracking object matching the first tracking object are determined to be the same obstacle, and the second tracking object matching the first tracking object is used as the object to be fused corresponding to the first tracking object; according to the tracking confidence of the first tracking object and the object to be fused corresponding to the first tracking object and the first tracking object, the first tracking object and the obstacle to be fused corresponding to the first tracking object are fused to obtain a second tracking result.

[0114] Optionally, the fusion module is used to: determine the tracking confidence of the obstacle target according to one or more of the following confidence functions: a first confidence function related to the number of tracking retention frames, a second confidence function related to detection confidence information, a third confidence function related to the obstacle distance, a fourth confidence function related to camera parameters, and a fifth confidence function related to kinematic parameters; if the tracking confidence of the first obstacle target is greater than or equal to the tracking confidence of the second obstacle target, retain the tracking information of the first obstacle target as the fused tracking information; or, if the tracking confidence of the first obstacle target is less than the tracking confidence of the second obstacle target, retain the tracking information of the second obstacle target as the fused tracking information; or, based on the tracking confidence of the first obstacle target and the tracking confidence of the second obstacle target, perform weighted averaging on the tracking information of the first obstacle and the second obstacle, and use the weighted average tracking information as the fused tracking information.

[0115] Optionally, the fusion module is used to: compare the category information, size information of the detection frame, and position information of the obstacle target in the current frame image captured by the multiple monocular cameras to determine whether the obstacle target in the current frame image captured by the multiple monocular cameras is the same obstacle; if so, according to the tracking confidence of the obstacle target in the current frame image captured by each of the monocular cameras, the obstacle targets corresponding to the same obstacle are fused to obtain the current frame fused obstacle target.

[0116] Optionally, the fusion module is used to: determine the tracking confidence of the obstacle target according to one or more of the following confidence functions: a first confidence function related to the number of tracking retention frames, a second confidence function related to detection confidence information, a third confidence function related to the obstacle distance, a fourth confidence function related to camera parameters, and a fifth confidence function related to kinematic parameters; if the tracking confidence of the first obstacle target is greater than or equal to the tracking confidence of the second obstacle target, retain the tracking information of the first obstacle target as the fused tracking information; or, if the tracking confidence of the first obstacle target is less than the tracking confidence of the second obstacle target, retain the tracking information of the second obstacle target as the fused tracking information; or, based on the tracking confidence of the first obstacle target and the tracking confidence of the second obstacle target, perform weighted averaging on the tracking information of the first obstacle and the second obstacle, and use the weighted average tracking information as the fused tracking information.

[0117] Optionally, the fusion module is used to: compare the category information, orientation information, detection frame size information, and detection frame position information of the obstacle targets in the current frame images captured by the multiple monocular cameras to determine whether the obstacle targets in the current frame images captured by the multiple monocular cameras are the same obstacle; if so, according to the tracking confidence of the obstacle targets in the current frame images captured by each of the monocular cameras, the obstacle targets corresponding to the same obstacle are fused to obtain the current frame fused obstacle targets.

[0118] Optionally, the tracking module is used to: create a second tracking list, record the obstacle target fused in the current frame as a tracking object in the second tracking list; predict the state of the obstacle target fused in the current frame according to the Kalman filtering method and the detection information of the obstacle target fused in the current frame, and determine the prediction information of the obstacle target fused in the current frame on the next frame image; detect multiple next frame images collected by the multiple monocular cameras, determine the obstacle targets on the multiple next frame images and the detection information of the obstacle targets, fuse the obstacle targets on the multiple next frame images, and obtain the obstacle target fused in the next frame; determine whether the obstacle target fused in the next frame and the obstacle target fused in the current frame are the same obstacle according to the detection information of the obstacle target fused in the next frame and the prediction information of the obstacle target fused in the current frame on the next frame image; if so, update the obstacle target fused in the current frame in the second tracking list based on the detection information of the obstacle target fused in the next frame; if not, record the obstacle target fused in the next frame as a new tracking object in the second tracking list.

[0119] Optionally, the tracking module is used to: count the number of unsuccessful matches of the tracking object recorded in the second tracking list; and delete the tracking object from the second tracking list when the number of unsuccessful matches is greater than or equal to a preset second threshold.

[0120] Optionally, the prediction information of the obstacle target fused in the current frame on the next frame image includes category prediction information, orientation prediction information, prediction size information of the detection frame, and prediction position information of the obstacle target fused in the current frame on the next frame image;

[0121] The tracking module is used to respectively compare the category information, orientation information, size information of the detection frame, and position information of the obstacle target in the next frame fused image with the category prediction information, orientation prediction information, predicted size information of the detection frame, and predicted position information of the detection frame of the obstacle target in the current frame fused image in the next frame image, so as to determine whether the obstacle target in the next frame fused image and the obstacle target in the current frame fused image are the same obstacle.

[0122] Optionally, the tracking module is used to: use the preset recognition model to match the next frame fused obstacle target with the current frame obstacle target to determine a similarity value between the next frame obstacle target and the current frame obstacle target; and determine whether the next frame fused obstacle target and the current frame fused obstacle target are the same obstacle based on the similarity value, detection information of the next frame fused obstacle target and prediction information of the current frame fused obstacle target on the next frame image.

[0123] The above device can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the target tracking method provided by the embodiment of the present invention.

[0124] Figure 8 FIG. 1 is a schematic diagram showing the structure of an electronic device according to an embodiment of the present invention. Figure 8 As shown, the electronic device includes:

[0125] Processor 801, communication interface 802, memory 803 and communication bus 804, wherein the processor 801, communication interface 802, memory 803 communicate with each other via the communication bus 804.

[0126] Memory 803, used for storing computer programs;

[0127] The processor 801 is used to execute the program stored in the memory 803, and implements the following steps:

[0128] Detecting the images captured by each monocular camera to determine obstacle targets in the images and detection information of the obstacle targets;

[0129] For each monocular camera, according to a preset target tracking method and detection information of the obstacle target in the current frame image collected by the monocular camera, the obstacle target in the current frame image is tracked to obtain a first tracking result corresponding to the monocular camera; according to a preset fusion method and the tracking information of the obstacle target in the first tracking result, the first tracking results corresponding to two adjacent monocular cameras are fused to obtain a second tracking result;

[0130] or,

[0131] According to a preset fusion method and detection information of obstacle targets in the current frame images collected by the multiple monocular cameras, the obstacle targets in the current frame images collected by the multiple monocular cameras are fused to obtain a current frame fused obstacle target; according to the preset target tracking method and detection information of the current frame fused obstacle target, the current frame fused obstacle target is tracked to obtain a third tracking result.

[0132] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0133] The communication interface is used for communication between the above terminal and other devices.

[0134] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0135] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0136] In another embodiment of the present invention, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer executes the target tracking method described in any one of the above embodiments.

[0137] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the target tracking method described in any one of the above embodiments.

[0138] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0139] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0140] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A target tracking method, characterized in that: The method includes a plurality of monocular cameras deployed on the same vehicle; the method also includes: Detecting the images captured by each monocular camera to determine obstacle targets in the images and detection information of the obstacle targets; For each monocular camera, according to a preset target tracking method and detection information of the obstacle target in the current frame image collected by the monocular camera, the obstacle target in the current frame image is tracked to obtain a first tracking result corresponding to the monocular camera; according to a preset fusion method and the tracking information of the obstacle target in the first tracking result, the first tracking results corresponding to two adjacent monocular cameras are fused to obtain a second tracking result; or, According to the preset fusion method and the detection information of the obstacle target in the current frame image collected by the multiple monocular cameras, the obstacle target in the current frame image collected by the multiple monocular cameras is fused to obtain the current frame fused obstacle target; according to the preset target tracking method and the detection information of the current frame fused obstacle target, the current frame fused obstacle target is tracked to obtain a third tracking result.

2. The method according to claim 1, characterized in that The detecting of the image captured by each monocular camera to determine the obstacle target in the image and the detection information of the obstacle target includes: For each image captured by the monocular camera, the image is input as input data into a preset first detection model to detect the image and determine an obstacle target in the image and detection information of the obstacle target; or, According to the number of the plurality of monocular cameras, setting the number of images included in a single detection batch; According to the number of images included in the single detection batch, the multiple images captured by the multiple monocular cameras are simultaneously input into a preset second detection model to simultaneously detect the multiple images and determine the obstacle targets in each image and the detection information of the obstacle targets.

3. The method according to claim 1, characterized in that The obstacle target detection information includes one or more of the following: size information of the detection frame, position information of the detection frame, orientation information of the obstacle target, category information of the obstacle target and detection confidence information. The detection frame is a 2D frame or a 3D frame.

4. The method according to claim 1 or 3, characterized in that: The method of tracking the obstacle target in the current frame image according to the preset target tracking method and the detection information of the obstacle target in the current frame image collected by the monocular camera to obtain a first tracking result corresponding to the monocular camera includes: For each monocular camera, create a first tracking list corresponding to the monocular camera, and record the obstacle target in the current frame image captured by the monocular camera as a tracking object in the first tracking list; According to the Kalman filtering method and the detection information of the obstacle target in the current frame image collected by the monocular camera, the state of the obstacle target in the current frame image is predicted, and the prediction information of the obstacle target in the current frame image in the next frame image is determined; Detecting a next frame of image captured by the monocular camera to determine an obstacle target in the next frame of image and detection information of the obstacle target; determining, according to the detection information of the obstacle target in the next frame image and the prediction information, whether the obstacle target in the next frame image and the obstacle target in the current frame image are the same obstacle; When the obstacle target in the next frame image is the same obstacle as the obstacle target in the current frame image, updating the tracking information of the tracking object in the first tracking list based on the detection information of the obstacle target in the next frame image; In the case that the obstacle target in the next frame image is not the same obstacle as the obstacle target in the current frame image, the obstacle target in the next frame image is recorded in the first tracking list as a new tracking object.

5. The method according to claim 4, characterized in that The method further comprises: The number of unsuccessful matches of the tracking object recorded in the first tracking list is counted; when the number of unsuccessful matches is greater than or equal to a specified threshold, the tracking object is deleted from the first tracking list.

6. The method according to claim 4, characterized in that The prediction information of the obstacle target in the current frame image on the next frame image includes category prediction information, orientation prediction information, prediction size information of the detection frame, and prediction position information of the detection frame of the obstacle target in the current frame image on the next frame image; The determining, based on the detection information of the obstacle target in the next frame image and the prediction information, whether the obstacle target in the next frame image and the obstacle target in the current frame image are the same obstacle includes: The category information, orientation information, size information of the detection frame, and position information of the detection frame of the obstacle target in the next frame image are respectively compared with the category prediction information, orientation prediction information, predicted size information of the detection frame, and predicted position information of the detection frame of the obstacle target in the current frame image in the next frame image to determine whether the obstacle target in the next frame image is the same obstacle as the obstacle target in the current frame image.

7. The method according to claim 4 or 6, characterized in that: The determining, based on the detection information of the obstacle target in the next frame image and the prediction information, whether the obstacle target in the next frame image and the obstacle target in the current frame image are the same obstacle includes: Using a preset recognition model, matching the obstacle target in the next frame image with the obstacle target in the current frame image, and determining a similarity value between the obstacle target in the next frame image and the obstacle target in the current frame image; According to the similarity value, the detection information of the obstacle target in the next frame image and the prediction information, it is determined whether the obstacle target in the next frame image and the obstacle target in the current frame image are the same obstacle.

8. The method according to claim 4, characterized in that The first tracking result corresponding to the monocular camera is a tracking list corresponding to the monocular camera, and the tracking information of the obstacle target in the first tracking result includes: the position, size, trajectory, kinematic parameters and category of the tracking object in the tracking list; the first tracking results corresponding to two adjacent monocular cameras are fused according to the preset fusion method and the tracking information of the obstacle target in the first tracking result to obtain the second tracking result, including: The tracking objects in the two tracking lists corresponding to the two adjacent monocular cameras are respectively taken as the first tracking object and the second tracking object, and the matching loss value between the first tracking object and the second tracking object is calculated according to the position difference, size difference, trajectory difference, kinematic parameter difference and category difference between the first tracking object and the second tracking object; For each first tracked object, if there is a second tracked object whose matching loss value with the first tracked object is less than or equal to a preset third threshold, determine that the first tracked object and the second tracked object are the same obstacle, and use the second tracked object as the object to be fused corresponding to the first tracked object; In the case that there are at least two second tracked objects whose matching loss values ​​with the first tracked object are less than or equal to a preset third threshold, determining, according to a preset matching method, a second tracked object matching the first tracked object from the at least two second tracked objects, determining that the first tracked object and the second tracked object matching the first tracked object are the same obstacle, and using the second tracked object matching the first tracked object as an object to be fused corresponding to the first tracked object; According to the tracking confidences of the first tracked object and the corresponding object to be fused, the first tracked object and the corresponding obstacle to be fused are fused to obtain a second tracking result.

9. The method according to claim 8, characterized in that According to the tracking confidence of the first obstacle target and the tracking confidence of the second obstacle target, the first obstacle target and the second obstacle target are fused, including: Determining the tracking confidence of the obstacle target according to one or more of the following confidence functions: a first confidence function related to the number of tracking retention frames, a second confidence function related to detection confidence information, a third confidence function related to the obstacle distance, a fourth confidence function related to camera parameters, and a fifth confidence function related to kinematic parameters; If the tracking confidence of the first obstacle target is greater than or equal to the tracking confidence of the second obstacle target, the tracking information of the first obstacle target is retained as the fused tracking information; or, If the tracking confidence of the first obstacle target is less than the tracking confidence of the second obstacle target, the tracking information of the second obstacle target is retained as the fused tracking information; or, According to the tracking confidence of the first obstacle target and the tracking confidence of the second obstacle target, a weighted average is performed on the tracking information of the first obstacle and the second obstacle, and the tracking information after the weighted average is used as the fused tracking information.

10. The method according to claim 9, characterized in that The step of fusing the obstacle targets in the current frame images collected by the multiple monocular cameras according to the preset fusion method and the detection information of the obstacle targets in the current frame images collected by the multiple monocular cameras to obtain the current frame fused obstacle targets includes: Compare the category information, orientation information, size information of the detection frame, and position information of the detection frame of the obstacle targets in the current frame images captured by the multiple monocular cameras to determine whether the obstacle targets in the current frame images captured by the multiple monocular cameras are the same obstacle; If so, according to the tracking confidence of the obstacle target in the current frame image captured by each monocular camera, the obstacle targets corresponding to the same obstacle are fused to obtain the current frame fused obstacle target.

11. The method according to claim 9, characterized in that The step of tracking the obstacle target fused in the current frame according to the preset target tracking method and the detection information of the obstacle target fused in the current frame to obtain a third tracking result includes: Creating a second tracking list, and recording the current frame fused obstacle target as a tracking object in the second tracking list; According to the Kalman filtering method and the detection information of the obstacle target fused in the current frame, predicting the state of the obstacle target fused in the current frame, and determining the prediction information of the obstacle target fused in the current frame on the next frame image; Detecting multiple next frame images captured by the multiple monocular cameras, determining obstacle targets on the multiple next frame images and detection information of the obstacle targets, and fusing the obstacle targets on the multiple next frame images to obtain a next frame fused obstacle target; Determining whether the obstacle target fused in the next frame and the obstacle target fused in the current frame are the same obstacle according to the detection information of the obstacle target fused in the next frame and the prediction information of the obstacle target fused in the current frame on the next frame image; If so, updating the current frame fused obstacle target in the second tracking list based on the detection information of the next frame fused obstacle target; If not, the obstacle target fused in the next frame is recorded in the second tracking list as a new tracking object.

12. The method according to claim 11, characterized in that The method further comprises: The number of unsuccessful matches of the tracking object recorded in the second tracking list is counted; when the number of unsuccessful matches is greater than or equal to a preset second threshold, the tracking object is deleted from the second tracking list.

13. The method according to claim 12, characterized in that The prediction information of the obstacle target fused in the current frame on the next frame image includes category prediction information, orientation prediction information, prediction size information of the detection frame, and prediction position information of the obstacle target fused in the current frame on the next frame image; The determining, based on the detection information of the next frame fused obstacle target and the prediction information of the current frame fused obstacle target on the next frame image, whether the next frame fused obstacle target and the current frame fused obstacle target are the same obstacle includes: The category information, orientation information, size information of the detection frame, and position information of the obstacle target in the next frame fused image are respectively compared with the category prediction information, orientation prediction information, predicted size information of the detection frame, and predicted position information of the detection frame of the obstacle target in the current frame fused image in the next frame image to determine whether the obstacle target in the next frame fused image and the obstacle target in the current frame fused image are the same obstacle.

14. The method according to claim 13, characterized in that Determining whether the next frame fused obstacle target and the current frame fused obstacle target are the same obstacle according to the detection information of the next frame fused obstacle target and the prediction information of the current frame fused obstacle target on the next frame image includes: Using the preset recognition model, matching the next frame fused obstacle target with the current frame obstacle target to determine a similarity value between the next frame obstacle target and the current frame obstacle target; According to the similarity value, the detection information of the next frame fused obstacle target and the prediction information of the current frame fused obstacle target on the next frame image, it is determined whether the next frame fused obstacle target and the current frame fused obstacle target are the same obstacle.

15. A target tracking device, characterized in that: The device includes a plurality of monocular cameras deployed on the same vehicle; the device also includes: A detection module, used to detect the images captured by each monocular camera, determine the obstacle targets in the images and the detection information of the obstacle targets; A tracking module, configured to track the obstacle target in the current frame image for each monocular camera according to a preset target tracking method and detection information of the obstacle target in the current frame image collected by the monocular camera, and obtain a first tracking result corresponding to the monocular camera; The fusion module is used to fuse the first tracking results corresponding to two adjacent monocular cameras according to a preset fusion method and the tracking information of the obstacle target in the first tracking result to obtain a second tracking result. or, The fusion module is used to fuse the obstacle targets in the current frame images collected by the multiple monocular cameras according to a preset fusion method and the detection information of the obstacle targets in the current frame images collected by the multiple monocular cameras to obtain a current frame fused obstacle target; The tracking module is used to track the obstacle target fused in the current frame according to a preset target tracking method and the detection information of the obstacle target fused in the current frame to obtain a third tracking result.

16. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, configured to implement the method according to any one of claims 1 to 14 when executing a program stored in a memory.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

Images