Target tracking method and device

Abstract

The invention provides a target tracking method and device, and the method comprises the steps: obtaining the detection frame information corresponding to a to-be-detected target in the current frame image data, and the vehicle motion information corresponding to the detection frame information; respectively determining coordinate information of a historical target and a to-be-detected target in the adjacent frame image data in a self-vehicle coordinate system; predicting the predicted position of the historical target in the self-vehicle coordinate system at the current moment; obtaining pixel coordinates and prediction frame information of the historical target according to the target size of the historical target; associating the to-be-detected target and the historical target of the same target type based on a preset association algorithm and the target type; and after the association is successful, updating the size of the detection frame corresponding to the to-be-detected target. On the basis, the purposes of supporting various scenes, not depending on appearance characteristics and particularly providing a stable tracking result for a small target are achieved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving visual perception, and in particular to a target tracking method and device. Background Art

[0002] In recent years, with the research and development of artificial intelligence and autonomous driving by major manufacturers and research institutions, all levels of autonomous driving related technologies have developed rapidly. In the perception technology of autonomous driving vehicles, target detection and continuous frame tracking are the key to whether autonomous driving vehicles can achieve continuous and accurate perception of targets. The correctness of the perception results directly affects the behavioral decision-making of autonomous driving and is an important factor in ensuring safe driving.

[0003] In the existing multi-target tracking technology, the tracking method often relies on the target appearance features or the target motion model in the image domain to track the target, or uses edge computing devices to track the target. In the application of autonomous driving scenarios, it is easy to encounter situations where the appearance of each target is similar and there are a large number of small targets. The tracking method that relies on the target appearance features or the target motion model in the image domain is difficult to track, which affects the tracking robustness and has a large noise in the target corresponding motion prediction results. The use of edge computing devices for target tracking has tight computing resources and is not convenient for widespread use.

[0004] Therefore, designing a target tracking method with enhanced tracking performance is an urgent problem to be solved. Summary of the invention

[0005] In view of this, the present invention provides a target tracking method and device, which have better robustness to support various scenarios, are independent of appearance features, and provide stable tracking results especially for small targets.

[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0007] A first aspect of an embodiment of the present invention discloses a target tracking method, the method comprising:

[0008] Acquire current frame image data and vehicle motion information corresponding to the current frame image data;

[0009] Detecting at least one target to be detected in the current frame image data and detection frame information corresponding to the target to be detected based on a preset detection network, wherein the detection frame information includes a size of the detection frame and a target category;

[0010] Respectively determining the coordinate information of the historical target and the target to be detected in the adjacent frame image data in the vehicle coordinate system;

[0011] In the ego-vehicle coordinate system, based on the coordinate information of the historical target and the target to be detected, and the vehicle motion information corresponding to the adjacent frame image data, predict the predicted position of the historical target in the ego-vehicle coordinate system at the current moment;

[0012] According to the target size of the historical target, the predicted position is converted into a pixel coordinate system to obtain the pixel coordinates and prediction frame information of the historical target;

[0013] Searching for the target to be detected in the current frame image with the pixel coordinates as fixed points, and associating the target to be detected and the historical targets of the same target type based on a preset association algorithm and target category;

[0014] If the association is successful, the size of the detection frame corresponding to the target to be detected is updated based on the predicted frame information.

[0015] Preferably, obtaining the current frame image data and the vehicle motion information corresponding to the current frame image data includes:

[0016] continuously acquiring image data at a first preset frequency, and continuously acquiring vehicle motion data at a second preset frequency;

[0017] Determine a sampling image timestamp of the current frame image data, and vehicle motion data corresponding to a sampling motion timestamp having a minimum difference with the sampling image timestamp;

[0018] The current frame image data and the vehicle motion data are time compensated to extract vehicle motion information contained in the vehicle motion data.

[0019] Preferably, respectively determining the coordinate information of the historical target and the target to be detected in the adjacent frame image data in the vehicle coordinate system includes:

[0020] Respectively analyzing the historical targets and the targets to be detected in the adjacent frame image data to determine the depth information corresponding to the historical targets and the targets to be detected in the vehicle coordinate system, wherein the current frame image data and the adjacent frame image data are collected by a depth camera;

[0021] or,

[0022] For the target to be detected, parsing the detection frame information to obtain the target type and detection frame size of the target to be detected;

[0023] Obtaining a preset size of the target to be detected according to the target type, each target type having a corresponding preset size;

[0024] Convert the size of the detection frame and the preset size from the pixel coordinate system to the camera coordinate system to obtain the depth information of the target to be detected in the camera coordinate system;

[0025] Convert the depth information of the target to be detected in the camera coordinate system to the vehicle coordinate system to obtain the depth information of the target to be detected in the vehicle coordinate system and the coordinate information of the target in the vehicle coordinate system;

[0026] For the historical target in the adjacent frame image data, the coordinates of the historical target in the pixel coordinate system are acquired and converted into the camera coordinate system to obtain the depth information of the historical target in the camera coordinate system;

[0027] The depth information of the historical target in the camera coordinate system is converted into the vehicle coordinate system to obtain the depth information of the historical target in the vehicle coordinate system and the coordinate information of the historical target in the vehicle coordinate system.

[0028] Preferably, in the ego-vehicle coordinate system, based on the coordinate information of the historical target and the target to be detected, and the vehicle motion information corresponding to the adjacent frame image data, predicting the predicted position of the historical target in the ego-vehicle coordinate system at the current moment includes:

[0029] In the vehicle coordinate system, according to the coordinate information of the historical target and the target to be detected, and the vehicle motion information corresponding to the adjacent frame image data, use Predict the predicted position (x) of the historical target in the vehicle coordinate system at the current moment t ,y t , z t );

[0030] Among them, t represents the time, Δt represents the time difference between adjacent frames, (x t-Δt ,y t-Δt , z t-Δt ) represents the coordinates of the historical target in the vehicle coordinate system at time t-Δt, R represents the rotation matrix of the vehicle within the Δt interval, and T represents the translation matrix generated by the vehicle movement within the Δt time difference.

[0031] Preferably, the method further comprises:

[0032] If the association fails and the target to be detected is a newly added target, the target type of the newly added target is determined, and the preset size of the newly added target is obtained based on the target type, and the process returns to the step of acquiring the current frame image data.

[0033] Preferably, the method further comprises:

[0034] If the association fails, and the target to be detected is not a newly added target, determining the number of association failures corresponding to the target to be detected;

[0035] Comparing the number of association failures with an exit threshold;

[0036] If the number of association failures is less than or equal to the exit threshold, return to the step of obtaining the current frame image data;

[0037] If the number of association failures is greater than the exit threshold, exit target tracking.

[0038] Preferably, updating the size of the detection frame corresponding to the target to be detected based on the prediction frame information includes:

[0039] Obtaining the size of the prediction box in the prediction box information, and calculating a change value between the size of the detection box corresponding to the target to be detected and the size of the prediction box;

[0040] Calculate based on the dimensional variation coefficient formula and the variation value to obtain the dimensional variation coefficient;

[0041] According to the size change coefficient and the update formula, an updated size of the detection frame corresponding to the target to be detected is obtained;

[0042] Wherein, the dimensional variation coefficient formula is:

[0043]

[0044]

[0045] Among them, δ w is the detection frame width size variation coefficient, δ h is the detection frame length size variation coefficient, w track-box2d is the tracking result of the detection box width, w measure-bpx2d is the observation result of the detection box width, h track-box2d is the tracking result of the detection box length, h measure-box2d is the observation result of the detection box length, α and β are weight coefficients;

[0046] The update formula is:

[0047] w=δ w *w track ;

[0048] h=δ h *h track ;

[0049] w is the actual width of the target to be detected after update, h is the actual length of the target to be detected after update, w track,h track Indicates the size of the detection frame in the detection frame information.

[0050] A second aspect of an embodiment of the present invention discloses a target tracking device, the device comprising:

[0051] A first acquisition unit, used to acquire current frame image data and vehicle motion information corresponding to the current frame image data;

[0052] A second acquisition unit, configured to detect at least one target to be detected in the current frame image data and detection frame information corresponding to the target to be detected based on a preset detection network, wherein the detection frame information includes a size of the detection frame and a target category;

[0053] A prediction unit, used to respectively determine the coordinate information of the historical target and the target to be detected in the adjacent frame image data in the vehicle coordinate system; in the vehicle coordinate system, based on the coordinate information of the historical target and the target to be detected, and the vehicle motion information corresponding to the adjacent frame image data, predict the predicted position of the historical target in the vehicle coordinate system at the current moment;

[0054] A conversion unit, configured to convert the predicted position into a pixel coordinate system according to a target size of the historical target, so as to obtain pixel coordinates and prediction frame information of the historical target;

[0055] The association unit is used to search for the target to be detected in the current frame image with the pixel coordinates as a fixed point, and associate the target to be detected and the historical targets of the same target type based on a preset association algorithm and target category; if the association is successful, the size of the detection frame corresponding to the target to be detected is updated based on the predicted frame information.

[0056] Optionally, the first acquiring unit includes:

[0057] A first acquisition module is used to continuously acquire image data at a first preset frequency and continuously acquire vehicle motion data at a second preset frequency; determine a sampling image timestamp of the current frame image data and vehicle motion data corresponding to a sampling motion timestamp having a minimum difference with the sampling image timestamp;

[0058] The motion compensation module is used to perform time compensation on the current frame image data and the vehicle motion data, and extract the vehicle motion information contained in the vehicle motion data.

[0059] Optionally, the prediction unit for respectively determining the coordinate information of the historical target in the adjacent frame image data and the target to be detected in the vehicle coordinate system includes:

[0060] A first parsing unit, configured to parse the historical targets and the target to be detected in the adjacent frame image data respectively, and determine the depth information corresponding to the historical targets and the target to be detected in the vehicle coordinate system, wherein the current frame image data and the adjacent frame image data are collected by a depth camera;

[0061] and / or,

[0062] The target processing unit to be detected is used to parse the detection frame information to obtain the target type and detection frame size of the target to be detected; obtain the preset size of the target to be detected according to the target type, and each target type has a corresponding preset size; convert the size and preset size of the detection frame from the pixel coordinate system to the camera coordinate system to obtain the depth information of the target to be detected in the camera coordinate system; convert the depth information of the target to be detected in the camera coordinate system to the vehicle coordinate system to obtain the depth information of the target to be detected in the vehicle coordinate system and the coordinate information in the vehicle coordinate system;

[0063] The historical target processing unit is used to obtain and convert the coordinates of the historical target in the pixel coordinate system to the camera coordinate system to obtain the depth information of the historical target in the camera coordinate system; convert the depth information of the historical target in the camera coordinate system to the vehicle coordinate system to obtain the depth information of the historical target in the vehicle coordinate system and the coordinate information in the vehicle coordinate system.

[0064] Compared with the prior art, the present invention includes the following advantages: by obtaining the detection frame information corresponding to the target to be detected in the current frame image data, and the corresponding vehicle motion information; respectively determining the coordinate information of the historical target and the target to be detected in the adjacent frame image data in the vehicle coordinate system; and predicting the predicted position of the historical target in the vehicle coordinate system at the current moment; obtaining the pixel coordinates and prediction frame information of the historical target according to the target size of the historical target; based on the preset association algorithm and target category, associating the target to be detected and the historical target of the same target type; and after the association is successful, updating the size of the detection frame corresponding to the target to be detected. The method disclosed in the present invention is not limited to various scenarios, and its tracking performance is safe and reliable, and has better robustness. Based on this, the purpose of supporting various scenarios, not relying on appearance features, and providing stable tracking results, especially for small targets, is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0066] Figure 1 A method flow chart of a target tracking method provided by an embodiment of the present invention;

[0067] Figure 2 A method flow chart of another target tracking method provided by an embodiment of the present invention;

[0068] Figure 3 The position of the traffic light perceived in the previous frame in the example provided by the embodiment of the present invention;

[0069] Figure 4 The position of the traffic light after motion compensation in the example provided in the embodiment of the present invention;

[0070] Figure 5 The result of finally tracking the traffic light in the example provided in the embodiment of the present invention;

[0071] Figure 6 A device structure diagram of a target tracking device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0074] The present invention can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or devices, etc.

[0075] The embodiment of the present invention provides a target tracking method, which can be applied to a variety of hardware platforms, and the execution subject can be a processor of various mobile devices; preferably, the hardware platform can also include a calibrated monocular camera sensor and an IMU sensor that can record the motion information of the vehicle. The method flow chart of the method is as follows: Figure 1 As shown, specifically including:

[0076] S101: Acquire current frame image data and vehicle motion information corresponding to the current frame image data.

[0077] In S101, the current frame image data includes a sampling image timestamp.

[0078] Optionally, using a hardware platform that includes a calibrated monocular camera sensor and an IMU sensor that can record the vehicle's motion information, various sensor data can be sampled. The sensor data includes image information of the visual sensor and motion information of an inertial measurement unit (IMU, Inertial Measurement Unit). The image information of the visual sensor is image data, and the motion information of the inertial measurement unit is vehicle motion data.

[0079] The process of acquiring image information of the visual sensor is as follows: through a calibrated camera (the camera contains a visual sensor), the image information of the road conditions where the vehicle is located is continuously sampled at a fixed frequency, and the camera's internal parameters, external parameters and sampling timestamp are recorded. The sampling timestamp here is the sampling image timestamp.

[0080] The process of acquiring the motion information of the inertial measurement unit is as follows: the IMU sensor continuously acquires the vehicle's motion information at a fixed frequency that is not less than the camera sampling frequency, including vehicle speed, yaw angular velocity information and sampling timestamp. The sampling timestamp here is the sampling vehicle timestamp / IMU sensor timestamp.

[0081] Optionally, each sampled sensor data includes image data of a current frame and image data of each previous frame. Among the image data of each previous frame, image data of adjacent frames adjacent to the encoding of the current frame are determined, and then vehicle motion data corresponding to a sampled motion timestamp closest to the sampled image timestamp is obtained. The image data and the vehicle motion data are time compensated, that is, the image data and the vehicle motion data are preprocessed to quickly extract the vehicle motion information contained in the vehicle motion data.

[0082] Optionally, to determine the closest sampled image timestamp and sampled motion timestamp, the time differences between the timestamps may be determined first, the minimum time difference may be determined therefrom, and the two timestamps with the minimum time difference may be determined as the two closest timestamps.

[0083] In some embodiments of the present invention, obtaining current frame image data and vehicle motion information corresponding to the current frame image data includes:

[0084] First, image data is continuously collected at a first preset frequency, and vehicle motion data is continuously acquired at a second preset frequency.

[0085] Secondly, the sampling image timestamp of the current frame image data and the vehicle motion data corresponding to the sampling motion timestamp with the smallest difference with the sampling image timestamp are determined.

[0086] Finally, the current frame image data and the vehicle motion data are time compensated to extract the vehicle motion information contained in the vehicle motion data.

[0087] S102: Detect at least one target to be detected in the current frame image data and detection frame information corresponding to the target to be detected based on a preset detection network.

[0088] In S102, the detection frame information includes the size of the detection frame and the target category. The size of the detection frame includes length and width information. The target category refers to the object category of the target to be detected, such as a traffic light.

[0089] In S102, the detection network may be a YOLO network or a CNN network, etc.

[0090] It should be noted that the current frame image data contains one or more targets to be detected.

[0091] In the specific process of executing S102, a deep learning detection network is used to detect each target to be detected in the current frame image data to obtain detection frame information corresponding to each target to be detected.

[0092] S103: respectively determining the coordinate information of the historical targets and the targets to be detected in the adjacent frame image data in the vehicle coordinate system.

[0093] In S103, the adjacent frame here means adjacent to the current frame in coding relationship, specifically, it describes the previous frame of the current frame in coding relationship. The adjacent frame image data contains the historical objects in each adjacent frame.

[0094] Optionally, the adjacent frame image data corresponding to the current image data and the vehicle motion information record corresponding to the adjacent frame image data may be determined by sampling the image timestamp.

[0095] In S103, the vehicle coordinate system refers to a three-dimensional coordinate system centered on the vehicle, with the positive x direction being forward, the positive y direction being leftward, and the positive z direction being upward;

[0096] It should be noted that, depending on the different devices for collecting images, different methods may be used in the specific execution of S103.

[0097] In some embodiments of the present invention, if the current frame image data and the adjacent frame image data are collected by a depth camera, the depth information of the target to be detected in the current frame image data and the historical target in the adjacent frame image data in the vehicle coordinate system can be obtained through the depth camera.

[0098] The specific process of executing S103 is: respectively analyzing the historical targets and the target to be detected in the adjacent frame image data, and determining the corresponding depth information of the historical targets and the target to be detected in the vehicle coordinate system.

[0099] In some embodiments of the present invention, if the current frame image data and the adjacent frame image data are collected by a common vehicle-mounted camera, the specific process of executing S103 is:

[0100] First, for the target to be detected, the detection frame information is parsed to obtain the target type and detection frame size of the target to be detected.

[0101] Secondly, the preset size of the target to be detected is obtained according to the target type.

[0102] Each target type has a corresponding preset size, which is a priori size information of the target type.

[0103] It should be noted that the prior size information can be set according to the prior data corresponding to each preset target or target type. It can be understood that each prior data can be obtained by counting the various size information corresponding to each target or target type, or by counting the various size information corresponding to each target. For example, the prior length and width of the traffic light are set to (1.3m, 0.7m), and the prior length and width of the traffic light are the various size information corresponding to a target type.

[0104] Secondly, the size of the detection frame and the preset size are converted from the pixel coordinate system to the camera coordinate system to obtain the depth information of the target to be detected in the camera coordinate system.

[0105] Secondly, the depth information of the target to be detected in the camera coordinate system is converted to the vehicle coordinate system to obtain the depth information of the target to be detected in the vehicle coordinate system and the coordinate information in the vehicle coordinate system;

[0106] Secondly, for the historical targets in the adjacent frame image data, the coordinates of the historical targets in the pixel coordinate system are obtained and converted to the camera coordinate system to obtain the depth information of the historical targets in the camera coordinate system.

[0107] Finally, the depth information of the historical target in the camera coordinate system is converted to the vehicle coordinate system to obtain the depth information of the historical target in the vehicle coordinate system and the coordinate information in the vehicle coordinate system.

[0108] Based on the above content, in the process of obtaining the depth information of the target to be detected and the historical target in the vehicle coordinate system, if the visual sensor adopts a depth camera, the depth information of the target to be detected and the historical target can be obtained through the depth camera.

[0109] If the visual sensor uses an ordinary vehicle-mounted camera, it is necessary to parse the detection frame information to determine the target type corresponding to the target to be detected, and determine the preset size of the target to be detected according to the pre-set a priori size information of the target to be detected under the target type. The size of the detection frame and the preset size are used to estimate the depth information of the target to be detected in the vehicle coordinate system through the preset transformation matrix between the vehicle coordinate system and the pixel coordinate system. For historical targets, the depth information of historical targets in the vehicle coordinate system is also estimated through the preset transformation matrix between the vehicle coordinate system and the pixel coordinate system.

[0110] It should be noted that the process of converting the detection frame information to the vehicle coordinate system can be understood as mapping the detection frame information from the 2D plane to the 3D plane. First, convert the image or pixel coordinate system to the camera coordinate system (the second depth z c ), and then convert the depth from the camera coordinate system to the vehicle coordinate system (the first depth z). The upper left point of the image is taken as the origin, the positive direction of u is to the right, and the positive direction of v is downward. The pixel coordinate system refers to the coordinate system of the 2D detection box on the image.

[0111] Specifically, the detection frame information is converted to the vehicle coordinate system as follows:

[0112]

[0113]

[0114] In formula (1) and formula (2), (u, v) represents the coordinate position of the target to be detected in the pixel coordinate system, (x c ,y c , z c ) represents the coordinate position of the target to be detected in the camera coordinate system, (x, y, z) represents the coordinate position of the target to be detected in the vehicle coordinate system, I is the camera intrinsic parameter, E is the camera extrinsic parameter, and the camera intrinsic parameter and camera extrinsic parameter are obtained by camera calibration.

[0115] S104: In the ego-vehicle coordinate system, based on the coordinate information of the historical target and the target to be detected, and the vehicle motion information corresponding to the adjacent frame image data, predict the predicted position of the historical target in the ego-vehicle coordinate system at the current moment.

[0116] In the specific execution of S104, when the coordinate information of the historical target and the target to be detected in the vehicle coordinate system has been confirmed, the predicted position of the historical target in the vehicle coordinate system at the current moment can be predicted based on formula (3) in combination with the vehicle motion information of the adjacent frame image data.

[0117] Specifically, formula (3) is as follows:

[0118]

[0119] In formula (3), (x t ,y t , z t ) indicates the predicted position of the historical target at time t in the ego vehicle coordinate system, (x t-Δt ,y t-Δt , z t-Δt ) is the coordinate of the historical target in the ego vehicle coordinate system at time t-△t, R represents the rotation matrix of the ego vehicle in the △t interval, R -1 represents the inverse matrix of the rotation matrix of the ego vehicle within the △t interval, and T represents the translation matrix generated by the movement of the ego vehicle within the △t time difference.

[0120] S105: According to the target size of the historical target, the predicted position is converted into a pixel coordinate system to obtain the pixel coordinates and prediction box information of the historical target.

[0121] In the specific execution process of S105, based on formula (3), formula (2) and formula (1), combined with the target size of the historical target, the predicted position of the historical target in the vehicle coordinate system at the current moment is converted to the pixel coordinate system, and the pixel coordinates (u′, v′) and prediction box information of the historical target are obtained.

[0122] S106: Search for the target to be detected in the current frame image with the pixel coordinates as the fixed point, and associate the target to be detected and the historical targets of the same target type based on the preset association algorithm and target category; if the association is successful, execute S107; if the association fails, execute S108.

[0123] In S106, the preset association algorithm includes but is not limited to an optimal matching algorithm such as a Hungarian algorithm and a Kuh Munkras algorithm.

[0124] In the specific process of executing S106, the target to be detected is searched in the current frame image with the pixel coordinates of the historical target as a fixed point, and the target to be detected and the historical target belonging to the same target type are associated based on a preset association algorithm and target category.

[0125] If the association is successful, it means that the historical object in the adjacent frame image data and the target to be detected in the current frame image data are the same target object, and S107 is executed.

[0126] If the association fails, it means that the historical object in the adjacent frame image data and the target to be detected in the current frame image data are not the same target object, and S108 is executed.

[0127] S107: Update the size of the detection frame corresponding to the target to be detected based on the prediction frame information.

[0128] In the specific execution of S107, the size change coefficient is calculated through the size change of the 2D-box of the same target object, the change of the actual size of the target is estimated through the size change coefficient, and the detection box size information is updated through a filtering algorithm such as Kalman filtering.

[0129] Specifically, the actual size of the detection frame of the target object can be updated by smoothing the variable width (width) and height (height) of the filter, wherein the size change coefficient formula is shown in formula (4) and formula (5):

[0130]

[0131]

[0132] In formula (4) and formula (5), δ w is the detection frame width size variation coefficient, δ h is the detection frame length size variation coefficient, w track-box2d is the tracking result of the detection box width, w measure-box2d is the observation result of the detection box width, h track-box2d is the tracking result of the detection box length, h measure-box2d is the observation result of the detection box length, and α and β are weight coefficients.

[0133] The update formula is shown in formula (6) and formula (7):

[0134] w=δ w *w track (6)

[0135] h=δ h *h track (7)

[0136] In formula (6) and formula (7), w is the updated actual width of the target to be detected, h is the updated actual length of the target to be detected, and w is track ,h track Indicates the size of the detection frame in the detection frame information.

[0137] Among them, the Kalman filter parameters are set as follows:

[0138] Take w, h, a as the input of Kalman filter, a represents the aspect ratio, and its motion vector is (v w ,v h ,v a ), calculate the system state vector x, as follows:

[0139]

[0140] In formula (8), T represents the transposition of the matrix, which is a matrix containing (w, h, a, v w ,v h ,v a ) is transposed to obtain the system state vector x.

[0141] The observed quantities are w, h, a respectively, and the observation vector M is calculated as follows:

[0142] M=(x,h,a) T (9)

[0143] In formula (9), T represents the transposition of the matrix. A matrix containing (w, h, a) is transposed to obtain the observation vector M.

[0144] The system state transfer matrix F is as follows (10):

[0145]

[0146] The observation matrix H is as follows (11):

[0147]

[0148] In formula (10) and formula (11), d t Indicates the time interval between two frames of images. 0 indicates a successful match, and 1 indicates a failed match.

[0149] S108: Determine whether the target to be detected is a newly added target. If the target to be detected is a newly added target, execute S109; if the detected target is not a newly added target, execute S110.

[0150] S109: Determine the target type of the newly added target, and obtain the preset size of the newly added target based on the target type, and return to execute S101.

[0151] In the specific process of executing S109, if it is determined that the target to be detected is a newly added target, its target type is determined, and according to its target type, a corresponding priori size (preset size) is assigned, and the priori size is obtained through statistics.

[0152] S110: Execute the exit or maintenance mechanism.

[0153] In the specific process of executing S110 , if the association fails and the target to be detected is not a newly added target but a target that has been tracked, it is determined whether to continue tracking the target to be detected through the exit / maintain mechanism.

[0154] In some embodiments of the present invention, executing S110 mainly includes the following steps:

[0155] First, the number of association failures corresponding to the target to be detected is determined.

[0156] Secondly, the number of association failures is compared with the exit threshold; if the number of association failures is less than or equal to the exit threshold, the process returns to S101. If the number of association failures is greater than the exit threshold, the target tracking is exited.

[0157] Based on the target tracking method provided by the embodiment of the present invention, the detection frame information corresponding to the target to be detected in the current frame image data and the corresponding vehicle motion information are obtained; the coordinate information of the historical target and the target to be detected in the adjacent frame image data in the vehicle coordinate system are respectively determined; and the predicted position of the historical target in the vehicle coordinate system at the current moment is predicted; the pixel coordinates and prediction frame information of the historical target are obtained according to the target size of the historical target; based on the preset association algorithm and target category, the target to be detected and the historical target of the same target type are associated; and after the association is successful, the size of the detection frame corresponding to the target to be detected is updated. The method disclosed by the present invention is not limited to various scenarios, and its tracking performance is safe and reliable, and has better robustness. Based on this, the purpose of supporting various scenarios, not relying on appearance features, and providing stable tracking results, especially for small targets, is achieved.

[0158] In order to better illustrate the method provided by the embodiment of the present invention, the embodiment of the present invention provides another method flow chart of a target tracking method, such as Figure 2 shown.

[0159] S201: Obtain current frame image information.

[0160] S202: Acquire vehicle IMU signal.

[0161] S203: Obtaining internal and external parameters of the camera.

[0162] Based on the above content, steps S201 to S203 can all be obtained through the hardware platform. Specifically, the camera sampled image information (current frame image data) calibrated in the platform and the internal and external parameters of the camera are recorded, and the vehicle IMU signal (vehicle motion data) is obtained by the IMU sensor in the platform.

[0163] S204: Signal alignment.

[0164] It can be understood that the current frame image data and the vehicle motion data are time compensated to extract the IMU information corresponding to the image data.

[0165] S205: Motion compensation of the tracked target.

[0166] Specifically, the motion compensation of the tracked target is completed through the temporal size information, calibration parameters (internal and external parameters) and extracted IMU information of the tracked target object. The tracked target here refers to the target object in the adjacent frame image data, and its adjacent frame image is determined by the sampling timestamp corresponding to the current frame image data, that is, the adjacent frame image closest to the sampling timestamp is determined.

[0167] After determining the temporal size information and calibration parameters of each tracked target object and the extracted IMU information, motion compensation is performed on each tracked target object based on this information. Motion compensation can be understood as moving the corresponding position of the tracked target object to a certain position in the current frame.

[0168] S206: Determine the target to be detected and its corresponding 2D detection box (2D-box).

[0169] Based on the preset detection network, each target to be detected contained in the image information is detected, and the detection frame information corresponding to each target to be detected can also be detected.

[0170] According to the detection frame information, the depth information corresponding to each target to be detected can be determined. It should be noted that if the calibrated camera in the hardware platform is a depth camera, the depth information corresponding to each target object to be detected can be directly obtained by the depth camera.

[0171] After obtaining various depth information, the predicted position of the tracked target object in the vehicle coordinate system at the current moment can be predicted based on the IMU information and various depth information.

[0172] S207: Target association.

[0173] If the position of the tracked target object after motion compensation is consistent with the target position corresponding to the target to be detected, it means that the tracked target object and the target to be detected are the same target object. Based on the preset association algorithm, the tracked target object can be associated with the target to be detected.

[0174] If the association is successful, the process proceeds to step S208 ; if the association fails, the process proceeds to step S212 or step S214 .

[0175] S208: Successfully associated with the target.

[0176] S209: Approximately estimate the depth.

[0177] S210: Approximately estimate the size.

[0178] For successfully associated targets, after approximately estimating the depth, obtain the detection frame size information contained in the corresponding detection frame information of the target to be detected in the adjacent frame image data, and obtain the detection frame size information contained in the corresponding detection frame information of the target to be detected in the current frame image data; based on the detection frame size information in the adjacent frame image data and the detection frame size information in the current frame image data, determine the size change coefficient corresponding to the target size change; based on the size change coefficient, further estimate the actual size of the target to be detected in the current frame image data.

[0179] S211: Filter update.

[0180] After estimating the change in the actual size of the target to be detected, the detection frame size information of the target to be detected is updated through the filter.

[0181] For example, if the target to be detected is a traffic light, Figure 3 The figure shows the position of the traffic light perceived in the previous frame. Figure 3 A rectangular detection frame is shown in FIG. Figure 4 The position of the traffic light after motion compensation is shown in FIG. Figure 5 The final result of tracking the traffic light is shown in the figure as a detection box + coordinates.

[0182] S212: New target.

[0183] If the association fails and the target to be detected is a new target, proceed to step S213.

[0184] S213: A priori size assumptions.

[0185] If the target to be detected is a newly appeared target object, the corresponding prior size is assigned according to the target object category, and the target tracking method is subsequently re-performed, that is, the prior size is used as the target attribute in tracking, combined with the internal and external parameters of the camera, and the target object is continued to be tracked.

[0186] S214: Other targets.

[0187] If the target to be detected is a target object that has been tracked but failed to be associated, the process proceeds to step S215.

[0188] S215: Maintain / Exit.

[0189] Maintain the tracker, that is, perform the target tracking method again. If the tracker does not match the target for the nth consecutive frame, delete the tracker.

[0190] Because the target to be detected is a tracked target object, the size corresponding to the tracked target object can be used as the tracked target attribute corresponding to the target to be detected.

[0191] S216: Attributes of the target being tracked.

[0192] The target attributes in tracking can refer to various size information, such as prior size, actual size, etc.

[0193] The target tracking method provided by the embodiment of the present invention is not limited to various scenarios, and its tracking performance is safe and reliable, and has better robustness. Based on this, it supports various scenarios, does not rely on appearance features, and provides stable tracking results for small targets in particular.

[0194] In order to better illustrate the method provided by the embodiment of the present invention, based on the above content, the present invention is illustrated by way of example, and the specific implementation method is as follows:

[0195] Step 1: Use the camera sensor deployed on the autonomous driving vehicle to continuously collect image data IMG at a relatively high fixed frequency, and record the camera's internal parameters I, external parameters E, and sampling timestamp TC. i (unit: ms), recorded in the set TC.

[0196] Step 2: The IMU sensor deployed on the autonomous vehicle continuously acquires vehicle motion information at a fixed sampling frequency higher than that of the camera, including the vehicle's motion speed (v x , v y , v z ), angular velocity of the vehicle (v yaw , v pitch , v roll ), and the sampling timestamp TO k (unit: ms), recorded in the set to.

[0197] Step 3: Use the current frame camera sampling timestamp TC i As a benchmark, the index timestamp collection TO is the same as TC i The timestamp with the smallest difference TO p , get the corresponding vehicle motion information.

[0198] Step 4: Use a deep learning-based detection method (such as YOLO) to detect the target information in the image IMG, including the 2D-box (detection box) and the corresponding category C.

[0199] Step 5: Set the prior size information for the specified category of targets. For example, the prior length and width of the traffic light are set to (1.3m, 0.7m). The relevant prior data is obtained through statistical results.

[0200] Step 6: Track the target in the continuously acquired images. The tracking module is mainly divided into two steps. The first part is target prediction and association, and the second part is attribute update.

[0201] The four corner points of the target’s 2D-box are approximately considered to be coplanar points. The corner points are transformed into the ego-vehicle coordinate system through the internal and external parameter projection relationship of the above formula (1) and formula (2). Combined with the target size information, the depth z of the target in the ego-vehicle coordinate system and the depth z in the camera coordinate system are estimated. c .

[0202] In the ego-vehicle coordinate system, the ego-vehicle motion information of adjacent frames is combined, and the above formula (3) is used to predict the position of the historical target in the ego-vehicle coordinate system at the current moment, and then convert it to the position in the pixel coordinate system.

[0203] Step 7: Search for the current frame target around the predicted position in the target image domain, and complete the matching of the current frame target with the target in the historical frame through the association algorithm (optimal matching algorithm such as Hungarian algorithm and Kuh Munkras algorithm).

[0204] For the successfully associated targets, the size change coefficient δ is calculated through the size change of the 2D-box of the same target, δ is approximated as the size change coefficient, and the change of the actual size of the target is estimated, and the size is updated through Kalman filtering.

[0205] For newly appeared targets, the corresponding prior size is assigned according to the category, and the value is obtained through statistics. For the remaining targets, the exit / maintain mechanism is used to determine whether to continue tracking the target.

[0206] The specific implementation processes and derivative methods of the above-mentioned embodiments are all within the protection scope of the present invention.

[0207] and Figure 1 Corresponding to the target tracking method, the embodiment of the present invention also provides a target tracking device for Figure 1 The specific implementation of the target tracking method in is shown in the following figure. Figure 6 As shown, specifically including:

[0208] The first acquisition unit 601 is used to acquire current frame image data and vehicle motion information corresponding to the current frame image data.

[0209] The second acquisition unit 602 is used to detect at least one target to be detected in the current frame image data based on a preset detection network, and the detection frame information corresponding to the target to be detected, where the detection frame information includes the size of the detection frame and the target category.

[0210] The prediction unit 603 is used to respectively determine the coordinate information of the historical target and the target to be detected in the adjacent frame image data in the vehicle coordinate system; in the vehicle coordinate system, based on the coordinate information of the historical target and the target to be detected, and the vehicle motion information corresponding to the adjacent frame image data, predict the predicted position of the historical target in the vehicle coordinate system at the current moment.

[0211] The conversion unit 604 is used to convert the predicted position into a pixel coordinate system according to the target size of the historical target, so as to obtain the pixel coordinates and prediction frame information of the historical target.

[0212] The association unit 605 is used to search for the target to be detected in the current frame image with the pixel coordinates as the fixed point, and associate the target to be detected and the historical targets of the same target type based on the preset association algorithm and target category; if the association is successful, the size of the detection frame corresponding to the target to be detected is updated based on the predicted frame information.

[0213] On the basis of the device provided in the above embodiment, in the device provided in the embodiment of the present application, the first acquisition unit 601 includes:

[0214] The first acquisition module is used to continuously acquire image data at a first preset frequency and continuously obtain vehicle motion data at a second preset frequency; determine the sampling image timestamp of the current frame image data, and the vehicle motion data corresponding to the sampling motion timestamp with the smallest difference with the sampling image timestamp.

[0215] The motion compensation module is used to perform time compensation between the current frame image data and the vehicle motion data, and extract the vehicle motion information contained in the vehicle motion data.

[0216] On the basis of the device provided in the above embodiment, in the device provided in the embodiment of the present application, the prediction unit 603 for respectively determining the coordinate information of the historical target and the target to be detected in the adjacent frame image data in the vehicle coordinate system includes:

[0217] The first parsing module is used to parse the historical targets and the targets to be detected in the adjacent frame image data respectively, and determine the depth information corresponding to the historical targets and the targets to be detected in the vehicle coordinate system. The current frame image data and the adjacent frame image data are collected by the depth camera.

[0218] and / or,

[0219] The target processing module to be detected is used to parse the detection frame information to obtain the target type and detection frame size of the target to be detected; obtain the preset size of the target to be detected according to the target type, and each target type has a corresponding preset size; convert the size of the detection frame and the preset size from the pixel coordinate system to the camera coordinate system to obtain the depth information of the target to be detected in the camera coordinate system; convert the depth information of the target to be detected in the camera coordinate system to the vehicle coordinate system to obtain the depth information of the target to be detected in the vehicle coordinate system and the coordinate information in the vehicle coordinate system;

[0220] The historical target processing module is used to obtain and convert the coordinates of the historical target in the pixel coordinate system to the camera coordinate system to obtain the depth information of the historical target in the camera coordinate system; convert the depth information of the historical target in the camera coordinate system to the vehicle coordinate system to obtain the depth information of the historical target in the vehicle coordinate system and the coordinate information in the vehicle coordinate system.

[0221] On the basis of the device provided in the above embodiment, in the device provided in the embodiment of the present application, in the vehicle coordinate system, according to the coordinate information of the historical target and the target to be detected, and the vehicle motion information corresponding to the adjacent frame image data, the prediction unit 603 predicts the predicted position of the historical target in the vehicle coordinate system at the current moment, including:

[0222] In the ego vehicle coordinate system, according to the coordinate information of the historical target and the target to be detected, as well as the vehicle motion information corresponding to the adjacent frame image data, the predicted position (x t ,y t , z t ).

[0223] On the basis of the device provided in the above embodiment, the device provided in the embodiment of the present application further includes: a newly added target processing unit.

[0224] The newly added target processing unit is used to determine the target type of the newly added target if the association fails and the target to be detected is a newly added target, and obtain the preset size of the newly added target based on the target type, and return to execute the first acquisition unit 601.

[0225] On the basis of the device provided in the above embodiment, the device provided in the embodiment of the present application further includes: an exit or maintenance mechanism unit.

[0226] The exit or maintenance mechanism unit is used to determine the number of association failures corresponding to the target to be detected if the association fails and the target to be detected is not a newly added target; compare the number of association failures with the exit threshold; if the number of association failures is less than or equal to the exit threshold, return to execute the first acquisition unit 601. If the number of association failures is greater than the exit threshold, exit target tracking.

[0227] On the basis of the device provided in the above embodiment, in the device provided in the embodiment of the present application, the associating unit 605 for updating the size of the detection frame corresponding to the target to be detected based on the prediction frame information specifically includes:

[0228] The acquisition module is used to obtain the size of the prediction box in the prediction box information, and calculate the change value between the size of the detection box corresponding to the target to be detected and the size of the prediction box.

[0229] The calculation module is used to calculate based on the size change coefficient formula (4) and formula (5) and the change value to obtain the size change coefficient.

[0230] The updating module is used to obtain the updated size of the detection frame corresponding to the target to be detected according to the size change coefficient and the updating formula (6) and formula (7).

[0231] The target tracking device provided by the embodiment of the present invention is not limited to various scenes, and its tracking performance is safe and reliable, and has better robustness. Based on this, it supports various scenes, does not rely on appearance features, and provides stable tracking results for small targets in particular.

[0232] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0233] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, computer software, or a combination of both.

[0234] In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0235] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target tracking method, characterized in that: The method comprises: Acquire current frame image data and vehicle motion information corresponding to the current frame image data; Detecting at least one target to be detected in the current frame image data and detection frame information corresponding to the target to be detected based on a preset detection network, wherein the detection frame information includes a size of the detection frame and a target category; Respectively determining the coordinate information of the historical target and the target to be detected in the adjacent frame image data in the vehicle coordinate system; In the ego-vehicle coordinate system, based on the coordinate information of the historical target and the target to be detected, and the vehicle motion information corresponding to the adjacent frame image data, predict the predicted position of the historical target in the ego-vehicle coordinate system at the current moment; According to the target size of the historical target, the predicted position is converted into a pixel coordinate system to obtain the pixel coordinates and prediction frame information of the historical target; Searching for the target to be detected in the current frame image with the pixel coordinates as fixed points, and associating the target to be detected and the historical targets of the same target type based on a preset association algorithm and target category; If the association is successful, the size of the detection frame corresponding to the target to be detected is updated based on the predicted frame information.

2. The method according to claim 1, characterized in that Acquiring current frame image data and vehicle motion information corresponding to the current frame image data, including: continuously acquiring image data at a first preset frequency, and continuously acquiring vehicle motion data at a second preset frequency; Determine a sampling image timestamp of the current frame image data, and vehicle motion data corresponding to a sampling motion timestamp having a minimum difference with the sampling image timestamp; The current frame image data and the vehicle motion data are time compensated to extract vehicle motion information contained in the vehicle motion data.

3. The method according to claim 1, characterized in that Determining coordinate information of the historical target and the target to be detected in the adjacent frame image data in the vehicle coordinate system respectively includes: Respectively analyzing the historical targets and the targets to be detected in the adjacent frame image data to determine the depth information corresponding to the historical targets and the targets to be detected in the vehicle coordinate system, wherein the current frame image data and the adjacent frame image data are collected by a depth camera; or, For the target to be detected, parsing the detection frame information to obtain the target type and detection frame size of the target to be detected; Obtaining a preset size of the target to be detected according to the target type, each target type having a corresponding preset size; Convert the size of the detection frame and the preset size from the pixel coordinate system to the camera coordinate system to obtain the depth information of the target to be detected in the camera coordinate system; Convert the depth information of the target to be detected in the camera coordinate system to the vehicle coordinate system to obtain the depth information of the target to be detected in the vehicle coordinate system and the coordinate information of the target in the vehicle coordinate system; For the historical target in the adjacent frame image data, the coordinates of the historical target in the pixel coordinate system are acquired and converted into the camera coordinate system to obtain the depth information of the historical target in the camera coordinate system; The depth information of the historical target in the camera coordinate system is converted into the vehicle coordinate system to obtain the depth information of the historical target in the vehicle coordinate system and the coordinate information of the historical target in the vehicle coordinate system.

4. The method according to any one of claims 1 to 3, characterized in that In the ego-vehicle coordinate system, according to the coordinate information of the historical target and the target to be detected, and the vehicle motion information corresponding to the adjacent frame image data, predicting the predicted position of the historical target in the ego-vehicle coordinate system at the current moment, including: In the vehicle coordinate system, according to the coordinate information of the historical target and the target to be detected, and the vehicle motion information corresponding to the adjacent frame image data, use Predict the predicted position (x) of the historical target in the vehicle coordinate system at the current moment t ,y t , z t ); Among them, t represents the time, Δt represents the time difference between adjacent frames, (x t-Δt ,y t-Δt , z t-Δt ) represents the coordinates of the historical target in the vehicle coordinate system at time t-Δt, R represents the rotation matrix of the vehicle in the Δt interval, R -1 represents the inverse matrix of the rotation matrix of the ego vehicle in the △t interval, and T represents the translation matrix generated by the vehicle movement in the △t time difference.

5. The method according to any one of claims 1 to 3, characterized in that The method further comprises: If the association fails and the target to be detected is a newly added target, the target type of the newly added target is determined, and the preset size of the newly added target is obtained based on the target type, and the process returns to the step of acquiring the current frame image data.

6. The method according to any one of claims 1 to 3, characterized in that The method further comprises: If the association fails, and the target to be detected is not a newly added target, determining the number of association failures corresponding to the target to be detected; Comparing the number of association failures with an exit threshold; If the number of association failures is less than or equal to the exit threshold, return to the step of obtaining the current frame image data; If the number of association failures is greater than the exit threshold, exit target tracking.

7. The method according to any one of claims 1 to 3, characterized in that Updating the size of the detection frame corresponding to the target to be detected based on the prediction frame information includes: Obtaining the size of the prediction box in the prediction box information, and calculating a change value between the size of the detection box corresponding to the target to be detected and the size of the prediction box; Calculate based on the dimensional variation coefficient formula and the variation value to obtain the dimensional variation coefficient; According to the size change coefficient and the update formula, an updated size of the detection frame corresponding to the target to be detected is obtained; Wherein, the dimensional variation coefficient formula is: Among them, δ w is the detection frame width size variation coefficient, δ h is the detection frame length size variation coefficient, w track-box2d is the tracking result of the detection box width, w measure-box2d is the observation result of the detection box width, h track-box2d is the tracking result of the detection box length, h measure-box2d is the observation result of the detection box length, α and β are weight coefficients; The update formula is: w=δ w *w track ; h=δ h *h track ; w is the actual width of the target to be detected after update, h is the actual length of the target to be detected after update, w track ,h track Indicates the size of the detection frame in the detection frame information.

8. A target tracking device, characterized in that: The device comprises: A first acquisition unit, used to acquire current frame image data and vehicle motion information corresponding to the current frame image data; A second acquisition unit, configured to detect at least one target to be detected in the current frame image data and detection frame information corresponding to the target to be detected based on a preset detection network, wherein the detection frame information includes a size of the detection frame and a target category; A prediction unit, used to respectively determine the coordinate information of the historical target and the target to be detected in the adjacent frame image data in the vehicle coordinate system; in the vehicle coordinate system, based on the coordinate information of the historical target and the target to be detected, and the vehicle motion information corresponding to the adjacent frame image data, predict the predicted position of the historical target in the vehicle coordinate system at the current moment; A conversion unit, configured to convert the predicted position into a pixel coordinate system according to a target size of the historical target, so as to obtain pixel coordinates and prediction frame information of the historical target; The association unit is used to search for the target to be detected in the current frame image with the pixel coordinates as a fixed point, and associate the target to be detected and the historical targets of the same target type based on a preset association algorithm and target category; if the association is successful, the size of the detection frame corresponding to the target to be detected is updated based on the predicted frame information.

9. The device according to claim 8, characterized in that The first acquisition unit includes: A first acquisition module is used to continuously acquire image data at a first preset frequency and continuously acquire vehicle motion data at a second preset frequency; determine a sampling image timestamp of the current frame image data and vehicle motion data corresponding to a sampling motion timestamp having a minimum difference with the sampling image timestamp; The motion compensation module is used to perform time compensation on the current frame image data and the vehicle motion data, and extract the vehicle motion information contained in the vehicle motion data.

10. The device according to claim 8 or 9, characterized in that The prediction unit for respectively determining the coordinate information of the historical target in the adjacent frame image data and the target to be detected in the vehicle coordinate system includes: A first parsing module, configured to parse the historical targets and the target to be detected in the adjacent frame image data respectively, and determine the depth information corresponding to the historical targets and the target to be detected in the vehicle coordinate system, wherein the current frame image data and the adjacent frame image data are collected by a depth camera; and / or, The target processing module to be detected is used to parse the detection frame information to obtain the target type and detection frame size of the target to be detected; obtain the preset size of the target to be detected according to the target type, and each target type has a corresponding preset size; convert the size and preset size of the detection frame from the pixel coordinate system to the camera coordinate system to obtain the depth information of the target to be detected in the camera coordinate system; convert the depth information of the target to be detected in the camera coordinate system to the vehicle coordinate system to obtain the depth information of the target to be detected in the vehicle coordinate system and the coordinate information in the vehicle coordinate system; The historical target processing module is used to obtain and convert the coordinates of the historical target in the pixel coordinate system to the camera coordinate system to obtain the depth information of the historical target in the camera coordinate system; convert the depth information of the historical target in the camera coordinate system to the vehicle coordinate system to obtain the depth information of the historical target in the vehicle coordinate system and the coordinate information in the vehicle coordinate system.

Images