A target tracking method and apparatus

Abstract

The application provides a target tracking method and device, which comprises the following steps: obtaining detection box information corresponding to a to-be-detected target in current frame image data and vehicle motion information corresponding to the to-be-detected target; determining coordinate information of a historical target and the to-be-detected target in a self-vehicle coordinate system in adjacent frame image data respectively; predicting a predicted position of the historical target in the self-vehicle coordinate system at a current time; obtaining pixel coordinates and prediction box information of the historical target according to a target size of the historical target; associating the to-be-detected target and the historical target of the same target type based on a preset association algorithm and a target category; and updating a size of a detection box corresponding to the to-be-detected target after successful association. Therefore, the application can support various scenes, is independent of appearance features, and can provide stable tracking results for small targets.

Description

Technical Field

[0001] This invention relates to the field of visual perception for autonomous driving, and in particular to a target tracking method and apparatus. Background Technology

[0002] In recent years, with the research and development of artificial intelligence and autonomous driving by major manufacturers and research institutions, technologies related to various levels of autonomous driving have developed rapidly. In the perception technology of autonomous vehicles, target detection and continuous frame tracking are the key to whether autonomous vehicles can continuously and accurately perceive targets. The correctness of the perception results directly affects the behavioral decisions of autonomous driving and is an important factor in ensuring safe driving.

[0003] Existing multi-target tracking technologies often rely on target appearance features or motion models in the image domain for tracking, or utilize edge computing devices for target tracking. In autonomous driving applications, situations arise where targets have similar appearances and are numerous and small. Tracking methods that depend on target appearance features or motion models in the image domain are difficult to implement, impacting tracking robustness and resulting in high noise in the motion prediction results. Furthermore, using edge computing devices for target tracking is computationally expensive and not widely applicable.

[0004] Therefore, designing a target tracking method to enhance tracking performance is an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the present invention provides a target tracking method and apparatus that have better robustness, so as to support various scenarios, not rely on appearance features, and provide stable tracking results, especially for small targets.

[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0007] The first aspect of this invention discloses a target tracking method, the method comprising:

[0008] Acquire the current frame image data and the vehicle motion information corresponding to the current frame image data;

[0009] Based on a preset detection network, at least one target to be detected in the current frame image data is detected, as well as the detection box information corresponding to the target to be detected, wherein the detection box information includes the size of the detection box and the target category;

[0010] The coordinate information of the historical targets and the target to be detected in the vehicle coordinate system in adjacent frame image data is determined respectively;

[0011] In the vehicle coordinate system, based on the coordinate information of the historical target and the target to be detected, as well as the vehicle motion information corresponding to the adjacent frame image data, the predicted position of the historical target in the vehicle coordinate system at the current moment is predicted.

[0012] Based on the target size of the historical target, the predicted position is transformed into a pixel coordinate system to obtain the pixel coordinates and prediction box information of the historical target;

[0013] Using the pixel coordinates as fixed points, the target to be detected is searched within the current frame image. Based on a preset association algorithm and target category, the target to be detected of the same type is associated with historical targets.

[0014] If the association is successful, the size of the detection box corresponding to the target to be detected is updated based on the predicted box information.

[0015] Preferably, acquiring the current frame image data and the vehicle motion information corresponding to the current frame image data includes:

[0016] Image data is continuously acquired at a first preset frequency, and vehicle motion data is continuously acquired at a second preset frequency.

[0017] Determine the sampling image timestamp of the current frame image data, and the vehicle motion data corresponding to the sampling motion timestamp with the smallest difference from the sampling image timestamp;

[0018] The current frame image data and the vehicle motion data are time-compensated to extract the vehicle motion information contained in the vehicle motion data.

[0019] Preferably, determining the coordinate information of historical targets and the target to be detected in the vehicle coordinate system in adjacent frame image data includes:

[0020] The historical targets and the target to be detected in the adjacent frame image data are analyzed respectively to determine the depth information of the historical targets and the target to be detected in the vehicle coordinate system. The current frame image data and the adjacent frame image data are acquired by a depth camera.

[0021] or,

[0022] For the target to be detected, the detection box information is parsed to obtain the target type and detection box size of the target to be detected;

[0023] The preset size of the target to be detected is obtained according to the target type, and each target type has a corresponding preset size;

[0024] The size and preset size of the detection box are transformed from the pixel coordinate system to the camera coordinate system to obtain the depth information of the target to be detected in the camera coordinate system.

[0025] The depth information of the target to be detected in the camera coordinate system is transformed into the vehicle coordinate system to obtain the depth information and coordinate information of the target to be detected in the vehicle coordinate system.

[0026] For historical targets in the adjacent frame image data, the coordinates of the historical targets in the pixel coordinate system are obtained and transformed to the camera coordinate system to obtain the depth information of the historical targets in the camera coordinate system.

[0027] The depth information of the historical target in the camera coordinate system is transformed into the vehicle coordinate system to obtain the depth information and coordinate information of the historical target in the vehicle coordinate system.

[0028] Preferably, in the vehicle coordinate system, based on the coordinate information of the historical target and the target to be detected, and the vehicle motion information corresponding to adjacent frame image data, predicting the predicted position of the historical target in the vehicle coordinate system at the current moment includes:

[0029] In the vehicle coordinate system, based on the coordinate information of the historical target and the target to be detected, and the vehicle motion information corresponding to adjacent frame image data, the following is utilized: Predict the predicted position (x, y) of the historical target in the vehicle coordinate system at the current time. t y t , z t );

[0030] Where t represents time, Δt represents the time difference between adjacent frames, (x t-Δt y t-Δt , z t-Δt ) represents the coordinates of the historical target in the vehicle coordinate system at time t-Δt, R represents the rotation matrix of the vehicle within the Δt interval, and T represents the translation matrix generated by the vehicle's motion within the Δt time difference.

[0031] Preferably, the method further includes:

[0032] If the association fails and the target to be detected is a newly added target, determine the target type of the newly added target, obtain the preset size of the newly added target based on the target type, and return to the step of obtaining the current frame image data.

[0033] Preferably, the method further includes:

[0034] If the association fails, and the target to be detected is not a newly added target, determine the number of association failures corresponding to the target to be detected;

[0035] Compare the number of association failures with the exit threshold.

[0036] If the number of association failures is less than or equal to the exit threshold, return to the step of obtaining the current frame image data;

[0037] If the number of association failures exceeds the exit threshold, exit target tracking.

[0038] Preferably, updating the size of the detection box corresponding to the target to be detected based on the predicted box information includes:

[0039] Obtain the size of the predicted box in the predicted box information, and calculate the change between the size of the detection box corresponding to the target to be detected and the size of the predicted box;

[0040] The size change coefficient is calculated based on the formula for the size change coefficient and the change value.

[0041] Based on the size change coefficient and update formula, the updated size of the detection box corresponding to the target to be detected is obtained;

[0042] The formula for the size variation coefficient is as follows:

[0043]

[0044]

[0045] Where, δ w δ is the coefficient for detecting the change in the width of the detection frame. h w is the coefficient for the change in the length of the detection frame. track-box2d To detect the tracking results of the bounding box width, w measure-bpx2d For the observation results of the detection frame width, h track-box2d For the tracking results of the detection box length, h measure-box2d The observed results are the length of the detection box, where α and β are weighting coefficients;

[0046] The update formula is:

[0047] w = δ w *w track ;

[0048] h = δ h *h track ;

[0049] w represents the updated actual width of the target object, and h represents the updated actual length of the target object. trackh track This indicates the size of the detection box in the detection box information.

[0050] A second aspect of this invention discloses a target tracking device, the device comprising:

[0051] The first acquisition unit is used to acquire the current frame image data and the vehicle motion information corresponding to the current frame image data;

[0052] The second acquisition unit is used to detect at least one target to be detected in the current frame image data based on a preset detection network, and to obtain detection box information corresponding to the target to be detected, wherein the detection box information includes the size of the detection box and the target category;

[0053] The prediction unit is used to determine the coordinate information of the historical target and the target to be detected in the vehicle coordinate system in adjacent frame image data respectively; in the vehicle coordinate system, based on the coordinate information of the historical target and the target to be detected, and the vehicle motion information corresponding to the adjacent frame image data, the predicted position of the historical target in the vehicle coordinate system at the current time is predicted.

[0054] The transformation unit is used to transform the predicted position to a pixel coordinate system based on the target size of the historical target, so as to obtain the pixel coordinates and prediction box information of the historical target.

[0055] The association unit is used to locate the target to be detected in the current frame image using the pixel coordinates as fixed points, and to associate the target to be detected and historical targets of the same target type based on a preset association algorithm and target category; if the association is successful, the size of the detection box corresponding to the target to be detected is updated based on the prediction box information.

[0056] Optionally, the first acquisition unit includes:

[0057] The first acquisition module is used to continuously acquire image data at a first preset frequency and continuously acquire vehicle motion data at a second preset frequency; determine the sampling image timestamp of the current frame image data, and the vehicle motion data corresponding to the sampling motion timestamp with the smallest difference from the sampling image timestamp;

[0058] The motion compensation module is used to perform time compensation between the current frame image data and the vehicle motion data, and extract the vehicle motion information contained in the vehicle motion data.

[0059] Optionally, the prediction unit that determines the coordinate information of historical targets in adjacent frame image data and the target to be detected in the vehicle coordinate system includes:

[0060] The first parsing unit is used to parse the historical target and the target to be detected in the adjacent frame image data respectively, and determine the depth information of the historical target and the target to be detected in the vehicle coordinate system. The current frame image data and the adjacent frame image data are acquired by a depth camera.

[0061] And / or,

[0062] The target processing unit is used to parse the detection box information to obtain the target type and detection box size of the target to be detected; obtain the preset size of the target to be detected according to the target type, with each target type having a corresponding preset size; transform the size of the detection box and the preset size from the pixel coordinate system to the camera coordinate system to obtain the depth information of the target to be detected in the camera coordinate system; transform the depth information of the target to be detected in the camera coordinate system to the vehicle coordinate system to obtain the depth information and coordinate information of the target to be detected in the vehicle coordinate system.

[0063] The historical target processing unit is used to acquire and transform the coordinates of the historical target in the pixel coordinate system to the camera coordinate system to obtain the depth information of the historical target in the camera coordinate system; and to transform the depth information of the historical target in the camera coordinate system to the vehicle coordinate system to obtain the depth information and coordinate information of the historical target in the vehicle coordinate system.

[0064] Compared with existing technologies, this invention has the following advantages: It acquires the detection box information corresponding to the target to be detected in the current frame image data, as well as its corresponding vehicle motion information; it determines the coordinate information of historical targets and the target to be detected in the vehicle coordinate system in adjacent frame image data; it predicts the predicted position of the historical target in the vehicle coordinate system at the current moment; it obtains the pixel coordinates and predicted box information of the historical target based on the target size of the historical target; it associates the target to be detected and historical targets of the same target type based on a preset association algorithm and target category; and it updates the size of the detection box corresponding to the target to be detected after successful association. The method disclosed in this invention is not limited to various scenarios, and its tracking performance is safe and reliable with better robustness. Based on this, it achieves the goal of supporting various scenarios, not relying on appearance features, and providing stable tracking results, especially for small targets. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0066] Figure 1 A flowchart of a target tracking method provided in an embodiment of the present invention;

[0067] Figure 2 This is another flowchart of a target tracking method provided in an embodiment of the present invention;

[0068] Figure 3 This is an example provided in this embodiment of the invention showing the position of the traffic light perceived in the previous frame;

[0069] Figure 4 The position of the traffic light after motion compensation is provided in the example of the embodiment of the present invention;

[0070] Figure 5 This is the final result of tracking traffic lights in the example provided in this embodiment of the invention;

[0071] Figure 6 This is a structural diagram of a target tracking device provided in an embodiment of the present invention. Detailed Implementation

[0072] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0073] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0074] This invention can be used in a wide variety of general-purpose or special-purpose computing environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor devices, distributed computing environments including any of the above devices, etc.

[0075] This invention provides a target tracking method that can be applied to various hardware platforms, and its execution entity can be the processor of various mobile devices; preferably, the hardware platform may further include a calibrated monocular camera sensor and an IMU sensor capable of recording vehicle motion information. The method flowchart is shown below. Figure 1 As shown, it specifically includes:

[0076] S101: Obtain the current frame image data and the vehicle motion information corresponding to the current frame image data.

[0077] In S101, the current frame image data includes the timestamp of the sampled image.

[0078] Optionally, a hardware platform including a calibrated monocular camera sensor and an IMU sensor capable of recording vehicle motion information can be used to sample data from various sensors. The sensor data includes image information from the vision sensor and motion information from the inertial measurement unit (IMU). The image information from the vision sensor is the image data, and the motion information from the inertial measurement unit is the vehicle motion data.

[0079] The process of acquiring image information by the vision sensor is as follows: the calibrated camera (which contains a vision sensor) continuously samples the image information of the road conditions where the vehicle is located at a fixed frequency, and records the camera's intrinsic parameters, extrinsic parameters and sampling timestamps. Here, the sampling timestamp is the sampling image timestamp.

[0080] The process of acquiring motion information of the inertial measurement unit is as follows: the IMU sensor continuously acquires the vehicle's motion information at a fixed frequency that is not less than the camera's sampling frequency, including vehicle speed, yaw rate information and sampling timestamps. Here, the sampling timestamp is the sampling vehicle timestamp / IMU sensor timestamp.

[0081] Optionally, the sampled sensor data includes image data of the current frame and image data of each previous frame. In the image data of each previous frame, the image data of the adjacent frame that is adjacent to the encoding of the current frame is determined. Then, the vehicle motion data corresponding to the sampling motion timestamp that is closest to the timestamp of the sampled image is obtained. By performing time compensation on the image data and the vehicle motion data, i.e., preprocessing the image data and the vehicle motion data, the vehicle motion information contained in the vehicle motion data can be extracted quickly.

[0082] Optionally, to determine the closest sampled image timestamp and sampled motion timestamp, the time difference between each timestamp can be determined first, and the minimum time difference can be identified. The two timestamps with the minimum time difference are then identified as the closest two timestamps.

[0083] In some embodiments of the present invention, acquiring current frame image data and vehicle motion information corresponding to the current frame image data includes:

[0084] First, image data is continuously acquired at a first preset frequency, and vehicle motion data is continuously acquired at a second preset frequency.

[0085] Secondly, determine the sampling image timestamp of the current frame image data, and the vehicle motion data corresponding to the sampling motion timestamp with the smallest difference from the sampling image timestamp.

[0086] Finally, time compensation is performed on the current frame image data and vehicle motion data to extract the vehicle motion information contained in the vehicle motion data.

[0087] S102: Detect at least one target to be detected in the current frame image data based on a preset detection network, and obtain the detection box information corresponding to the target to be detected.

[0088] In S102, the detection box information includes the size of the detection box and the target category. The size of the detection box includes length and width information. The target category refers to the object category of the target to be detected, such as a traffic light.

[0089] In S102, the detection network can be a YOLO network or a CNN network, etc.

[0090] It should be noted that the current frame image data contains one or more targets to be detected.

[0091] In the specific execution of S102, a deep learning detection network is used to detect each target to be detected in the current frame image data, so as to obtain the detection box information corresponding to each target to be detected.

[0092] S103: Determine the coordinate information of the historical target and the target to be detected in the vehicle coordinate system in adjacent frame image data respectively.

[0093] In S103, adjacent frames refer to frames that are adjacent to the current frame in terms of coding relationship; specifically, they describe the frame preceding the current frame in terms of coding relationship. The adjacent frame image data contains historical targets from each adjacent frame.

[0094] Optionally, the adjacent frame image data corresponding to the current image data and the vehicle motion information record corresponding to the adjacent frame image data can be determined by the timestamp of the sampled image.

[0095] In S103, the vehicle coordinate system refers to a three-dimensional coordinate system centered on the vehicle, with the positive x direction being forward, the positive y direction being left, and the positive z direction being upward;

[0096] It should be noted that, depending on the device used to acquire the images, different methods may be used in the specific execution of S103.

[0097] In some embodiments of the present invention, if the current frame image data and the adjacent frame image data are acquired by a depth camera, the depth information of the target to be detected in the current frame image data and the historical target in the adjacent frame image data in the vehicle coordinate system can be obtained through the depth camera.

[0098] The specific execution of S103 is as follows: parse the historical target and the target to be detected in the adjacent frame image data respectively, and determine the depth information of the historical target and the target to be detected in the vehicle coordinate system.

[0099] In some embodiments of the present invention, if the current frame image data and the adjacent frame image data are acquired by a common vehicle-mounted camera, the specific execution of S103 is as follows:

[0100] First, for the target to be detected, the detection box information is parsed to obtain the target type and detection box size.

[0101] Secondly, the preset size of the target to be detected is obtained according to the target type.

[0102] Each target type has a corresponding preset size. The preset size is the prior size information of that target type.

[0103] It should be noted that the prior size information can be set according to the preset prior data corresponding to each target or target type. It can be understood that each prior data can be obtained by statistically analyzing the size information corresponding to each target or target type, or by statistically analyzing the size information corresponding to each target. For example, if the prior length and width of a traffic light are set to (1.3m, 0.7m), the prior length and width of the traffic light are the size information corresponding to a target type.

[0104] Secondly, the size of the detection box and the preset size are transformed from the pixel coordinate system to the camera coordinate system to obtain the depth information of the target to be detected in the camera coordinate system.

[0105] Secondly, the depth information of the target to be detected in the camera coordinate system is transformed into the vehicle coordinate system to obtain the depth information and coordinate information of the target to be detected in the vehicle coordinate system.

[0106] Secondly, for historical targets in adjacent frame image data, the coordinates of the historical targets in the pixel coordinate system are obtained and transformed to the camera coordinate system to obtain the depth information of the historical targets in the camera coordinate system.

[0107] Finally, the depth information of the historical target in the camera coordinate system is transformed into the vehicle coordinate system to obtain the depth information and coordinate information of the historical target in the vehicle coordinate system.

[0108] Based on the above, in the process of acquiring the depth information of the target to be detected and historical targets in the vehicle coordinate system, if the vision sensor uses a depth camera, the depth information of the target to be detected and historical targets can be acquired through the depth camera.

[0109] If a standard automotive camera is used as the vision sensor, the detection bounding box information needs to be analyzed to determine the target type. Based on the pre-defined prior size information of the target under that target type, the preset size of the target is determined. Then, using a preset transformation matrix between the vehicle coordinate system and the pixel coordinate system, the depth information of the target in the vehicle coordinate system is estimated. Similarly, for historical targets, the depth information of the historical targets in the vehicle coordinate system is estimated using the same preset transformation matrix.

[0110] It should be noted that the process of transforming the detection bounding box information to the vehicle coordinate system can be understood as mapping the detection bounding box information from a 2D plane to a 3D plane. First, it transforms from the image (pixel coordinate system) to the camera coordinate system (second depth z). c Then, the depth (first depth z) is transformed from the camera coordinate system to the vehicle coordinate system. The origin is the top left point of the image, with the positive u direction to the right and the positive v direction downwards. The pixel coordinate system refers to the coordinate system of the 2D detection box on the image.

[0111] Specifically, the detection box information is converted to the vehicle coordinate system using the following formulas (1) and (2):

[0112]

[0113]

[0114] In formulas (1) and (2), (u, v) represent the coordinates of the target to be detected in the pixel coordinate system, and (x... c y c , z c (x, y, z) represents the coordinate position of the target to be detected in the camera coordinate system, (x, y, z) represents the coordinate position of the target to be detected in the vehicle coordinate system, I is the camera intrinsic parameter, and E is the camera extrinsic parameter. The camera intrinsic and extrinsic parameters are obtained from the camera calibration.

[0115] S104: In the vehicle coordinate system, based on the coordinate information of the historical target and the target to be detected, as well as the vehicle motion information corresponding to the adjacent frame image data, predict the predicted position of the historical target in the vehicle coordinate system at the current moment.

[0116] In the specific execution of S104, after confirming the coordinate information of the historical target and the target to be detected in the vehicle coordinate system, the vehicle motion information of the adjacent frame image data can be combined with the formula (3) to predict the predicted position of the historical target in the vehicle coordinate system at the current moment.

[0117] Specifically, formula (3) is as follows:

[0118]

[0119] In formula (3), (x t y t , z t (x) represents the predicted position of the historical target at time t in the vehicle coordinate system. t-Δt y t-Δt , z t-Δt Let t represent the coordinates of the historical target in the vehicle coordinate system at time t-Δt, and R represent the rotation matrix of the vehicle (vehicle) within the interval Δt. -1 T represents the inverse of the rotation matrix of the vehicle within the interval Δt, and T represents the translation matrix generated by the movement of the vehicle within the time difference Δt.

[0120] S105: Based on the target size of the historical target, the predicted position is transformed into the pixel coordinate system to obtain the pixel coordinates and predicted bounding box information of the historical target.

[0121] In the specific execution of S105, based on formulas (3), (2) and (1), and combined with the target size of the historical target, the predicted position of the historical target in the vehicle coordinate system at the current moment is transformed into the pixel coordinate system, and the pixel coordinates (u′, v′) and prediction box information of the historical target are obtained.

[0122] S106: Using pixel coordinates as fixed points, search for the target to be detected in the current frame image. Based on the preset association algorithm and target category, associate the target to be detected with historical targets of the same target type. If the association is successful, proceed to S107; if the association fails, proceed to S108.

[0123] In S106, the preset association algorithm includes, but is not limited to, the Hungarian algorithm, the Kuh Munkras algorithm, and other optimal matching algorithms.

[0124] In the specific execution of S106, the pixel coordinates of the historical target are used as fixed points to search for the target to be detected in the current frame image. Based on the preset association algorithm and target category, the target to be detected and the historical target belonging to the same target type are associated.

[0125] If the association is successful, it means that the historical object in the adjacent frame image data and the target to be detected in the current frame image data are the same target object, and S107 is executed.

[0126] The association failed, indicating that the historical object in the adjacent frame image data and the target to be detected in the current frame image data are not the same target object, and S108 is executed.

[0127] S107: Update the size of the detection box corresponding to the target to be detected based on the prediction box information.

[0128] In the specific execution of S107, the size change coefficient is calculated by the size change of the 2D-box of the same target object. The actual size change of the target is estimated by the size change coefficient, and the detection box size information is updated by filtering algorithms such as Kalman filtering.

[0129] Specifically, the actual size of the detection box of the target object can be updated by smoothing the width and height variables through filters. The formulas for the size change coefficients are shown in formulas (4) and (5):

[0130]

[0131]

[0132] In formulas (4) and (5), δ w δ is the coefficient for detecting the change in the width of the detection frame. h w is the coefficient for the change in the length of the detection frame. track-box2d To detect the tracking results of the bounding box width, w measure-box2d For the observation results of the detection frame width, h track-box2d For the tracking results of the detection box length, h measure-box2d The observed results represent the length of the detection box, where α and β are weighting coefficients.

[0133] The updated formulas are shown in formulas (6) and (7):

[0134] w = δ w *w track (6)

[0135] h = δ h *h track (7)

[0136] In formulas (6) and (7), w is the updated actual width of the target to be detected, and h is the updated actual length of the target to be detected. track h track This indicates the size of the detection box in the detection box information.

[0137] The Kalman filter parameters are set as follows:

[0138] Using w, h, and a as inputs to the Kalman filter, where a represents the aspect ratio, the motion vector is (v w ,v h ,v a ), calculate the system state vector x, as shown in the following formula (8):

[0139]

[0140] In formula (8), T represents the transpose of a matrix, transforming a matrix containing (w,h,a,v) into a transpose. w ,v h ,v a Transpose the matrix to obtain the system state vector x.

[0141] Let the observations be w, h, and a, respectively. Calculate the observation vector M as shown in the following formula (9):

[0142] M = (x, h, a) T (9)

[0143] In formula (9), T represents the transpose of a matrix, which is a matrix containing (w,h,a) that is transposed to obtain the observation vector M.

[0144] The system state transition matrix F is given by the following formula (10):

[0145]

[0146] The observation matrix H is given by the following formula (11):

[0147]

[0148] In formulas (10) and (11), d t This represents the time interval between two frames of images. 0 indicates a successful match, and 1 indicates a failed match.

[0149] S108: Determine whether the target to be detected is a newly added target. If the target to be detected is a newly added target, then execute S109; if the target to be detected is not a newly added target, then execute S110.

[0150] S109: Determine the target type of the new target, obtain the preset size of the new target based on the target type, and return to execute S101.

[0151] In the specific execution of S109, if the target to be detected is determined to be a new target, its target type is determined, and a corresponding prior size (preset size) is assigned according to its target type. This prior size is obtained through statistics.

[0152] S110: Execute exit or maintain mechanism.

[0153] During the execution of S110, if the association fails and the target to be detected is not a newly added target but a target that has already been tracked, the exit / maintenance mechanism is used to determine whether to continue tracking the target to be detected.

[0154] In some embodiments of the present invention, performing S110 mainly includes the following steps:

[0155] First, determine the number of association failures corresponding to the target to be detected.

[0156] Next, compare the number of association failures with the exit threshold; if the number of association failures is less than or equal to the exit threshold, return to execute S101. If the number of association failures is greater than the exit threshold, exit target tracking.

[0157] Based on the target tracking method provided in this invention, the following steps are taken: 1) Obtain the detection box information corresponding to the target to be detected in the current frame image data, and its corresponding vehicle motion information; 2) Determine the coordinate information of historical targets and the target to be detected in the vehicle coordinate system in adjacent frame image data; 3) Predict the predicted position of the historical target in the vehicle coordinate system at the current moment; 4) Obtain the pixel coordinates and predicted box information of the historical target based on its target size; 5) Associate the target to be detected and historical targets of the same target type based on a preset association algorithm and target category; 6) Update the size of the detection box corresponding to the target to be detected after successful association. The method disclosed in this invention is not limited to various scenarios, and its tracking performance is safe and reliable with better robustness. Based on this, it achieves the goal of supporting various scenarios, not relying on appearance features, and providing stable tracking results, especially for small targets.

[0158] To better illustrate the method provided in the embodiments of the present invention, the present invention provides another flowchart of a target tracking method, as follows: Figure 2 As shown.

[0159] S201: Obtain the image information of the current frame.

[0160] S202: Acquire vehicle IMU signal.

[0161] S203: Obtain camera internal and external parameters.

[0162] Based on the above, steps S201 to S203 can all be obtained through a hardware platform. Specifically, the platform samples image information (current frame image data) from the calibrated camera and records the camera's internal and external parameters. The platform's IMU sensor then acquires the vehicle's IMU signal (vehicle motion data).

[0163] S204: Signal alignment.

[0164] This can be understood as performing time compensation between the current frame image data and the vehicle motion data in order to extract the IMU information corresponding to the image data.

[0165] S205: Target motion compensation has been tracked.

[0166] Specifically, motion compensation for the tracked target is achieved by using the temporal size information, calibration parameters (internal and external parameters), and extracted IMU information of the tracked target object. Here, the tracked target refers to the target object in the adjacent frame image data. Its adjacent frame image is determined by the sampling timestamp corresponding to the current frame image data, that is, the adjacent frame image closest to the sampling timestamp is determined.

[0167] After determining the temporal size information and calibration parameters of each tracked target object, and extracting the IMU information, motion compensation is performed on each tracked target object based on this information. Motion compensation can be understood as moving the position corresponding to the tracked target object to a certain position in the current frame.

[0168] S206: Determine the target to be detected and its corresponding 2D detection box.

[0169] Based on a pre-set detection network, it detects each target in the image information and can also detect the corresponding bounding box information of each target.

[0170] Based on the detection frame information, the depth information corresponding to each target to be detected can be determined. It should be noted that if the camera calibrated in the hardware platform is a depth camera, the depth information corresponding to each target object can be directly obtained by the depth camera.

[0171] After acquiring the depth information, the predicted position of the tracked target object in the vehicle coordinate system at the current moment can be predicted based on the IMU information and the depth information.

[0172] S207: Target association.

[0173] If the position of the tracked target object after motion compensation is consistent with the target position of the target to be detected, it means that the tracked target object and the target to be detected are the same target object. Based on the preset association algorithm, the tracked target object and the target to be detected can be associated.

[0174] If the association is successful, proceed to step S208; if the association fails, proceed to step S212 or step S214.

[0175] S208: Successfully associated with the target.

[0176] S209: Approximate depth estimation.

[0177] S210: Approximate size estimate.

[0178] For a successfully associated target, after approximating the depth, the size information of the detection box contained in the detection box information of the target in the adjacent frame image data is obtained, and the size information of the detection box contained in the detection box information of the target in the current frame image data is also obtained. Based on the size information of the detection box in the adjacent frame image data and the size information of the detection box in the current frame image data, the size change coefficient corresponding to the target size change is determined. Based on the size change coefficient, the actual size of the target in the current frame image data is estimated.

[0179] S211: Filter update.

[0180] After estimating the change in the actual size of the target to be detected, the detection frame size information of the target to be detected is updated through a filter.

[0181] For example, if the target to be detected is a traffic light, such as Figure 3 The image shows the location of the traffic light as perceived in the previous frame. Specifically... Figure 3 The rectangular detection box is shown in the image; for example... Figure 4 The image shows the position of the traffic lights after motion compensation; as shown. Figure 5 To ultimately track the traffic light results, the results are displayed in the image as a detection box plus coordinates.

[0182] S212: New Target.

[0183] If the association fails and the target to be detected is a new target, proceed to step S213.

[0184] S213: Prior size assumption.

[0185] If the target to be detected is a newly appearing target object, then a corresponding prior size is assigned according to the target object category, and the target tracking method is re-executed. That is, the prior size is used as the target attribute in the tracking process, combined with the camera's intrinsic and extrinsic parameters, to continue tracking the target object.

[0186] S214: Other objectives.

[0187] If the target to be detected is a tracked object that has failed to be associated, proceed to step S215.

[0188] S215: Maintain / Exit.

[0189] The tracker is maintained, i.e., the target tracking method is performed again. If the tracker fails to match the target for the nth consecutive frame, the tracker is deleted.

[0190] Since the target to be detected is a tracked object, the size of the tracked object can be used as the target attribute in the tracking process for the target to be detected.

[0191] S216: Target attributes during tracking.

[0192] During tracking, target attributes can refer to various dimensional information, such as prior dimensions and actual dimensions.

[0193] The target tracking method provided in this invention is not limited to various scenarios, and its tracking performance is safe, reliable, and robust. Therefore, it supports various scenarios, is independent of appearance features, and provides stable tracking results, especially for small targets.

[0194] To better illustrate the method provided in the embodiments of the present invention, based on the above content, the present invention provides an example, and the specific implementation method is as follows:

[0195] Step 1: Using camera sensors deployed on the autonomous vehicle, continuously acquire image data (IMG) at a high fixed frequency, and record the camera's intrinsic parameters (I), extrinsic parameters (E), and sampling timestamp (TC). i (Unit: ms) is recorded in set TC.

[0196] Step 2: Continuously acquire vehicle motion information, including vehicle speed (v), using IMU sensors deployed on the autonomous vehicle at a fixed sampling frequency higher than that of cameras. x v y v z ), the angular velocity of the vehicle (v) yaw v pitch v roll ), and sampling timestamp TO k (Unit: ms), recorded in set to.

[0197] Step 3: Use the current frame camera sampling timestamp TC i Based on the index timestamp set TO, and TC i The timestamp with the smallest difference TO p , retrieve the corresponding vehicle motion information.

[0198] Step 4: Use a deep learning-based detection method (such as YOLO) to detect information about the target in the image IMG, including the 2D-box and the corresponding category C.

[0199] Step 5: Set prior size information for targets of a specified category. For example, the prior length and width of a traffic light are set to (1.3m, 0.7m). The relevant prior data are obtained through statistical results.

[0200] Step 6: Track the target in the continuously acquired images. The tracking module is mainly divided into two steps: the first part is target prediction and association, and the second part is attribute update.

[0201] The four corner points of the target's 2D box are approximately considered to be on the same plane. Using the projection relationship between the intrinsic and extrinsic parameters in the aforementioned formulas (1) and (2), the corner points are transformed into the vehicle coordinate system. Combined with the target size information, the depth z of the target in the vehicle coordinate system and the depth z in the camera coordinate system are estimated. c .

[0202] In the vehicle coordinate system, combined with the vehicle motion information of adjacent frames, the position of the historical target in the vehicle coordinate system at the current moment is predicted by the aforementioned formula (3), and then transformed to the position in the pixel coordinate system.

[0203] Step 7: Search for the target in the current frame around the predicted location in the target image domain, and use an association algorithm (such as the Hungarian algorithm, the Kuh Munkras algorithm, or other optimal matching algorithms) to match the target in the current frame with the target in the historical frames.

[0204] For successfully associated targets, the size change coefficient δ is calculated by the 2D-box size change of the same target. δ is approximated as the size change coefficient, and the actual size change of the target is estimated. The size is then updated by Kalman filtering.

[0205] For newly emerging targets, a corresponding prior size is assigned based on their category; this value is derived statistically. For other targets, an exit / maintenance mechanism is used to determine whether to continue tracking them.

[0206] The specific implementation processes and derivative methods of the above embodiments are all within the protection scope of this invention.

[0207] and Figure 1 Corresponding to the target tracking method described above, this embodiment of the invention also provides a target tracking device for tracking targets. Figure 1 The specific implementation of the target tracking method in [the document] is shown in the following diagram. Figure 6 As shown, it specifically includes:

[0208] The first acquisition unit 601 is used to acquire the current frame image data and the vehicle motion information corresponding to the current frame image data.

[0209] The second acquisition unit 602 is used to detect at least one target to be detected in the current frame image data based on a preset detection network, and the detection box information corresponding to the target to be detected, the detection box information including the size of the detection box and the target category.

[0210] The prediction unit 603 is used to determine the coordinate information of the historical target and the target to be detected in the vehicle coordinate system in adjacent frame image data respectively; in the vehicle coordinate system, based on the coordinate information of the historical target and the target to be detected, as well as the vehicle motion information corresponding to the adjacent frame image data, the predicted position of the historical target in the vehicle coordinate system at the current time is predicted.

[0211] The transformation unit 604 is used to transform the predicted position to the pixel coordinate system based on the target size of the historical target, so as to obtain the pixel coordinates and prediction box information of the historical target.

[0212] The association unit 605 is used to find the target to be detected in the current frame image with pixel coordinates as fixed points. Based on the preset association algorithm and target category, it associates the target to be detected with historical targets of the same target type. If the association is successful, it updates the size of the detection box corresponding to the target to be detected based on the prediction box information.

[0213] Based on the apparatus provided in the above embodiments, the apparatus provided in this application embodiment includes a first acquisition unit 601, comprising:

[0214] The first acquisition module is used to continuously acquire image data at a first preset frequency and continuously acquire vehicle motion data at a second preset frequency; determine the sampling image timestamp of the current frame image data, and the vehicle motion data corresponding to the sampling motion timestamp with the smallest difference from the sampling image timestamp.

[0215] The motion compensation module is used to perform time compensation between the current frame image data and the vehicle motion data, and to extract the vehicle motion information contained in the vehicle motion data.

[0216] Based on the apparatus provided in the above embodiments, the apparatus provided in this application includes a prediction unit 603 for determining the coordinate information of historical targets and targets to be detected in the vehicle coordinate system in adjacent frame image data, comprising:

[0217] The first parsing module is used to parse the historical targets and the targets to be detected in adjacent frame image data respectively, and determine the depth information of the historical targets and the targets to be detected in the vehicle coordinate system. The current frame image data and the adjacent frame image data are acquired by the depth camera.

[0218] And / or,

[0219] The target processing module is used to parse the detection box information to obtain the target type and detection box size of the target to be detected; obtain the preset size of the target to be detected according to the target type, with each target type having a corresponding preset size; transform the size of the detection box and the preset size from the pixel coordinate system to the camera coordinate system to obtain the depth information of the target to be detected in the camera coordinate system; transform the depth information of the target to be detected in the camera coordinate system to the vehicle coordinate system to obtain the depth information and coordinate information of the target to be detected in the vehicle coordinate system.

[0220] The historical target processing module is used to acquire and transform the coordinates of the historical target in the pixel coordinate system to the camera coordinate system to obtain the depth information of the historical target in the camera coordinate system; and to transform the depth information of the historical target in the camera coordinate system to the vehicle coordinate system to obtain the depth information and coordinate information of the historical target in the vehicle coordinate system.

[0221] Based on the apparatus provided in the above embodiments, the apparatus provided in this application includes a prediction unit 603 that predicts the predicted position of a historical target in the vehicle coordinate system at the current moment, based on the coordinate information of the historical target and the target to be detected, and the vehicle motion information corresponding to adjacent frame image data, in the vehicle coordinate system. The prediction unit 603 includes:

[0222] In the vehicle coordinate system, based on the coordinate information of the historical target and the target to be detected, as well as the vehicle motion information corresponding to the adjacent frame image data, the predicted position (x, y, x) of the historical target in the vehicle coordinate system at the current moment is predicted using formula (3). t y t , z t ).

[0223] Based on the apparatus provided in the above embodiments, the apparatus provided in this application embodiment further includes: a newly added target processing unit.

[0224] The newly added target processing unit is used to determine the target type of the new target if the association fails and the target to be detected is a new target, and to obtain the preset size of the new target based on the target type, and then return to execute the first acquisition unit 601.

[0225] Based on the apparatus provided in the above embodiments, the apparatus provided in this application further includes: an exit or maintenance mechanism unit.

[0226] This exit or maintenance mechanism unit is used to determine the number of association failures for the target to be detected if the association fails and the target to be detected is not a newly added target; compare the number of association failures with the exit threshold; if the number of association failures is less than or equal to the exit threshold, return to execute the first acquisition unit 601. If the number of association failures is greater than the exit threshold, exit target tracking.

[0227] Based on the apparatus provided in the above embodiments, the apparatus provided in this application embodiment includes, specifically, the association unit 605 for updating the size of the detection box corresponding to the target to be detected based on the prediction box information, which specifically includes:

[0228] The acquisition module is used to acquire the size of the prediction box in the prediction box information and calculate the change value between the size of the detection box corresponding to the target to be detected and the size of the prediction box.

[0229] The calculation module is used to calculate the size change coefficient based on the size change coefficient formulas (4) and (5) and the change value.

[0230] The update module is used to obtain the updated size of the detection box corresponding to the target to be detected based on the size change coefficient and update formulas (6) and (7).

[0231] The target tracking device provided in this invention is not limited to various scenarios, and its tracking performance is safe, reliable, and robust. Therefore, it supports various scenarios, is independent of appearance features, and provides stable tracking results, especially for small targets.

[0232] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0233] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both.

[0234] To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality above. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0235] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target tracking method, characterized in that, The method includes: Acquire the current frame image data and the vehicle motion information corresponding to the current frame image data; Based on a preset detection network, at least one target to be detected in the current frame image data is detected, as well as the detection box information corresponding to the target to be detected, wherein the detection box information includes the size of the detection box and the target category; The coordinate information of the historical targets and the target to be detected in the vehicle coordinate system in adjacent frame image data is determined respectively; In the vehicle coordinate system, based on the coordinate information of the historical target and the target to be detected, as well as the vehicle motion information corresponding to the adjacent frame image data, the predicted position of the historical target in the vehicle coordinate system at the current moment is predicted. Based on the target size of the historical target, the predicted position is transformed into a pixel coordinate system to obtain the pixel coordinates and prediction box information of the historical target; Using the pixel coordinates as fixed points, the target to be detected is searched within the current frame image. Based on a preset association algorithm and target category, the target to be detected of the same type is associated with historical targets. If the association is successful, the size of the detection box corresponding to the target to be detected is updated based on the predicted box information; Updating the size of the detection box corresponding to the target to be detected based on the predicted box information includes: Obtain the size of the predicted box in the predicted box information, and calculate the change value between the size of the detection box corresponding to the target to be detected and the size of the predicted box; the change value includes the tracking result and observation result of the detection box width, and the tracking result and observation result of the detection box length; The size change coefficient is calculated based on the formula for the size change coefficient and the change value. Based on the size change coefficient and update formula, the updated size of the detection box corresponding to the target to be detected is obtained; The formula for the size variation coefficient is as follows: ； ； in, To detect the coefficient of change in the width of the detection frame. The coefficient for the change in the length of the detection frame. To detect the tracking results of the bounding box width, For the observation results of the detection frame width, For the tracking results of the detection box length, The observed results are the length of the detection box, where α and β are weighting coefficients; The update formula is: ； ； h represents the updated actual width of the target object, and h represents the updated actual length of the target object. This indicates the size of the detection box in the detection box information.

2. The method according to claim 1, characterized in that, Acquiring the current frame image data and the vehicle motion information corresponding to the current frame image data, including: Image data is continuously acquired at a first preset frequency, and vehicle motion data is continuously acquired at a second preset frequency. Determine the sampling image timestamp of the current frame image data, and the vehicle motion data corresponding to the sampling motion timestamp with the smallest difference from the sampling image timestamp; The current frame image data and the vehicle motion data are time-compensated to extract the vehicle motion information contained in the vehicle motion data.

3. The method according to claim 1, characterized in that, Determine the coordinate information of historical targets and the target to be detected in the vehicle coordinate system in adjacent frame image data, including: The historical targets and the target to be detected in the adjacent frame image data are analyzed respectively to determine the depth information of the historical targets and the target to be detected in the vehicle coordinate system. The current frame image data and the adjacent frame image data are acquired by a depth camera. or, For the target to be detected, the detection box information is parsed to obtain the target type and detection box size of the target to be detected; The preset size of the target to be detected is obtained according to the target type, and each target type has a corresponding preset size; The size and preset size of the detection box are transformed from the pixel coordinate system to the camera coordinate system to obtain the depth information of the target to be detected in the camera coordinate system. The depth information of the target to be detected in the camera coordinate system is transformed into the vehicle coordinate system to obtain the depth information and coordinate information of the target to be detected in the vehicle coordinate system. For historical targets in the adjacent frame image data, the coordinates of the historical targets in the pixel coordinate system are obtained and transformed to the camera coordinate system to obtain the depth information of the historical targets in the camera coordinate system. The depth information of the historical target in the camera coordinate system is transformed into the vehicle coordinate system to obtain the depth information and coordinate information of the historical target in the vehicle coordinate system.

4. The method according to any one of claims 1 to 3, characterized in that, In the vehicle coordinate system, based on the coordinate information of the historical target and the target to be detected, and the vehicle motion information corresponding to adjacent frame image data, the predicted position of the historical target in the vehicle coordinate system at the current moment is predicted, including: In the vehicle coordinate system, based on the coordinate information of the historical target and the target to be detected, and the vehicle motion information corresponding to adjacent frame image data, the following is utilized: Predict the predicted position of the historical target in the vehicle coordinate system at the current time. ); Where t represents time. Indicates the time difference between adjacent frames. express The coordinates of the historical target at a given moment in the vehicle's coordinate system, where R represents the vehicle's position. Rotation matrix within the interval, R -1 Indicates that the car is The inverse of the rotation matrix within the interval, T represents the... The translation matrix generated by the vehicle's movement within the time difference.

5. The method according to any one of claims 1 to 3, characterized in that, The method further includes: If the association fails and the target to be detected is a newly added target, determine the target type of the newly added target, obtain the preset size of the newly added target based on the target type, and return to the step of obtaining the current frame image data.

6. The method according to any one of claims 1 to 3, characterized in that, The method further includes: If the association fails, and the target to be detected is not a newly added target, determine the number of association failures corresponding to the target to be detected; Compare the number of association failures with the exit threshold. If the number of association failures is less than or equal to the exit threshold, return to the step of obtaining the current frame image data; If the number of association failures exceeds the exit threshold, exit target tracking.

7. A target tracking device, characterized in that, The device includes: The first acquisition unit is used to acquire the current frame image data and the vehicle motion information corresponding to the current frame image data; The second acquisition unit is used to detect at least one target to be detected in the current frame image data based on a preset detection network, and to obtain detection box information corresponding to the target to be detected, wherein the detection box information includes the size of the detection box and the target category; The prediction unit is used to determine the coordinate information of the historical target and the target to be detected in the vehicle coordinate system in adjacent frame image data respectively; in the vehicle coordinate system, based on the coordinate information of the historical target and the target to be detected, and the vehicle motion information corresponding to the adjacent frame image data, the predicted position of the historical target in the vehicle coordinate system at the current time is predicted. The transformation unit is used to transform the predicted position to a pixel coordinate system based on the target size of the historical target, so as to obtain the pixel coordinates and prediction box information of the historical target. The association unit is used to locate the target to be detected in the current frame image using the pixel coordinates as fixed points, and to associate the target to be detected and historical targets of the same target type based on a preset association algorithm and target category; if the association is successful, the size of the detection box corresponding to the target to be detected is updated based on the prediction box information. The association unit that updates the size of the detection box corresponding to the target to be detected based on the predicted box information includes: The acquisition module is used to acquire the size of the prediction box in the prediction box information and calculate the change value between the size of the detection box corresponding to the target to be detected and the size of the prediction box; the change value includes the tracking result and observation result of the detection box width, and the tracking result and observation result of the detection box length; The calculation module is used to calculate the size change coefficient based on the size change coefficient formula and the change value; The update module is used to obtain the updated size of the detection box corresponding to the target to be detected based on the size change coefficient and the update formula; The formula for the size variation coefficient is as follows: ； ； in, To detect the coefficient of change in the width of the detection frame. The coefficient for the change in the length of the detection frame. To detect the tracking results of the bounding box width, For the observation results of the detection frame width, For the tracking results of the detection box length, The observed results are the length of the detection box, where α and β are weighting coefficients; The update formula is: ； ； h represents the updated actual width of the target object, and h represents the updated actual length of the target object. This indicates the size of the detection box in the detection box information.

8. The apparatus according to claim 7, characterized in that, The first acquisition unit includes: The first acquisition module is used to continuously acquire image data at a first preset frequency and continuously acquire vehicle motion data at a second preset frequency; determine the sampling image timestamp of the current frame image data, and the vehicle motion data corresponding to the sampling motion timestamp with the smallest difference from the sampling image timestamp; The motion compensation module is used to perform time compensation between the current frame image data and the vehicle motion data, and extract the vehicle motion information contained in the vehicle motion data.

9. The apparatus according to claim 7 or 8, characterized in that, The prediction unit that determines the coordinate information of historical targets and the target to be detected in the vehicle coordinate system in adjacent frame image data includes: The first parsing module is used to parse the historical target and the target to be detected in the adjacent frame image data respectively, and determine the depth information of the historical target and the target to be detected in the vehicle coordinate system. The current frame image data and the adjacent frame image data are acquired by a depth camera. And / or, The target processing module is used to parse the detection box information to obtain the target type and detection box size of the target to be detected; obtain the preset size of the target to be detected according to the target type, with each target type having a corresponding preset size; transform the size of the detection box and the preset size from the pixel coordinate system to the camera coordinate system to obtain the depth information of the target to be detected in the camera coordinate system; transform the depth information of the target to be detected in the camera coordinate system to the vehicle coordinate system to obtain the depth information and coordinate information of the target to be detected in the vehicle coordinate system. The historical target processing module is used to acquire and transform the coordinates of the historical target in the pixel coordinate system to the camera coordinate system to obtain the depth information of the historical target in the camera coordinate system; and to transform the depth information of the historical target in the camera coordinate system to the vehicle coordinate system to obtain the depth information and coordinate information of the historical target in the vehicle coordinate system.

Images