Dynamic traffic object marking method and device, automobile controller and storage medium

Abstract

The invention relates to a dynamic traffic object marking method and device, an automobile controller and a storage medium. The method comprises the following steps: acquiring vehicle-mounted camera image data, laser point cloud data and IMU data of each time frame in a preset time interval; obtaining vehicle relative pose information corresponding to each time frame according to the IMU data and each static point cloud data; obtaining fusion point cloud data corresponding to each time frame by using the relative pose information of each vehicle and each laser point cloud data, and obtaining first object detection information of a dynamic traffic object corresponding to each time frame according to each fusion point cloud data and each vehicle-mounted camera image data; and obtaining an object movement track of the dynamic traffic object in a preset time interval by using the first object detection information, and obtaining first object labeling data of the dynamic traffic object at different time points in the preset time interval based on the object movement track. By adopting the method, the marking accuracy of the dynamic traffic object can be improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a dynamic traffic object labeling method, device, automobile controller, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of autonomous driving technology, a technology has emerged to perceive the driving environment around the vehicle during driving. By carrying a car camera in the vehicle, 360° field of view perception is achieved. However, the images taken by the car camera lack depth information, and it is impossible to directly complete the annotation of traffic objects in the driving scene through the car camera. At the same time, lidar is widely used for traffic object annotation because it can accurately measure the depth information of the surrounding environment.

[0003] At present, the labeling of traffic objects using lidar usually requires first obtaining autonomous driving data containing camera data and point cloud data, then reconstructing the three-dimensional scene by preprocessing the driving scene, and finally obtaining the labeling results of traffic objects in the reconstructed scene.

[0004] The above-mentioned traffic object labeling method realized by scene reconstruction has a relatively good labeling effect for static traffic objects such as lane lines, cones, etc. However, for traffic objects in motion, such as vehicles and pedestrians, motion ghosting will appear in the process of scene reconstruction. Therefore, the existing traffic object labeling method has a low degree of accuracy in labeling dynamic traffic objects. Summary of the invention

[0005] Based on this, it is necessary to provide a dynamic traffic object labeling method, device, automobile controller, computer-readable storage medium and computer program product that can improve the labeling accuracy of dynamic traffic objects in response to the above technical problems.

[0006] In a first aspect, the present application provides a dynamic traffic object labeling method, comprising:

[0007] Acquire vehicle-mounted camera image data, laser point cloud data and IMU data of each time frame in a preset time interval; each of the laser point cloud data includes static point cloud data and dynamic point cloud data;

[0008] According to the IMU data and the static point cloud data, the relative posture information of the vehicle corresponding to each time frame is obtained;

[0009] Using the relative posture information of each vehicle and each laser point cloud data, fused point cloud data corresponding to each time frame is obtained, and according to each fused point cloud data and each vehicle-mounted camera image data, first object detection information of a dynamic traffic object corresponding to each time frame is obtained; the dynamic traffic object is a traffic object represented by the dynamic point cloud data;

[0010] The object motion trajectory of the dynamic traffic object in the preset time interval is obtained by using each of the first object detection information, and the first object labeling data of the dynamic traffic object at different time points in the preset time interval is obtained based on the object motion trajectory.

[0011] In one embodiment, each first object detection information and each first object annotation data respectively correspond to a vehicle coordinate system at different time points; using each first object detection information to obtain the object motion trajectory of the dynamic traffic object in the preset time interval, and based on the object motion trajectory, obtaining the first object annotation data of the dynamic traffic object at different time points in the preset time interval, including: using the vehicle relative posture information corresponding to each of the time frames to perform coordinate transformation on each of the first object detection information, to obtain the second object detection information corresponding to each of the time frames; the second object detection information corresponds to the global coordinate system; each of the vehicle relative posture information represents the relative position of each vehicle coordinate system to the The transformation relationship between the global coordinate systems; according to each of the second object detection information, the dynamic traffic object is tracked to obtain the object motion trajectory, and the object motion trajectory is used to obtain the second object annotation data of the dynamic traffic object at different time points in the preset time interval; the second object annotation data corresponds to the global coordinate system; according to the vehicle relative posture information corresponding to each of the time frames, the vehicle relative posture information at different time points in the preset time interval is obtained, and the vehicle relative posture information at different time points in the preset time interval is used to perform coordinate inverse transformation on each of the second object annotation data to obtain the first object annotation data of the dynamic traffic object at different time points in the preset time interval.

[0012] In one embodiment, each second object detection information includes observed position data of the dynamic traffic object corresponding to each time frame; and according to each second object detection information, the dynamic traffic object is tracked to obtain the object motion trajectory, including: based on the observed position data of each time frame, the dynamic traffic object is forward tracked by a Kalman filter to obtain the first corrected position data of the last frame in each time frame; based on the first corrected position data of the last frame and the observed position data of each time frame, the dynamic traffic object is reversely tracked by a Kalman filter to obtain the second corrected position data of each time frame; and the object motion trajectory is obtained according to the second corrected position data of each time frame.

[0013] In one embodiment, the method of forward tracking the dynamic traffic object based on the observed position data of each time frame through a Kalman filter to obtain the first corrected position data of the tail frame in each time frame includes: taking the first frame of each time frame as the current time frame, and taking the observed position data of the first frame as the first corrected position data of the current time frame; inputting the first corrected position data of the current time frame into the Kalman filter to obtain the forward predicted position data of the next time frame corresponding to the current time frame; obtaining the first corrected position data of the next time frame according to the forward predicted position data of the next time frame and the observed position data of the next time frame, and updating the Kalman filter with the first corrected position data; when the current time frame is not the tail frame, taking the next time frame as the new current time frame, and returning to execute the step of inputting the first corrected position data of the current time frame into the Kalman filter until the current time frame is the tail frame; when the current time frame is the tail frame, taking the first corrected position data of the current time frame as the first corrected position data of the tail frame.

[0014] In one embodiment, the first corrected position data of the tail frame and the observed position data of each time frame are used to reversely track the dynamic traffic object through a Kalman filter to obtain the second corrected position data of each time frame, including: taking the tail frame as the current time frame and taking the first corrected position data of the tail frame as the second corrected position data of the current time frame; inputting the second corrected position data of the current time frame into the Kalman filter to obtain the reverse predicted position data of the previous time frame corresponding to the current time frame; obtaining the second corrected position data of the previous time frame according to the reverse predicted position data of the previous time frame and the observed position data of the previous time frame, and updating the Kalman filter using the second corrected position data; when the current time frame is not the first frame of each time frame, taking the previous time frame as the new current time frame, and returning to execute the step of inputting the second corrected position data of the current time frame into the Kalman filter until the current time frame is the first frame; when the current time frame is the first frame, taking the second corrected position data of each current time frame as the second corrected position data of each time frame.

[0015] In one embodiment, each second object annotation data includes second position data, second speed data and second orientation data of the dynamic traffic object at each time point in the preset time interval; each second object detection information includes orientation data of the dynamic traffic object corresponding to each time frame; using the object motion trajectory to obtain the second object annotation data of the dynamic traffic object at different time points in the preset time interval includes: obtaining the second position data of the dynamic traffic object at each time point according to the object motion trajectory; performing differential calculation on the object motion trajectory to obtain the second speed data of the dynamic traffic object at each time point; and performing interpolation processing on the orientation data of each time frame to obtain the second orientation data of the dynamic traffic object at each time point.

[0016] In one embodiment, each first object annotation data includes the first position data, first speed data and first orientation data of the dynamic traffic object corresponding to each time point; the relative posture information of the vehicle at different time points in the preset time interval is used to perform an inverse coordinate transformation on each second object annotation data to obtain the first object annotation data of the dynamic traffic object at different time points in the preset time interval, including: using the relative posture information of the vehicle at each time point to obtain the rotation matrix information corresponding to each time point in the preset time interval; each rotation matrix information represents the rotation relationship between each vehicle coordinate system and the global coordinate system; using the relative posture information of the vehicle at each time point, performing an inverse coordinate transformation on each second position data to obtain each first position data; using the rotation matrix information corresponding to each time point, performing an inverse coordinate transformation on each second speed data and second orientation data to obtain each first speed data and first orientation data.

[0017] In one embodiment, the use of the relative posture information of each vehicle and the laser point cloud data to obtain the fused point cloud data corresponding to each time frame includes: obtaining the current time frame and a preset number of historical time frames corresponding to the current time frame; the current time frame is any one of the time frames, and the historical time frames are a preset number of time frames adjacent to the current time frame and located before the current time frame; according to the laser point cloud data corresponding to the current time frame, the vehicle relative posture information corresponding to the current time frame, the laser point cloud data corresponding to each historical time frame, and the vehicle relative posture information corresponding to each historical time frame, the fused point cloud data corresponding to the current time frame is obtained.

[0018] In one of the embodiments, the first object detection information of the dynamic traffic objects corresponding to each of the time frames is obtained based on each of the fused point cloud data and each of the vehicle-mounted camera image data, including: inputting the fused point cloud data corresponding to the current time frame and the vehicle-mounted camera image data corresponding to the current time frame into a pre-constructed target detection model, and obtaining the fused point cloud features and vehicle-mounted camera image features corresponding to the current time frame through the target detection model; converting the fused point cloud features and vehicle-mounted camera image features into BEV vision, and fusing the fused point cloud features and vehicle-mounted camera image features after the BEV vision conversion to obtain a fused BEV feature; and obtaining the dynamic traffic objects contained in the current time frame and the first object detection information of the dynamic traffic objects based on the fused BEV features.

[0019] In one of the embodiments, obtaining the dynamic traffic objects included in the current time frame based on the fused BEV features includes: obtaining candidate dynamic traffic objects included in the current time frame and the confidence corresponding to each of the candidate dynamic traffic objects based on the fused BEV features; and taking the candidate dynamic traffic objects whose confidence is greater than a preset confidence threshold as the dynamic traffic objects included in the current time frame.

[0020] In a second aspect, the present application also provides a dynamic traffic object labeling device, comprising:

[0021] A detection data acquisition module, used to acquire vehicle-mounted camera image data, laser point cloud data and IMU data of each time frame in a preset time interval; each of the laser point cloud data includes static point cloud data and dynamic point cloud data;

[0022] A relative posture acquisition module, used for obtaining the relative posture information of the vehicle corresponding to each of the time frames according to the IMU data and each of the static point cloud data;

[0023] A detection information acquisition module, used to obtain fused point cloud data corresponding to each of the time frames using the relative posture information of each of the vehicles and each of the laser point cloud data, and to obtain first object detection information of a dynamic traffic object corresponding to each of the time frames according to each of the fused point cloud data and each of the vehicle-mounted camera image data; the dynamic traffic object is a traffic object represented by the dynamic point cloud data;

[0024] The annotation data acquisition module is used to use the first object detection information to obtain the object motion trajectory of the dynamic traffic object in the preset time interval, and based on the object motion trajectory, obtain the first object annotation data of the dynamic traffic object at different time points in the preset time interval.

[0025] In a third aspect, the present application further provides an automobile controller, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in any one of the embodiments of the first aspect when executing the computer program.

[0026] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of the embodiments of the first aspect.

[0027] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method described in any one of the embodiments of the first aspect.

[0028] The above-mentioned dynamic traffic object labeling method, device, automobile controller, computer-readable storage medium and computer program product obtain the vehicle-mounted camera image data, laser point cloud data and IMU data of each time frame in a preset time interval; each laser point cloud data includes static point cloud data and dynamic point cloud data; according to the IMU data and each static point cloud data, the relative posture information of the vehicle corresponding to each time frame is obtained; using the relative posture information of each vehicle and each laser point cloud data, the fused point cloud data corresponding to each time frame is obtained, and according to each fused point cloud data and each vehicle-mounted camera image data, the first object detection information of the dynamic traffic object corresponding to each time frame is obtained; the dynamic traffic object is a traffic object represented by the dynamic point cloud data; using each first object detection information, the object motion trajectory of the dynamic traffic object in the preset time interval is obtained, and based on the object motion trajectory, the first object labeling data of the dynamic traffic object at different time points in the preset time interval is obtained. The present application obtains multiple frames of vehicle-mounted camera image data within a preset time interval, including laser point cloud data and IMU data of static point cloud and dynamic point cloud, and then obtains the vehicle relative posture information corresponding to each time frame according to the IMU data and the static point cloud data, so as to fuse the laser point cloud data using the vehicle relative posture information, and then detects the target object using the fused point cloud data and the vehicle-mounted camera image data. After obtaining the first object detection information of the dynamic traffic object, the object motion trajectory of the dynamic traffic object in the preset time interval can be obtained, so as to obtain the object annotation data at different time points. In this way, the annotation of dynamic traffic objects can be achieved without scene reconstruction, so the annotation accuracy of dynamic traffic objects can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0030] FIG1 is a schematic diagram of a flow chart of a dynamic traffic object labeling method according to an embodiment;

[0031] FIG2 is a schematic diagram of a process of obtaining first object annotation data in one embodiment;

[0032] FIG3 is a schematic diagram of a process of obtaining a motion trajectory of an object in one embodiment;

[0033] FIG4 is a schematic diagram of a process of obtaining first corrected position data of a tail frame in one embodiment;

[0034] FIG5 is a schematic diagram of a process for obtaining second corrected position data for each time frame in one embodiment;

[0035] FIG6 is a schematic diagram of a process of obtaining first object detection information of a dynamic traffic object in one embodiment;

[0036] FIG7 is a schematic diagram of dynamic and static point cloud division in one embodiment;

[0037] FIG8 is a schematic diagram of the Bevfusion algorithm model structure in one embodiment;

[0038] FIG9 is a structural block diagram of a dynamic traffic object labeling device in one embodiment;

[0039] FIG. 10 is a diagram showing the internal structure of a vehicle controller in one embodiment. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0041] In one embodiment, as shown in FIG1 , a dynamic traffic object labeling method is provided. This embodiment uses the method applied to a car controller as an example for illustration. It is understandable that the method can also be applied to a server, and can also be applied to a system including a car controller and a server, and is implemented through the interaction between the car controller and the server. In this embodiment, the method includes the following steps:

[0042] Step S101, obtaining vehicle-mounted camera image data, laser point cloud data and IMU data of each time frame in a preset time interval; each laser point cloud data includes static point cloud data and dynamic point cloud data.

[0043] Among them, the preset time interval refers to a pre-set time interval for completing traffic object labeling, for example, it can be 1 second. The preset time interval can include multiple time frames, and each frame corresponds to corresponding vehicle-mounted camera image data, laser point cloud data and IMU data. For example, 1 second can be composed of 10 consecutive time frames, so the preset time interval can carry 10 frames of vehicle-mounted camera image data, laser point cloud data and IMU data.

[0044] Static point cloud data refers to the static points contained in each frame of laser point cloud data, which can be points corresponding to objects such as lane lines and cones. Dynamic point cloud data refers to the dynamic target points contained in each frame of laser point cloud data, such as points corresponding to objects such as pedestrians or vehicles.

[0045] Specifically, the car controller can obtain the camera images of continuous frames taken by the on-board camera within a preset time interval as the on-board camera image data of each time frame, and can also obtain the laser point cloud data of each time frame provided by the laser radar, as well as IMU data such as acceleration and angular velocity obtained by the inertial measurement unit. After that, the laser point cloud data can be divided into static point cloud data and dynamic point cloud data, where the division can be achieved through the high-precision point cloud segmentation model SPVNAS, which predicts the dynamic and static state of each point and outputs a predicted value. If the value is greater than the preset threshold, it is a dynamic point, otherwise it is a static point.

[0046] Step S102, obtaining the relative posture information of the vehicle corresponding to each time frame according to the IMU data and each static point cloud data.

[0047] The vehicle's relative posture information can be used to represent the relative posture of the vehicle in each time frame in a preset time interval. After obtaining the IMU data and each static point cloud data, the vehicle controller can combine the previous 10 consecutive frames of static point cloud and IMU data and use the lio-sam algorithm to output the vehicle's relative posture information.

[0048] Step S103, using the relative posture information of each vehicle and each laser point cloud data, obtain the fused point cloud data corresponding to each time frame, and obtain the first object detection information of the dynamic traffic object corresponding to each time frame according to each fused point cloud data and each vehicle-mounted camera image data; the dynamic traffic object is the traffic object represented by the dynamic point cloud data.

[0049] The fused point cloud data is obtained by fusing multiple frames of laser point cloud data. The fused point cloud data corresponding to each time frame can be, for example, the laser point cloud data of the historical time frame before the current frame is fused with the laser point cloud data of the current frame to obtain the fused point cloud data of the current frame. The first object detection information is the object detection information for dynamic traffic objects directly obtained from the fused point cloud data and the on-board camera image data, where the dynamic traffic objects refer to the traffic objects represented by the dynamic point cloud data, i.e. pedestrians and vehicles, etc.

[0050] Specifically, the automobile controller can use the relative posture information of the vehicle in each time frame and the laser point cloud data of each time frame to realize the fusion of laser point cloud data, thereby obtaining the fused point cloud data corresponding to each time frame, and then combine the fused point cloud data corresponding to each time frame and the on-board camera image data corresponding to each time frame, so as to output the first object detection information of each dynamic traffic object in each time frame.

[0051] Step S104, using each first object detection information, obtaining the object motion trajectory of the dynamic traffic object in the preset time interval, and obtaining the first object annotation data of the dynamic traffic object at different time points in the preset time interval based on the object motion trajectory.

[0052] The object motion trajectory refers to the motion trajectory of the dynamic traffic object in the preset time interval, which can be used to characterize the position change relationship of the dynamic traffic object in the preset time interval, and the first object annotation data refers to the dynamic object annotation data at any time point in the preset time interval, for example, it may include the position and speed of the dynamic object at different time points. Specifically, after the automobile controller obtains the first object detection information of each dynamic traffic object in each time frame, it can also obtain the motion trajectory of each dynamic traffic object in the preset time interval based on the above first object detection information, and then obtain the first object annotation data of the dynamic traffic object at any time point in the preset time interval based on each motion trajectory.

[0053] Taking 10 frames per second and each frame 0.1s apart as an example, the car controller can obtain 10 frames of first object detection information of dynamic traffic objects in step S103, which correspond to the first object detection information of 0.1s, 0.2s, ..., 1s respectively. After that, the car controller can obtain the object motion trajectory of the dynamic traffic object within 0-1s based on the above first object detection information, thereby obtaining the first object annotation data of the dynamic traffic object at any time point within 0-1s, for example, it can also include 0.05s.

[0054] In the above-mentioned dynamic traffic object labeling method, the vehicle-mounted camera image data, laser point cloud data and IMU data of each time frame in a preset time interval are obtained; each laser point cloud data includes static point cloud data and dynamic point cloud data; according to the IMU data and each static point cloud data, the relative posture information of the vehicle corresponding to each time frame is obtained; using the relative posture information of each vehicle and each laser point cloud data, the fused point cloud data corresponding to each time frame is obtained, and according to each fused point cloud data and each vehicle-mounted camera image data, the first object detection information of the dynamic traffic object corresponding to each time frame is obtained; the dynamic traffic object is a traffic object represented by the dynamic point cloud data; using each first object detection information, the object motion trajectory of the dynamic traffic object in the preset time interval is obtained, and based on the object motion trajectory, the first object labeling data of the dynamic traffic object at different time points in the preset time interval is obtained. The present application obtains multiple frames of vehicle-mounted camera image data within a preset time interval, including laser point cloud data and IMU data of static point cloud and dynamic point cloud, and then obtains the vehicle relative posture information corresponding to each time frame according to the IMU data and the static point cloud data, so as to fuse the laser point cloud data using the vehicle relative posture information, and then detects the target object using the fused point cloud data and the vehicle-mounted camera image data. After obtaining the first object detection information of the dynamic traffic object, the object motion trajectory of the dynamic traffic object in the preset time interval can be obtained, so as to obtain the object annotation data at different time points. In this way, the annotation of dynamic traffic objects can be achieved without scene reconstruction, so the annotation accuracy of dynamic traffic objects can be improved.

[0055] In one embodiment, each first object detection information and each first object annotation data respectively corresponds to a vehicle coordinate system at a different time point; as shown in FIG2 , step S104 may further include:

[0056] Step S201, using the relative posture information of the vehicle corresponding to each time frame, coordinate conversion is performed on each first object detection information to obtain the second object detection information corresponding to each time frame; the second object detection information corresponds to the global coordinate system; the relative posture information of each vehicle represents the transformation relationship between each vehicle coordinate system and the global coordinate system.

[0057] The vehicle coordinate system refers to the vehicle coordinate system. Since the position of the vehicle changes at different time points, the vehicle coordinate system will also change accordingly at different time points. Each first object detection information and each first object annotation data are the corresponding object detection information and object annotation data in the vehicle coordinate system at the corresponding time point. For example, the first object detection information of time frame 1 refers to the object detection information of time frame 1 in the vehicle coordinate system corresponding to time frame 1, and the first object detection information of time frame 2 refers to the object detection information of time frame 2 in the vehicle coordinate system corresponding to time frame 2, and so on.

[0058] The second object detection information is the object detection information in the global coordinate system. There is only one global coordinate system, which will not change with the position of the vehicle. That is, the same global coordinate system will be used at different time points. The second object detection information of time frame 1 refers to the object detection information of time frame 1 in the global coordinate system, and the second object detection information of time frame 2 refers to the object detection information of time frame 2 in the global coordinate system.

[0059] Since the vehicle's relative posture information can characterize the displacement and rotation relationship of the vehicle, it can also be used to characterize the transformation relationship between each vehicle coordinate system and the global coordinate system. Specifically, the vehicle controller can use the vehicle's relative posture information corresponding to each time frame to perform coordinate transformation on the first object detection information of each time frame, thereby obtaining the second object detection information of each time frame corresponding to the global coordinate system.

[0060] For example, coordinate conversion can refer to the following calculation formula:

[0061]

[0062] in, The second object detection information of the i-th time frame may be represented, can represent the relative position information of the vehicle in the i-th time frame, and The first object detection information of the i-th time frame may be represented, and It represents the size of the object detection box of the dynamic traffic object in the i-th time frame in the vehicle coordinate system and the global coordinate system respectively.

[0063] Step S202, according to each second object detection information, target tracking is performed on the dynamic traffic object to obtain the object motion trajectory, and the object motion trajectory is used to obtain second object annotation data of the dynamic traffic object at different time points in a preset time interval; the second object annotation data corresponds to the global coordinate system.

[0064] After obtaining the second object detection information of the dynamic traffic object in each time frame in the global coordinate system, since the second object detection information uses the same coordinate system at this time, the above discrete second object detection information can be used to track the dynamic traffic object to obtain a continuous object motion trajectory. After that, the continuous object motion trajectory can be used to obtain the second object annotation data of the dynamic traffic object corresponding to the global coordinate system at any different time point in the preset time interval.

[0065] Step S203, based on the vehicle relative posture information corresponding to each time frame, obtain the vehicle relative posture information at different time points in the preset time interval, and use the vehicle relative posture information at different time points in the preset time interval to perform coordinate inverse transformation on each second object annotation data, so as to obtain the first object annotation data of the dynamic traffic object at different time points in the preset time interval.

[0066] Finally, the relative posture information of the vehicle at any time point can be used to perform an inverse coordinate transformation on the second object annotation data of the dynamic traffic object corresponding to the global coordinate system at any time point to obtain the first object annotation data of the dynamic traffic object corresponding to the vehicle coordinate system at any time point. The relative posture information of the vehicle at any time point can be obtained through the relative posture information of the vehicle corresponding to each time frame. For example, the relative posture information of the vehicle at any time point between two time frames can be obtained by interpolation calculation.

[0067] In this embodiment, the first object detection information of each time frame can also be converted into coordinates through the relative posture information of the vehicle to obtain second object detection information corresponding to the global coordinate system, so as to achieve the unification of the coordinate system, and use the unified second object detection information to generate the object motion trajectory, which can improve the accuracy of the object motion trajectory. Thereafter, the second object annotation data at any time point is obtained by using the trajectory and then converted into the first object annotation data in the vehicle coordinate system corresponding to the arbitrary time point, which is conducive to further improving the acquisition accuracy of the first object annotation data.

[0068] Further, each second object detection information includes observation position data of the dynamic traffic object corresponding to each time frame; as shown in FIG3 , step S202 may further include:

[0069] Step S301 , based on the observed position data of each time frame, forward tracking is performed on the dynamic traffic object through a Kalman filter to obtain the first corrected position data of the last frame in each time frame.

[0070] The observed position data refers to the position data of the dynamic traffic object carried in the second object detection information. In this embodiment, each frame of the second object detection information of the dynamic traffic object may carry the position data of the dynamic traffic object, and the position data may be used as the observed position data, while the first corrected position data refers to the position data corrected during the forward tracking process using the Kalman filter.

[0071] Specifically, when the automobile controller obtains the second object detection information of each frame of the dynamic traffic object, it can extract the observation position data of each frame of the dynamic traffic object from the second object detection information. Then, in the process of forward tracking the dynamic traffic object by the Kalman filter, the observation position data of each frame can be used to obtain the first corrected position data of each frame of the dynamic traffic object. In this way, the first corrected position data of the last frame in each time frame can be obtained.

[0072] For example, there are a total of 10 frames of second object detection information, each carrying 10 frames of observation position data of dynamic traffic objects, namely observation position data 1, observation position data 2, ..., observation position data 10. After that, the automobile controller can track the dynamic traffic object through the Kalman filter based on the above observation position data to obtain the first corrected position data corresponding to each frame, namely the first corrected position data 1, the first corrected position data 2, ..., the first corrected position data 10. At this time, the first corrected position data 10 is used as the first corrected position data of the last frame in each time frame.

[0073] Step S302, based on the first corrected position data of the last frame and the observed position data of each time frame, the dynamic traffic object is tracked backwards through a Kalman filter to obtain second corrected position data of each time frame.

[0074] The second corrected position data refers to the position data corrected during the reverse tracking of the dynamic traffic object by the Kalman filter. In this embodiment, after forward tracking by the Kalman filter, the dynamic traffic object can be further reverse tracked to correct the position data of the dynamic traffic object in each time frame again, wherein the reverse tracking process requires the use of the first corrected position data of the last frame and the observed position data of each time frame.

[0075] Continuing with the example of 10 frames of second object detection information, after obtaining the first corrected position data 10, that is, the first corrected position data of the last frame, the second corrected position data of each frame can be obtained based on the first corrected position data 10 and 10 frames of observation position data of dynamic traffic objects, namely, second corrected position data 1, second corrected position data 2, ..., second corrected position data 10.

[0076] Step S303: obtaining the object motion trajectory according to the second corrected position data of each time frame.

[0077] Finally, after the second corrected position data of each frame of the dynamic traffic object is obtained, a linear fitting may be performed on the second corrected position data of each frame, thereby obtaining the object motion trajectory of the dynamic traffic object.

[0078] In this embodiment, the generated object motion trajectory can be obtained through the second corrected position data, and the second corrected position data is obtained by bidirectional tracking through the Kalman filter, that is, first forward tracking and then reverse tracking. This method can further improve the accuracy of obtaining the object motion trajectory.

[0079] In one embodiment, as shown in FIG. 4 , step S301 may further include:

[0080] Step S401, taking the first frame of each time frame as the current time frame, and taking the observed position data of the first frame as the first corrected position data of the current time frame;

[0081] Step S402: input the first corrected position data of the current time frame into a Kalman filter to obtain forward predicted position data of the next time frame corresponding to the current time frame.

[0082] The current time frame refers to any one of the time frames. During the forward tracking process, it is iterated from the first frame to the last frame. Therefore, the first frame needs to be used as the current time frame first. The forward predicted position data refers to the predicted position data of the dynamic traffic object in the next time frame of the current time frame obtained by the Kalman filter based on the first corrected position data of the input current time frame. When the current time frame is the first frame, the forward predicted position data of the next time frame of the current time frame is the forward predicted position data of the second frame.

[0083] Specifically, when performing forward tracking, the car controller can first use the first frame as the current time frame, and input the observed position data of the first frame as the first corrected position data of the current time frame into the Kalman filter, and the Kalman filter obtains the forward predicted position data of the second frame.

[0084] Step S403, obtaining first corrected position data of the next time frame according to the forward predicted position data of the next time frame and the observed position data of the next time frame, and updating the Kalman filter using the first corrected position data.

[0085] After obtaining the forward predicted position data of the next time frame, that is, the forward predicted position data of the second frame, the observed position data of the next time frame can be further combined to obtain the first corrected position data of the next time frame, that is, the forward predicted position data of the second frame is combined with the observed position data of the second frame to obtain the first corrected position data of the second frame. After that, the first corrected position data of the second frame can be used to update the Kalman filter, for example, the state estimate and covariance matrix of the Kalman filter can be updated.

[0086] Step S404, when the current time frame is not the last frame, the next time frame is used as the new current time frame, and the process returns to step S402 until the current time frame is the last frame of each time frame;

[0087] Step S405, when the current time frame is the last frame, the first corrected position data of the current time frame is used as the first corrected position data of the last frame.

[0088] Afterwards, it can be determined whether the current time frame is the tail frame. If not, the next time frame is used as the new current time frame, that is, the second frame is used as the new current time frame, and the execution is returned to input the first corrected position data of the second frame into the Kalman filter to obtain the forward predicted position data of the third frame, and so on, until the current time frame is the tail frame. At this time, the first corrected position data of the current time frame can be used as the first corrected position data of the tail frame.

[0089] Taking 10 frames of second object detection information as an example, the automobile controller can first input the observed position data of the first frame, that is, the observed position data 1, as the first corrected position data of the current time frame into the Kalman filter to obtain the forward predicted position data of the second frame, that is, the forward predicted position data 2. Then, the first corrected position data of the second frame, that is, the first corrected position data 2, can be obtained according to the forward predicted position data 2 and the observed position data of the second frame, that is, the observed position data 2, and the Kalman filter is updated at the same time. Since the second frame is not the last frame at this time, the second frame is used as the new current time frame. At this time, the first corrected position data 2 is the first corrected position data of the current time frame, so it can be input into the Kalman filter to obtain the forward predicted position data 3. Then, the first corrected position data of the third frame, that is, the first corrected position data 3, can be obtained according to the forward predicted position data 3 and the observed position data 3. The Kalman filter is updated at the same time, and so on, until the current time frame is the last frame. After obtaining the first corrected position data of the current time frame, that is, the first corrected position data 10, it can be used as the first corrected position data of the last frame.

[0090] In this embodiment, during the forward tracking process, the predicted output of the Kalman filter and the observed output of the second object detection information can be used to obtain the first corrected position data of each frame, and the Kalman filter can be continuously optimized in an iterative manner, thereby improving the accuracy of forward tracking.

[0091] In addition, as shown in FIG5 , step S302 may further include:

[0092] Step S501, taking the last frame as the current time frame, and taking the first corrected position data of the last frame as the second corrected position data of the current time frame;

[0093] Step S502: input the second corrected position data of the current time frame into the Kalman filter to obtain the reverse predicted position data of the previous time frame corresponding to the current time frame.

[0094] Contrary to forward tracking, in the process of reverse tracking, the iteration is performed from the last frame to the first frame. Therefore, the last frame needs to be used as the current time frame first. The reverse predicted position data refers to the predicted position data of the dynamic traffic object in the previous time frame of the current time frame obtained by the Kalman filter based on the second corrected position data of the input current time frame. When the current time frame is the last frame, the forward predicted position data of the previous time frame of the current time frame is the reverse predicted position data of the second to last frame.

[0095] Specifically, when performing reverse tracking, the car controller can first use the tail frame as the current time frame, and input the first corrected position data of the tail frame as the second corrected position data of the current time frame into the Kalman filter, and the Kalman filter obtains the reverse predicted position data of the penultimate frame.

[0096] Step S503, obtaining the second corrected position data of the previous time frame according to the reverse predicted position data of the previous time frame and the observed position data of the previous time frame, and updating the Kalman filter using the second corrected position data.

[0097] After obtaining the reverse predicted position data of the previous time frame, that is, the reverse predicted position data of the penultimate frame, the observed position data of the previous time frame can be further combined to obtain the second corrected position data of the previous time frame, that is, the reverse predicted position data of the penultimate frame is combined with the observed position data of the penultimate frame to obtain the second corrected position data of the penultimate frame. After that, the second corrected position data of the penultimate frame can be used to update the Kalman filter, for example, the state estimate and covariance matrix of the Kalman filter can be updated again.

[0098] Step S504, when the current time frame is not the first frame of all time frames, the previous time frame is used as the new current time frame, and the process returns to step S502 until the current time frame is the first frame;

[0099] Step S505, when the current time frame is the first frame, the second corrected position data of each current time frame is used as the second corrected position data of each time frame.

[0100] Afterwards, it can be determined whether the current time frame is the first frame. If not, the previous time frame is used as the new current time frame, that is, the second-to-last frame is used as the new current time frame, and the execution is returned to input the second corrected position data of the second-to-last frame into the Kalman filter to obtain the reverse predicted position data of the third-to-last frame, and so on, until the current time frame is the first frame, at which time the second corrected position data of each time frame can be obtained.

[0101] Taking 10 frames of second object detection information as an example, the automobile controller can first input the first corrected position data of the tail frame, that is, the first corrected position data 10, as the second corrected position data of the current time frame into the Kalman filter to obtain the reverse predicted position data of the 9th frame, that is, the reverse predicted position data 9. Then, the second corrected position data of the 9th frame, that is, the second corrected position data 9, can be obtained based on the reverse predicted position data 9 and the observed position data 9, and the Kalman filter is updated at the same time. Since the 9th frame is not the first frame at this time, the 9th frame is used as the new current time frame. At this time, the second corrected position data 9 is the second corrected position data of the current time frame, so it can be input into the Kalman filter to obtain the reverse predicted position data 8. Then, the second corrected position data of the 8th frame, that is, the second corrected position data 8, can be obtained based on the forward predicted position data 8 and the observed position data 8. The Kalman filter is updated at the same time, and so on, until the current time frame is the first frame, at which time the second corrected position data of each frame can be obtained.

[0102] In this embodiment, during the reverse tracking process, the predicted output of the Kalman filter and the observed output of the second object detection information can also be used to obtain the second corrected position data of each frame, and the Kalman filter can be continuously optimized in an iterative manner, which can further improve the accuracy of reverse tracking.

[0103] In one embodiment, each second object annotation data includes second position data, second speed data and second orientation data of the dynamic traffic object at each time point in a preset time interval; each second object detection information includes orientation data of the dynamic traffic object corresponding to each time frame; step S202 may further include: obtaining the second position data of the dynamic traffic object at each time point according to the object motion trajectory; performing differential calculation on the object motion trajectory to obtain the second speed data of the dynamic traffic object at each time point; interpolating the orientation data of each time frame to obtain the second orientation data of the dynamic traffic object at each time point.

[0104] In this embodiment, the second object annotation data may include three parts, namely, the position data of the dynamic traffic object corresponding to the global coordinate system at each time point, i.e., the second position data, the speed data of the dynamic traffic object corresponding to the global coordinate system at each time point, i.e., the second speed data, and the orientation data of the dynamic traffic object corresponding to the global coordinate system at each time point, i.e., the second orientation data. And the second object detection information may also include the orientation data of the dynamic traffic object corresponding to each time frame.

[0105] Specifically, the second position data of each time point can be obtained directly from the object's motion trajectory, that is, the trajectory point corresponding to the object's motion trajectory at each time point is used as the second position data of each time point, and the second speed data of each time point can be obtained by differentiating the object's motion trajectory, for example, a linear motion equation can be used in the x and y directions to fit the target's motion state and calculate the speed, and the second orientation data can be obtained by interpolating the orientation data of adjacent time frames through the slerp spherical linear interpolation method. The second position data, second speed data and second orientation data of each time point can be obtained in the above manner.

[0106] In this embodiment, the second position data, second speed data and second orientation data of the dynamic traffic object at each time point can also be obtained respectively through the object motion trajectory and the orientation data of each time frame carried in the second object detection information. In this way, the integrity of the second object annotation data can be improved.

[0107] Furthermore, each first object annotation data includes the first position data, first speed data and first direction data of the dynamic traffic object corresponding to each time point; step S203 may further include: using the relative posture information of the vehicle at each time point to obtain the rotation matrix information corresponding to each time point in the preset time interval; each rotation matrix information represents the rotation relationship between each vehicle coordinate system and the global coordinate system; using the relative posture information of the vehicle at each time point, performing an inverse coordinate transformation on each second position data to obtain each first position data; using the rotation matrix information corresponding to each time point, performing an inverse coordinate transformation on each second speed data and second direction data to obtain each first speed data and first direction data.

[0108] Similarly, the first object annotation data may also include three parts, namely, the position data of the dynamic traffic object at each time point corresponding to the vehicle coordinate system at that time point, i.e., the first position data, the speed data of the dynamic traffic object at each time point corresponding to the vehicle coordinate system at that time point, i.e., the first speed data, and the orientation data of the dynamic traffic object at each time point corresponding to the vehicle coordinate system at that time point, i.e., the first orientation data. The rotation matrix information is the matrix information used to characterize the rotation coordinate transformation relationship between the vehicle coordinate system and the global coordinate system, and the matrix information can be obtained based on the relative posture information of the vehicle at each time point.

[0109] Specifically, after the automobile controller obtains the relative position information of the vehicle at each time point, it can further obtain the rotation matrix information of each time point. Thereafter, the second position data at each time point can be inversely transformed through the relative position information of the vehicle at each time point to obtain the first position data at each time point. Furthermore, the second speed data and second direction data at each time point can be inversely transformed through the rotation matrix information corresponding to each time point to obtain the first speed data and first direction data at each time point.

[0110] For example, it can be calculated by the following formula:

[0111]

[0112] in, , , are respectively first position data, first orientation data and first speed data, , , are the second position data, the second orientation data and the second speed data respectively, is the relative position information of the vehicle at a certain time point, It is the rotation matrix information at a certain point in time.

[0113] In this embodiment, the second position data, second orientation data and second speed data can also be inversely transformed by using the relative posture information of the vehicle at each time point and the rotation matrix information, so as to obtain the first position data, first orientation data and first speed data. In this way, the accuracy and completeness of the first object annotation data can be improved.

[0114] In one embodiment, step S103 may further include: obtaining the current time frame, and a preset number of historical time frames corresponding to the current time frame; the current time frame is any one of the time frames, and the historical time frames are a preset number of time frames adjacent to the current time frame and located before the current time frame; according to the laser point cloud data corresponding to the current time frame, the vehicle relative posture information corresponding to the current time frame, the laser point cloud data corresponding to each historical time frame, and the vehicle relative posture information corresponding to each historical time frame, the fused point cloud data corresponding to the current time frame is obtained.

[0115] The historical time frame refers to a time frame that is adjacent to the current time frame and is located a preset number of times before the current time frame. For example, the preset number may be 4, and the historical time frame refers to the first 4 frames of the current time frame. In this embodiment, the fused point cloud data of the current time frame is obtained by fusing the laser point cloud data of the first 4 frames of the current time frame with the laser point cloud data of the current time frame. The fusion method is to use the laser point cloud data corresponding to the current time frame, the vehicle relative posture information corresponding to the current time frame, the laser point cloud data corresponding to each historical time frame, and the vehicle relative posture information corresponding to each historical time frame.

[0116] Specifically, the fused point cloud data can be obtained by the following formula:

[0117]

[0118] in, Indicates the frame i=0, that is, the fused point cloud data of the current frame, SE3 kg and SE3 ig Respectively represent the relative position information of the vehicle corresponding to the current frame and the i-th frame before the historical time frame, P i Represents the historical point cloud data of the i frames before the current frame, where i=0 represents the historical point cloud data of the current frame, and i<0 represents the laser point cloud data corresponding to each historical time frame.

[0119] In this embodiment, the fused point cloud data corresponding to the current time frame can be obtained by combining the laser point cloud data corresponding to the current time frame, the vehicle relative posture information corresponding to the current time frame, the laser point cloud data corresponding to a preset number of historical time frames before the current time frame, and the vehicle relative posture information corresponding to each historical time frame. In this way, the accuracy of the fused point cloud data can be improved.

[0120] In addition, as shown in FIG6 , step S103 may further include:

[0121] Step S601, input the fused point cloud data corresponding to the current time frame and the vehicle-mounted camera image data corresponding to the current time frame into a pre-built target detection model, and obtain the fused point cloud features and vehicle-mounted camera image features corresponding to the current time frame through the target detection model.

[0122] The target detection model can be a pre-trained Bevfusion algorithm model. In this embodiment, the vehicle controller can input the fused point cloud data corresponding to the current time frame and the on-board camera image data into the pre-built target detection model, thereby obtaining the fused point cloud features and the on-board camera image features corresponding to the current time frame through the encoder of the target detection model.

[0123] Step S602, converting the fused point cloud features and the onboard camera image features into BEV vision, and fusing the fused point cloud features and the onboard camera image features after the BEV vision conversion to obtain a fused BEV feature;

[0124] Step S603 , obtaining the dynamic traffic objects contained in the current time frame and the first object detection information of the dynamic traffic objects according to the fused BEV features.

[0125] After obtaining the fused point cloud features and the on-board camera image features, the target detection model can also convert the fused point cloud features and the on-board camera image features into BEV vision, and fuse the two features converted into BEV vision to obtain the fused BEV features, which are then decoded by the decoder of the target detection model to obtain the dynamic traffic objects contained in the current time frame and the first object detection information of the dynamic traffic objects.

[0126] In this embodiment, the target detection model can also be used to fuse the fused point cloud data corresponding to the current time frame and the vehicle-mounted camera image data corresponding to the current time frame, and convert them into BEV vision to obtain the final first object detection information. In this way, the accuracy of obtaining the first object detection information can be improved.

[0127] Furthermore, step S603 may further include: obtaining candidate dynamic traffic objects included in the current time frame and the confidence corresponding to each candidate dynamic traffic object based on the fused BEV features; and taking the candidate dynamic traffic objects whose confidence is greater than a preset confidence threshold as the dynamic traffic objects included in the current time frame.

[0128] In this embodiment, dynamic traffic objects can be presented in the form of target detection frames, and candidate dynamic traffic objects can be presented in the form of candidate target detection frames. In this embodiment, the target detection model will output a large number of candidate target detection frames, that is, output a large number of candidate dynamic traffic objects contained in the current time frame. In order to ensure the reliability of the target detection frame, the NMS algorithm can also be combined to obtain each candidate target detection frame, that is, the confidence corresponding to each candidate dynamic traffic object. Only the candidate dynamic traffic object whose confidence is greater than the preset confidence threshold will be used as the dynamic traffic object contained in the current time frame.

[0129] In this embodiment, after obtaining multiple candidate dynamic traffic objects, the confidence level corresponding to each candidate dynamic traffic object can also be obtained, and the dynamic traffic objects included in the final current time frame can be screened out by pre-setting the confidence threshold. In this way, the reliability of dynamic traffic object acquisition can be improved.

[0130] In one embodiment, a method for automatic labeling of traffic participant targets by point cloud image fusion is also provided. The method can realize automatic labeling of traffic participant targets based on computer vision and deep learning technology, combined with data from vehicle-mounted cameras and lidar, and specifically includes the following steps:

[0131] Step 1: Obtain continuous frames of camera images (360° panoramic images composed of multiple cameras), laser radar point cloud data, GNSS, IMU, wheel speed and other data. Each point in each frame of laser point cloud contains location and timestamp information, the GNSS / IMU data frequency is above 100hz, and the laser point cloud and GNSS / IMU must have the same clock source.

[0132] Step 2: All points are divided into dynamic target points and static points through the point cloud segmentation model SPVNAS. Specifically, the model predicts the dynamic and static state of each point and outputs a predicted value. If the value is greater than the preset threshold, it is a dynamic point, otherwise it is a static point, as shown in Figure 7, where dark points represent dynamic target points, such as the car included in Figure 7, and light points represent static points, such as the trees and buildings included in Figure 7.

[0133] Step 3: For static point clouds, combine the previous 10 consecutive frames of static point clouds and IMU data, and use the lio-sam algorithm to output a short-term laser odometer based on relative posture.

[0134] Step 4: For the dynamic target point cloud, fuse the previous 5 consecutive frames of point cloud, and use the bevfusion algorithm to fuse the point cloud and the corresponding image data to obtain the 3D target detection frame and specific type of the traffic participant target. In actual applications, the model will output a large number of candidate detection frames, which can be combined with the NMS algorithm and set a suitable confidence threshold. When the confidence of the 3D target detection frame output by the bevfusion algorithm is greater than the confidence threshold, the detection frame is considered reliable.

[0135] Among them, the multi-frame point cloud fusion solution can be expressed by the following formula:

[0136]

[0137] Where P represents the fused point cloud set, P i Indicates the i-frame point cloud before the current frame point cloud, SE3 kg and SE3 ig They represent the vehicle poses of the current frame and the i-th frame respectively, which are obtained from step 3. After testing, the model achieves the best balance between inference speed and accuracy by fusing the first five frames of point clouds.

[0138] The model structure of the Bevfusion algorithm is shown in FIG8 . The CNN model is used to extract feature information from the above-mentioned fused point cloud frame and the image data of the current frame, respectively. The CNN model includes an input layer and a feature extraction layer. The input layer is used to input the fused point cloud frame and image data. The feature extraction layer can extract image features based on the input image data, such as extracting image features through the convolution layer, pooling layer and fully connected layer contained in the feature extraction layer. It can also extract point cloud features based on the input fused point cloud frame, such as through PointNet or its variants, using a specific network structure (such as MLP, multi-layer perceptron, feature aggregation module, etc.) to process the point cloud data to extract its geometric features. Then, the feature transformation layer in the feature fusion layer is used to switch to the BEV perspective, and the feature fusion is performed through the fusion layer in the feature fusion layer. Finally, the fused BEV features are input into the detection head, that is, input into the output layer and the bounding box regression and classification layer to finally output 3D target detection information, including the three-dimensional spatial position, orientation, and category of the target. The Bevfusion algorithm also supports tasks such as map segmentation. In this embodiment, only the target detection task needs to be completed. The detected targets include vehicles, pedestrians, cyclists and other traffic participants.

[0139] Step 5: Use the laser odometer obtained in step 3 to convert the 3D target frame to the global coordinate system, use the bidirectional MOT algorithm to track the target and Kalman filter smoothing to obtain the target's motion trajectory, and calculate the target's motion state by differentiation. For example, use a linear motion equation in the x and y directions to fit the target's motion state and calculate the speed.

[0140] Among them, the 3D target frame can be transferred to the global coordinate system through the following formula:

[0141]

[0142] Bidirectional MOT is divided into forward and reverse multi-target tracking. First, forward MOT is performed. The Kalman filter is used to predict the position of the detected target frame in the next frame. Then the target detection result of the next frame is used as observation, and Hungarian matching is used to fuse and update the Kalman filter. Then, reverse MOT is performed from back to front. The state estimate and covariance matrix of the Kalman filter at the last moment obtained by forward tracking are used as the initial state estimate and covariance matrix of the reverse tracking recursive process to complete reverse multi-target tracking and fixed interval smoothing of the target position. Finally, a better target motion trajectory is obtained.

[0143] After obtaining the discrete target motion trajectory, a linear fit is performed on the target's motion in the x and y directions, and then the speed of the target in the x and y directions at any time is calculated by differentiation. For the target's orientation, slerp spherical linear interpolation is used for correction. The specific formula is as follows:

[0144]

[0145] Step 6: For all traffic participant target frames in the global coordinate system, transform them to the vehicle coordinate system at the current moment through the RT matrix at the current moment to obtain the final automatic annotation truth value.

[0146] Since the true values ​​of the position, orientation, speed, etc. of the traffic participant target in the global coordinate system have been obtained, the relative position and pose of the ego vehicle calculated in step 3 can be used to calculate their true values ​​in the ego vehicle coordinate system as the true values ​​of the data annotation:

[0147]

[0148] Among them, Pos, θ, V, respectively represent the position coordinates, orientation and speed of the target object, and the subscripts v and g represent the coordinate system of the vehicle and the global coordinate system, respectively. vg Represents the vehicle's position, that is, the transformation matrix from the vehicle coordinate system to the global coordinate system, SO3 vg Represents the rotation matrix from the vehicle coordinate system to the global coordinate system, and inv() means taking the inverse of the matrix.

[0149] In this embodiment, the size and orientation of the moving target are obtained by the 4D target detection algorithm bevfusion of image and point cloud time series fusion. The basic detection accuracy is improved compared with the pure point cloud method. In combination with the forward and reverse bidirectional multi-target tracking algorithm for continuous frames and the Kalman filter to smooth the trajectory of the target, the target detection result can be effectively optimized, and the accuracy and robustness of the detection algorithm are improved. In addition, the motion state of the moving target can be accurately estimated, and the true value of the target's motion speed can be calculated without the introduction of millimeter wave radar. Only continuous frame laser point cloud data and high-frequency IMU data of the vehicle are required, and no additional hardware investment is required. In addition, the computing resource consumption of this embodiment is low, and the balance between processing speed and accuracy is achieved. In the offline processing process, only 10 consecutive frames within the first second need to be processed offline, and the point cloud needs to be processed in batches. This method can not only improve the processing efficiency, but also ensure real-time performance, solving the problems of low processing efficiency and poor real-time performance that may occur in the existing algorithm when processing large-scale laser point cloud data.

[0150] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0151] Based on the same inventive concept, the embodiment of the present application also provides a dynamic traffic object labeling device for implementing the dynamic traffic object labeling method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more dynamic traffic object labeling device embodiments provided below can refer to the limitations of the dynamic traffic object labeling method above, and will not be repeated here.

[0152] In one embodiment, as shown in FIG. 9 , a dynamic traffic object labeling device is provided, including: a detection data acquisition module 901, a relative posture acquisition module 902, a detection information acquisition module 903 and a labeling data acquisition module 904, wherein:

[0153] The detection data acquisition module 901 is used to acquire the vehicle-mounted camera image data, laser point cloud data and IMU data of each time frame in a preset time interval; each of the laser point cloud data includes static point cloud data and dynamic point cloud data;

[0154] A relative posture acquisition module 902 is used to obtain the relative posture information of the vehicle corresponding to each of the time frames according to the IMU data and each of the static point cloud data;

[0155] The detection information acquisition module 903 is used to obtain the fused point cloud data corresponding to each of the time frames using the relative posture information of each of the vehicles and the laser point cloud data, and obtain the first object detection information of the dynamic traffic object corresponding to each of the time frames according to the fused point cloud data and the image data of each of the vehicle-mounted cameras; the dynamic traffic object is the traffic object represented by the dynamic point cloud data;

[0156] The annotation data acquisition module 904 is used to use the first object detection information to obtain the object motion trajectory of the dynamic traffic object in the preset time interval, and based on the object motion trajectory, obtain the first object annotation data of the dynamic traffic object at different time points in the preset time interval.

[0157] In one embodiment, each of the first object detection information and each of the first object annotation data corresponds to a vehicle coordinate system at different time points; the annotation data acquisition module 904 is further used to use the relative posture information of the vehicle corresponding to each of the time frames to perform coordinate transformation on each of the first object detection information to obtain second object detection information corresponding to each of the time frames; the second object detection information corresponds to a global coordinate system; each of the relative posture information of the vehicle represents the transformation relationship between each of the vehicle coordinate systems and the global coordinate system; according to each of the second object detection information, the dynamic traffic object is tracked to obtain the object motion trajectory, and the second object annotation data of the dynamic traffic object at different time points in the preset time interval is obtained by using the object motion trajectory; the second object annotation data corresponds to a global coordinate system; according to the relative posture information of the vehicle corresponding to each of the time frames, the relative posture information of the vehicle at different time points in the preset time interval is obtained, and the relative posture information of the vehicle at different time points in the preset time interval is used to perform coordinate inverse transformation on each of the second object annotation data to obtain the first object annotation data of the dynamic traffic object at different time points in the preset time interval.

[0158] In one embodiment, each second object detection information includes the observed position data of the dynamic traffic object corresponding to each time frame; the annotation data acquisition module 904 is further used to forward track the dynamic traffic object through a Kalman filter based on the observed position data of each time frame to obtain the first corrected position data of the tail frame in each time frame; based on the first corrected position data of the tail frame and the observed position data of each time frame, reverse track the dynamic traffic object through a Kalman filter to obtain the second corrected position data of each time frame; according to the second corrected position data of each time frame, obtain the object motion trajectory.

[0159] In one embodiment, the annotation data acquisition module 904 is further used to take the first frame of each time frame as the current time frame, and take the observed position data of the first frame as the first corrected position data of the current time frame; input the first corrected position data of the current time frame into the Kalman filter to obtain the forward predicted position data of the next time frame corresponding to the current time frame; obtain the first corrected position data of the next time frame according to the forward predicted position data of the next time frame and the observed position data of the next time frame, and update the Kalman filter using the first corrected position data; when the current time frame is not the tail frame, take the next time frame as the new current time frame, and return to execute the step of inputting the first corrected position data of the current time frame into the Kalman filter until the current time frame is the tail frame; when the current time frame is the tail frame, take the first corrected position data of the current time frame as the first corrected position data of the tail frame.

[0160] In one embodiment, the annotation data acquisition module 904 is further used to take the tail frame as the current time frame, and take the first corrected position data of the tail frame as the second corrected position data of the current time frame; input the second corrected position data of the current time frame into the Kalman filter to obtain the reverse predicted position data of the previous time frame corresponding to the current time frame; obtain the second corrected position data of the previous time frame according to the reverse predicted position data of the previous time frame and the observed position data of the previous time frame, and update the Kalman filter using the second corrected position data; when the current time frame is not the first frame of each time frame, take the previous time frame as the new current time frame, and return to execute the step of inputting the second corrected position data of the current time frame into the Kalman filter until the current time frame is the first frame; when the current time frame is the first frame, take the second corrected position data of each current time frame as the second corrected position data of each time frame.

[0161] In one embodiment, each of the second object annotation data includes second position data, second speed data and second orientation data of the dynamic traffic object at each time point in the preset time interval; each of the second object detection information includes orientation data of the dynamic traffic object corresponding to each of the time frames;

[0162] In one embodiment, the annotation data acquisition module 904 is further used to obtain the second position data of the dynamic traffic object at each time point based on the object motion trajectory; perform differential calculation on the object motion trajectory to obtain the second speed data of the dynamic traffic object at each time point; and perform interpolation processing on the orientation data of each time frame to obtain the second orientation data of the dynamic traffic object at each time point.

[0163] In one embodiment, each first object annotation data includes the first position data, first speed data and first orientation data of the dynamic traffic object corresponding to each time point; the annotation data acquisition module 904 is further used to use the relative posture information of the vehicle at each time point to obtain the rotation matrix information corresponding to each time point in the preset time interval; each rotation matrix information represents the rotation relationship between each vehicle coordinate system and the global coordinate system; using the relative posture information of the vehicle at each time point, the coordinates of each second position data are inversely transformed to obtain each first position data; using the rotation matrix information corresponding to each time point, the coordinates of each second speed data and second orientation data are inversely transformed to obtain each first speed data and first orientation data.

[0164] In one embodiment, the detection information acquisition module 903 is further used to acquire the current time frame and a preset number of historical time frames corresponding to the current time frame; the current time frame is any one of the time frames, and the historical time frames are a preset number of time frames adjacent to the current time frame and located before the current time frame; according to the laser point cloud data corresponding to the current time frame, the vehicle relative posture information corresponding to the current time frame, the laser point cloud data corresponding to each of the historical time frames, and the vehicle relative posture information corresponding to each of the historical time frames, the fused point cloud data corresponding to the current time frame is obtained.

[0165] In one embodiment, the detection information acquisition module 903 is further used to input the fused point cloud data corresponding to the current time frame and the vehicle-mounted camera image data corresponding to the current time frame into a pre-constructed target detection model, and obtain the fused point cloud features and vehicle-mounted camera image features corresponding to the current time frame through the target detection model; convert the fused point cloud features and vehicle-mounted camera image features into BEV vision, and fuse the fused point cloud features and vehicle-mounted camera image features after the BEV vision conversion to obtain a fused BEV feature; according to the fused BEV feature, obtain the dynamic traffic objects contained in the current time frame and the first object detection information of the dynamic traffic objects.

[0166] In one embodiment, the detection information acquisition module 903 is further used to obtain the candidate dynamic traffic objects included in the current time frame and the confidence corresponding to each of the candidate dynamic traffic objects based on the fused BEV features; and the candidate dynamic traffic objects whose confidence is greater than a preset confidence threshold are taken as the dynamic traffic objects included in the current time frame.

[0167] Each module in the above-mentioned dynamic traffic object labeling device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the vehicle controller in the form of hardware, or can be stored in the memory in the vehicle controller in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0168] In one embodiment, a car controller is provided, which can be a terminal, and its internal structure diagram can be shown in FIG10. The car controller includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the car controller is used to provide computing and control capabilities. The memory of the car controller includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the car controller is used to exchange information between the processor and an external device. The communication interface of the car controller is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a dynamic traffic object labeling method is implemented.

[0169] Those skilled in the art will understand that the structure shown in FIG. 10 is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the automobile controller to which the scheme of the present application is applied. A specific automobile controller may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0170] In one embodiment, an automobile controller is further provided, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0171] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0172] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0173] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0174] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0175] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0176] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A dynamic traffic object labeling method, characterized in that: The method comprises: Acquire vehicle-mounted camera image data, laser point cloud data and IMU data of each time frame in a preset time interval; each of the laser point cloud data includes static point cloud data and dynamic point cloud data; According to the IMU data and the static point cloud data, the relative posture information of the vehicle corresponding to each time frame is obtained; Using the relative posture information of each vehicle and each laser point cloud data, fused point cloud data corresponding to each time frame is obtained, and according to each fused point cloud data and each vehicle-mounted camera image data, first object detection information of a dynamic traffic object corresponding to each time frame is obtained; the dynamic traffic object is a traffic object represented by the dynamic point cloud data; The object motion trajectory of the dynamic traffic object in the preset time interval is obtained by using each of the first object detection information, and the first object labeling data of the dynamic traffic object at different time points in the preset time interval is obtained based on the object motion trajectory.

2. The method according to claim 1, characterized in that Each of the first object detection information and each of the first object annotation data corresponds to a vehicle coordinate system at a different time point; The method of using each of the first object detection information to obtain the object motion trajectory of the dynamic traffic object in the preset time interval, and obtaining first object labeling data of the dynamic traffic object at different time points in the preset time interval based on the object motion trajectory, includes: Using the relative posture information of the vehicles corresponding to the time frames, coordinate transformation is performed on each of the first object detection information to obtain second object detection information corresponding to each of the time frames; the second object detection information corresponds to a global coordinate system; and the relative posture information of each of the vehicles represents a transformation relationship between the coordinate system of each of the vehicles and the global coordinate system; According to each of the second object detection information, target tracking is performed on the dynamic traffic object to obtain the object motion trajectory, and the object motion trajectory is used to obtain second object annotation data of the dynamic traffic object at different time points in the preset time interval; the second object annotation data corresponds to a global coordinate system; The relative posture information of the vehicle at different time points in the preset time interval is obtained according to the relative posture information of the vehicle corresponding to each of the time frames, and the relative posture information of the vehicle at different time points in the preset time interval is used to perform an inverse coordinate transformation on each of the second object annotation data to obtain the first object annotation data of the dynamic traffic object at different time points in the preset time interval.

3. The method according to claim 2, characterized in that Each of the second object detection information includes observation position data of the dynamic traffic object corresponding to each of the time frames; and tracking the dynamic traffic object according to each of the second object detection information to obtain the object motion trajectory includes: Based on the observed position data of each time frame, forward tracking of the dynamic traffic object is performed through a Kalman filter to obtain first corrected position data of the last frame in each time frame; Based on the first corrected position data of the tail frame and the observed position data of each time frame, the dynamic traffic object is tracked backwards through a Kalman filter to obtain second corrected position data of each time frame; The object motion trajectory is obtained according to the second corrected position data of each time frame.

4. The method according to claim 3, characterized in that The method of forward tracking the dynamic traffic object based on the observed position data of each time frame through a Kalman filter to obtain the first corrected position data of the tail frame in each time frame includes: Taking the first frame of each of the time frames as the current time frame, and taking the observed position data of the first frame as the first corrected position data of the current time frame; Inputting the first corrected position data of the current time frame into a Kalman filter to obtain forward predicted position data of a next time frame corresponding to the current time frame; Obtaining first corrected position data of the next time frame according to the forward predicted position data of the next time frame and the observed position data of the next time frame, and updating the Kalman filter using the first corrected position data; In the case where the current time frame is not the tail frame, taking the next time frame as a new current time frame, and returning to the step of inputting the first corrected position data of the current time frame into the Kalman filter until the current time frame is the tail frame; When the current time frame is the last frame, the first corrected position data of the current time frame is used as the first corrected position data of the last frame.

5. The method according to claim 3, characterized in that: The first corrected position data based on the tail frame and the observed position data of each time frame are used to reversely track the dynamic traffic object through a Kalman filter to obtain the second corrected position data of each time frame, including: Taking the tail frame as the current time frame, and taking the first corrected position data of the tail frame as the second corrected position data of the current time frame; Inputting the second corrected position data of the current time frame into a Kalman filter to obtain reverse predicted position data of a previous time frame corresponding to the current time frame; Obtaining second corrected position data of the previous time frame according to the reverse predicted position data of the previous time frame and the observed position data of the previous time frame, and updating the Kalman filter using the second corrected position data; In the case where the current time frame is not the first frame of each of the time frames, the previous time frame is used as a new current time frame, and the step of inputting the second corrected position data of the current time frame into the Kalman filter is returned to be executed until the current time frame is the first frame; In the case where the current time frame is the first frame, the second corrected position data of each current time frame is used as the second corrected position data of each time frame.

6. The method according to claim 2, characterized in that Each of the second object annotation data includes the second position data, the second speed data and the second orientation data of the dynamic traffic object at each time point in the preset time interval; each of the second object detection information includes the orientation data of the dynamic traffic object corresponding to each of the time frames; The obtaining of second object annotation data of the dynamic traffic object at different time points in the preset time interval by using the object motion trajectory includes: According to the object motion trajectory, obtaining second position data of the dynamic traffic object at each of the time points; Performing differential calculation on the object motion trajectory to obtain second speed data of the dynamic traffic object at each time point; The orientation data of each of the time frames is interpolated to obtain second orientation data of the dynamic traffic object at each of the time points.

7. The method according to claim 6, characterized in that Each of the first object annotation data includes the first position data, the first speed data and the first orientation data of the dynamic traffic object corresponding to each of the time points; the coordinate inverse transformation of each of the second object annotation data is performed by using the relative posture information of the vehicle at different time points in the preset time interval to obtain the first object annotation data of the dynamic traffic object at different time points in the preset time interval, including: Using the relative position information of the vehicle at each time point, obtaining the rotation matrix information corresponding to each time point in the preset time interval; each rotation matrix information represents the rotation relationship between each vehicle coordinate system and the global coordinate system; Using the relative position information of the vehicle at each time point, performing an inverse coordinate transformation on each of the second position data to obtain each of the first position data; By using the rotation matrix information corresponding to each of the time points, coordinate inverse transformation is performed on each of the second speed data and the second orientation data to obtain each of the first speed data and the first orientation data.

8. The method according to claim 1, characterized in that The method of using the relative posture information of each vehicle and the laser point cloud data to obtain the fused point cloud data corresponding to each time frame includes: Acquire a current time frame and a preset number of historical time frames corresponding to the current time frame; the current time frame is any one of the time frames, and the historical time frames are a preset number of time frames adjacent to the current time frame and located before the current time frame; According to the laser point cloud data corresponding to the current time frame, the vehicle relative posture information corresponding to the current time frame, the laser point cloud data corresponding to each of the historical time frames, and the vehicle relative posture information corresponding to each of the historical time frames, the fused point cloud data corresponding to the current time frame is obtained.

9. The method according to claim 8, characterized in that The obtaining, according to each of the fused point cloud data and each of the on-board camera image data, first object detection information of the dynamic traffic object corresponding to each of the time frames respectively comprises: Inputting the fused point cloud data corresponding to the current time frame and the on-board camera image data corresponding to the current time frame into a pre-built target detection model, and obtaining the fused point cloud features and the on-board camera image features corresponding to the current time frame through the target detection model; Converting the fused point cloud features and the onboard camera image features into BEV vision, and fusing the fused point cloud features and the onboard camera image features after the BEV vision conversion to obtain a fused BEV feature; According to the fused BEV features, the dynamic traffic object contained in the current time frame and the first object detection information of the dynamic traffic object are obtained.

10. The method according to claim 9, characterized in that The obtaining of the dynamic traffic objects contained in the current time frame according to the fused BEV features includes: According to the fused BEV features, obtaining the candidate dynamic traffic objects included in the current time frame and the confidence corresponding to each of the candidate dynamic traffic objects; The candidate dynamic traffic objects whose confidence is greater than a preset confidence threshold are taken as the dynamic traffic objects included in the current time frame.

11. A dynamic traffic object labeling device, characterized in that: The device comprises: A detection data acquisition module, used to acquire vehicle-mounted camera image data, laser point cloud data and IMU data of each time frame in a preset time interval; each of the laser point cloud data includes static point cloud data and dynamic point cloud data; A relative posture acquisition module, used for obtaining the relative posture information of the vehicle corresponding to each of the time frames according to the IMU data and each of the static point cloud data; A detection information acquisition module, used to obtain fused point cloud data corresponding to each of the time frames using the relative posture information of each of the vehicles and each of the laser point cloud data, and to obtain first object detection information of a dynamic traffic object corresponding to each of the time frames according to each of the fused point cloud data and each of the vehicle-mounted camera image data; the dynamic traffic object is a traffic object represented by the dynamic point cloud data; The annotation data acquisition module is used to use the first object detection information to obtain the object motion trajectory of the dynamic traffic object in the preset time interval, and based on the object motion trajectory, obtain the first object annotation data of the dynamic traffic object at different time points in the preset time interval.

12. An automobile controller, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images