Object labeling method, device, electronic device and storage medium

Abstract

The present disclosure provides an object labeling method, including: acquiring data collected by multiple sensors; performing an initial spatiotemporal alignment on image data and laser point cloud data to obtain laser point cloud data and image data after the initial spatiotemporal alignment; grouping the laser point cloud data and image data after the initial spatiotemporal alignment by a preset time length to obtain the RTK variance corresponding to each segmented data; based on the RTK variance corresponding to each segmented data, determining whether to use the vehicle body IMU / wheel speed data to perform posture optimization on the vehicle body RTK posture sequence data to perform secondary spatiotemporal alignment on the image data and the laser point cloud data; performing 3D object detection and 2D object detection on the laser point cloud data and the image data, respectively, to perform object labeling. The present disclosure also provides a method based on multi-frame data accumulation for automatic labeling. The present disclosure also provides an object labeling device, an electronic device, and a storage medium.

Description

Technical Field

[0001] The present disclosure relates to the technical field of object annotation, and in particular to an object annotation method, device, electronic device and storage medium. Background Art

[0002] As one of the core technologies of unmanned driving, the accuracy of high-precision maps plays a vital role in the positioning, navigation, control and safety of unmanned vehicles. Since high-precision maps are a combination of three-dimensional point clouds and related semantic information, information loss is inevitable. In addition to the influence of natural environment, external occlusion, noise, etc., changes in road conditions themselves also add a lot of difficulties to improving the accuracy of high-precision maps. Therefore, how to design an automatic annotation method with wide applicability and capable of meeting complex and changeable road conditions based on existing high-precision maps has important theoretical significance and practical application value.

[0003] The core idea of ​​automatic annotation of high-precision maps is to assign RGB semantic information to point clouds, and improve annotation efficiency by using the boundaries and colors of the objects to be generated. This type of method performs semantic segmentation on the acquired image data, fuses the point cloud data with the segmented image data, and uses algorithms such as neural networks to draw maps.

[0004] Technical Solution 1: Patent No. CN113674287A, "High-precision Map Drawing Method, Device, Equipment and Storage Medium", involves computer vision, autonomous driving, intelligent transportation and other technical fields. The specific implementation scheme is: 1) semantic segmentation processing is performed on the acquired image data to obtain image data that identifies the object to be labeled; 2) the acquired point cloud data is fused with the image data that identifies the object to be labeled to obtain semantic map data; 3) using the pre-trained neural network model, according to the semantic map and point cloud data, a map is obtained that draws the position of the object to be labeled, the color of the object to be labeled and the boundary of the object to be labeled. This method integrates the features of image data and point cloud data for automatic map drawing and automatic labeling, which can not only generate the geometric position of the object to be labeled, but also the color and boundary of the object to be labeled. This method realizes the automation of map drawing and labeling through the integration of computer vision, autonomous driving, intelligent transportation and other technical fields, and can maintain the labeling accuracy, but this method does not remove motion distortion and time alignment of the collected data during the data fusion process, which will lead to the problem of difficulty in improving the labeling accuracy.

[0005] Technical Solution 2: Patent No. CN113255578A, "Method and Device for Identifying Traffic Signs, Electronic Equipment and Storage Medium", relates to a method and device for identifying traffic signs, electronic equipment and storage medium. The method includes: 1) acquiring images and point cloud data collected during vehicle driving; 2) detecting the image to obtain an initial detection value sequence of each traffic sign in the image, and selecting a target detection value from the initial detection value sequence of each traffic sign to obtain a target detection value sequence of the traffic sign; 3) grouping the target detection values ​​in the target detection value sequence, and selecting the point cloud data corresponding to each group from the point cloud data; 4) determining the point cloud data corresponding to each traffic sign in each group from the point cloud data corresponding to each group, and determining the outline of each traffic sign in the reference coordinate system based on the point cloud data corresponding to each traffic sign; 5) marking each traffic sign in the map based on the outline of each traffic sign in the reference coordinate system. This method achieves the technical effect of improving the accuracy and efficiency of marking traffic signs in high-precision maps by combining software and hardware. However, the method relaxes the timestamp, which will cause certain motion distortion of the laser point cloud, resulting in problems such as low overlap. In addition, when the overlap exceeds the third preset threshold image, the method chooses to delete the traffic sign with lower confidence in the detection value of the two traffic signs, and directly uses the other one as the correct traffic sign. Although there is manual re-inspection after automatic marking, there are still certain safety hazards. At the same time, this method mainly focuses on static objects that need to be identified by unmanned vehicles during driving, and is not very suitable for dynamic object recognition.

[0006] Technical Solution 3: Patent No. CN111325136A, "Method and Device for Marking Objects in Intelligent Vehicles, Unmanned Vehicles", relates to a method and device for marking objects in intelligent vehicles, and unmanned vehicles. The method includes: obtaining the initial image information of the area around the current vehicle in the target map, wherein the initial image information includes: the position and occupied area of ​​the object; identifying the composition structure of each object in the initial image information to obtain the object structure information; based on the object structure information and the preset annotation library, marking all objects on the target map, wherein the preset annotation library records the first mapping relationship between each object and the object identifier, and records the second mapping information between each object structure and the structure identifier, and the target map displays the point cloud data of each object and each object structure. This method solves the technical problem that the image displayed on the vehicle display screen in the related art cannot mark the surrounding obstacle objects, causing the vehicle to easily collide with other objects and affect the vehicle's driving, but this method mainly focuses on static objects that need to be identified by unmanned vehicles during driving, and is not very applicable to dynamic object recognition.

[0007] Technical solutions 1 and 2 integrate computer vision and neural networks, improving the scope and accuracy of automatic labeling while ensuring a wide range of adaptability. However, Technical Solution 1 does not perform time alignment on the collected data during the data fusion process, which makes it difficult to improve the labeling accuracy. Technical Solution 2 solves the problem of low overlap rate of multi-image data by directly deleting one of the data with low confidence, and the processing result is not ideal. In addition, its applicability for dynamic object recognition is not high. Technical Solution 3 uses a combination of software and hardware, with the help of algorithms such as deep learning, to accurately and specifically label some road conditions, but the areas of action are relatively specific and cannot be applied to all road conditions. Summary of the invention

[0008] In order to solve at least one of the above technical problems, the present disclosure provides an object labeling method, device, electronic device and storage medium.

[0009] According to one aspect of the present disclosure, there is provided an object labeling method, comprising:

[0010] S110, acquiring data collected by multiple sensors, the data including image data, laser point cloud data, vehicle RTK posture sequence data, and vehicle IMU / wheel speed data;

[0011] S120, based on the timestamp of the image data, the timestamp of the laser point cloud data and the vehicle body RTK posture sequence data, perform initial spatiotemporal alignment on the image data and the laser point cloud data to obtain the laser point cloud data and image data after the initial spatiotemporal alignment;

[0012] S130, grouping the laser point cloud data and image data after the initial spatiotemporal alignment according to a preset time length, and obtaining the RTK variance corresponding to each segmented data;

[0013] S140, based on the RTK variance corresponding to each segmented data, determine whether to use the vehicle IMU / wheel speed data to optimize the vehicle RTK posture sequence data to perform secondary spatiotemporal alignment on the image data and the laser point cloud data, and obtain the laser point cloud data and image data after secondary spatiotemporal alignment;

[0014] S160 , performing 3D object detection and 2D object detection on the accumulated laser point cloud data and the accumulated image data respectively to perform object labeling.

[0015] According to the object labeling method of at least one embodiment of the present disclosure, after step S140, it also includes: S115, accumulating laser point cloud data and image data; in step S160, performing the 3D object detection and the 2D object detection on the accumulated laser point cloud data and the accumulated image data respectively.

[0016] According to the object labeling method of at least one embodiment of the present disclosure, the 3D objects and the 2D objects include vehicles, pedestrians, lane lines, traffic signs, traffic lights, etc.

[0017] According to at least one embodiment of the object labeling method of the present disclosure, S110, acquiring data collected by multiple sensors, includes:

[0018] The image data is transformed from the camera coordinate system to the vehicle body coordinate system, and the laser point cloud data is transformed from the lidar coordinate system to the vehicle body coordinate system.

[0019] According to the object labeling method of at least one embodiment of the present disclosure, various types of data collected by multiple sensors are recorded on a unified time axis.

[0020] According to at least one embodiment of the object labeling method of the present disclosure, S120, based on the timestamp of the image data, the timestamp of the laser point cloud data and the vehicle body RTK posture sequence data, the image data and the laser point cloud data are initially space-time aligned to obtain the laser point cloud data and the image data after the initial space-time alignment, including:

[0021] S1202, for the laser point cloud data at a certain moment, find the image data at the moment closest to the moment, and obtain the time difference between the two moments;

[0022] S1204, extracting the RTK pose sequence in the world coordinate system including the above two moments from the vehicle body RTK pose sequence data, to obtain the poses corresponding to the above two moments respectively;

[0023] S1206, obtaining a posture difference in a world coordinate system corresponding to the time difference;

[0024] S1208. Correct the laser point cloud data in the vehicle body coordinate system based on the posture difference to obtain laser point cloud data that is spatially and temporally aligned with the image data in the vehicle body coordinate system.

[0025] According to at least one embodiment of the object labeling method of the present disclosure, S120, based on the timestamp of the image data, the timestamp of the laser point cloud data and the vehicle body RTK posture sequence data, the image data and the laser point cloud data are initially space-time aligned to obtain the laser point cloud data and the image data after the initial space-time alignment, including:

[0026] S1202, when the laser point cloud data and the image data are accumulated (accumulated for a preset time period), for the laser point cloud data at a certain moment, the time difference between the moment and the initial radar image acquisition moment within the accumulation time period is obtained;

[0027] S1204, extracting the RTK pose sequence in the world coordinate system including the above two moments from the vehicle body RTK pose sequence data, to obtain the poses corresponding to the above two moments respectively;

[0028] S1206, obtaining a posture difference in a world coordinate system corresponding to the time difference;

[0029] S1208. Correct the laser point cloud data in the vehicle body coordinate system based on the posture difference to obtain laser point cloud data that is spatially and temporally aligned with the image data in the vehicle body coordinate system.

[0030] According to at least one embodiment of the object labeling method of the present disclosure, S130, segmenting the laser point cloud data and image data after the initial spatiotemporal alignment by a preset time length, and obtaining the RTK variance corresponding to each segmented data, including:

[0031] When obtaining the RTK variance, statistics are collected on the RTK variances of all degrees of freedom.

[0032] According to the object labeling method of at least one embodiment of the present disclosure, S140, based on the RTK variance corresponding to each segmented data, determines whether to use the vehicle body IMU / wheel speed data to perform posture optimization on the vehicle body RTK posture sequence data to perform secondary spatiotemporal alignment on the image data and the laser point cloud data, and obtains the laser point cloud data and image data after the secondary spatiotemporal alignment, including:

[0033] When the RTK variance of each degree of freedom is less than or equal to the variance threshold, it is determined that the vehicle IMU / wheel speed data is not used to optimize the vehicle RTK pose sequence data; otherwise, it is determined that the vehicle IMU / wheel speed data is used to optimize the vehicle RTK pose sequence data.

[0034] According to the object labeling method of at least one embodiment of the present disclosure, in step S140, the posture optimization of the vehicle RTK posture sequence data is performed using the vehicle IMU / wheel speed data, including:

[0035] For a segment of data whose RTK variance exceeds the variance threshold, obtain the first IMU data closest to and earlier than the start time of the segment of data, and use the corresponding time as the starting point; obtain the second IMU data closest to and later than the end time of the segment of data, and use the corresponding time as the end point;

[0036] Between the starting point and the end point, for each image data acquisition time t1 and each laser point cloud data acquisition time t2, the following processing is performed:

[0037] The translation amount Δx of the vehicle body from the starting point to the end point is obtained based on the wheel speed data; the rotation angle differences Δyaw, Δpitch and Δroll of the vehicle body in the x, y and z directions from the starting point to the end point are obtained based on the IMU data;

[0038] By integrating the rotation angle differences Δyaw, Δpitch and Δroll with the translation Δx, the translation Δy and translation Δz of the vehicle body along the y-axis and z-axis directions during the time period from t1 to t2 are obtained, that is, the position change of the vehicle body during the time period from t1 to t2.

[0039] According to at least one embodiment of the object labeling method of the present disclosure, in step S140, the posture optimization of the vehicle RTK posture sequence data is performed using the vehicle IMU / wheel speed data, and further includes:

[0040] The pose difference corresponding to the data whose RTK variance exceeds the variance threshold is replaced by the pose change.

[0041] According to at least one embodiment of the object annotation method of the present disclosure, S160, performing 3D object detection and 2D object detection on the laser point cloud data and the image data respectively to perform object annotation, including:

[0042] Detect 3D objects in laser point cloud data using a BEV-based target detection method, a range view-based target detection method, a point-wise feature-based target detection method, or a fusion feature-based target detection method;

[0043] Detect 2D objects in image data using Cascade / Haar / SVM-based object detection algorithms, R-CNN / Fast R-CNN-based candidate region / box + deep learning classification algorithms, or RRC detection / Deformable CNN-based deep learning detection methods;

[0044] Based on the weights of the feature maps of image data and laser point cloud data, Mixture of Experts is used to build a model, and feature fusion is performed through splicing operations to complete the automatic detection of 3D objects and 2D objects.

[0045] According to at least one embodiment of the object annotation method of the present disclosure, S160, performing 3D object detection and 2D object detection on the laser point cloud data and the image data respectively to perform object annotation, including:

[0046] The 3D object is projected onto the corresponding 2D object, and the correctness of the detected object is determined based on the overlap between the 3D and 2D projections of the same object. If correct, the object is output; if incorrect, the object is not output, so as to perform object labeling.

[0047] According to the object labeling method of at least one embodiment of the present disclosure, in step S160, if it is incorrect, reprojection is performed based on time domain multi-frame optimization, and the object is labeled again.

[0048] According to at least one embodiment of the object labeling method of the present disclosure, a 3D object is projected onto a corresponding 2D object, and whether the detected object is correct is determined based on the overlap between the 3D and 2D projections of the same object. If correct, the object is output; if incorrect, the object is not output, including:

[0049] If the projection coincidence of an object is greater than or equal to the coincidence threshold, the detected object is judged to be correct;

[0050] If the projection coincidence of an object is lower than the coincidence threshold, further detection is performed through pose optimization.

[0051] According to at least one embodiment of the object labeling method of the present disclosure, step S160 further includes:

[0052] Based on the vehicle RTK posture sequence data, each vector object in the high-precision map is projected into the laser point cloud data and image data to update the laser point cloud data and image data for object annotation.

[0053] Among them, the vector objects described in the high-precision map are static objects, including fixed traffic signs.

[0054] According to at least one embodiment of the object labeling method of the present disclosure, based on the vehicle RTK posture sequence data, each vector object in the high-precision map is projected into the laser point cloud data and the image data to update the laser point cloud data and the image data to perform object labeling, including:

[0055] S1602, based on the vehicle body RTK posture sequence data, obtaining the coordinates of each vector object in the high-precision map in the world coordinate system within the observation range of the laser radar and the coordinates of the vehicle body in the world coordinate system, wherein the observation range takes the center of the laser radar as the origin;

[0056] S1604, transforming the coordinates of each vector object in the world coordinate system into the vehicle body coordinate system;

[0057] S1606, projecting each vector object in the vehicle body coordinate system into the laser point cloud data;

[0058] S1608, transforming the coordinates of each vector object in the vehicle body coordinate system into the camera coordinate system, and forming a projection in the image data;

[0059] S1610. When the variance of the RTK data is less than or equal to the variance threshold, each vector object in the high-precision map is projected into the laser point cloud data and image data. When the variance of the RTK data is greater than the variance threshold, the point cloud data is optimized based on the image data for posture reprojection to complete the detection of 3D objects and 2D objects.

[0060] According to the object labeling method of at least one embodiment of the present disclosure, in step S1610, when the variance of the RTK data is greater than the variance threshold, reprojection is performed based on time domain multi-frame optimization, and 3D objects and 2D objects are detected again.

[0061] According to at least one embodiment of the object labeling method of the present disclosure, step S160 further includes:

[0062] The laser point cloud data within a preset time period is superimposed to mark static objects and dynamic objects separately.

[0063] According to at least one embodiment of the object labeling method of the present disclosure, laser point cloud data within a preset time period is superimposed to label static objects and dynamic objects respectively, including:

[0064] Take the laser images collected by the laser radar within a preset time period, accumulate and overlay the point clouds, and obtain overlay data;

[0065] For static objects in the overlay data, one-time object detection and labeling are performed;

[0066] For dynamic objects in the superimposed data, the motion path of the dynamic object is extracted to determine whether the motion path conforms to the preset object motion model. If so, it is directly labeled based on the preset object motion model. If not, object detection and labeling are performed based on the deep learning algorithm.

[0067] According to at least one embodiment of the object labeling method of the present disclosure, for a static object in the superimposed data, a one-time object detection and labeling is performed, including:

[0068] Statistics are performed on all frame images within the preset time period to obtain the mean variance, and the change in the object position after statistics is detected to perform reprojection and object labeling; wherein the statistics include removing extreme frame images from all frame images within the preset time period (i.e., removing them as extreme values).

[0069] According to another aspect of the present disclosure, there is provided an object labeling device, comprising:

[0070] A sensor data acquisition module, wherein the sensor data acquisition module acquires data collected by multiple sensors, wherein the data includes image data, laser point cloud data, vehicle RTK posture sequence data, and vehicle IMU / wheel speed data;

[0071] A primary alignment module, wherein the primary alignment module performs a primary spatiotemporal alignment on the image data and the laser point cloud data based on the timestamp of the image data, the timestamp of the laser point cloud data, and the vehicle body RTK posture sequence data, and obtains the laser point cloud data and the image data after the primary spatiotemporal alignment;

[0072] A data segmentation module, wherein the data segmentation module groups the laser point cloud data and the image data after the initial spatiotemporal alignment according to a preset time length;

[0073] An RTK variance calculation / judgment module, wherein the RTK variance calculation / judgment module obtains the RTK variance corresponding to each segmented data; based on the RTK variance corresponding to each segmented data, it is determined whether to use the vehicle body IMU / wheel speed data to perform posture optimization on the vehicle body RTK posture sequence data to perform secondary spatiotemporal alignment on the image data and the laser point cloud data;

[0074] A secondary alignment module, wherein the secondary alignment module uses the vehicle body IMU / wheel speed data to perform posture optimization on the vehicle body RTK posture sequence data to perform secondary spatiotemporal alignment on the image data and the laser point cloud data, and obtains the laser point cloud data and image data after the secondary spatiotemporal alignment;

[0075] The object annotation module performs 3D object detection and 2D object detection on the laser point cloud data and the image data respectively to perform object annotation.

[0076] According to at least one embodiment of the present disclosure, the object labeling device further includes:

[0077] A data accumulation module, which accumulates laser point cloud data and image data;

[0078] The object annotation module performs the 3D object detection and the 2D object detection on the accumulated laser point cloud data and the accumulated image data.

[0079] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the object labeling method of any embodiment of the present disclosure.

[0080] According to another aspect of the present disclosure, a readable storage medium is provided, wherein the readable storage medium stores execution instructions, and when the execution instructions are executed by a processor, the object labeling method of any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0082] Figure 1 It is a flowchart of an object labeling method according to an embodiment of the present invention.

[0083] Figure 2 A schematic diagram of triggering a camera and a lidar according to an embodiment of the present disclosure is shown.

[0084] Figure 3 It is a schematic diagram of a process for performing initial spatiotemporal alignment of image data and laser point cloud data according to an embodiment of the present disclosure.

[0085] Figure 4 It is a flowchart of performing posture adjustment based on RTK variance according to an embodiment of the present disclosure.

[0086] Figure 5 It is a flowchart of performing 3D object detection and 2D object detection on accumulated laser point cloud data and accumulated image data respectively according to an embodiment of the present disclosure.

[0087] Figure 6 This is a flowchart of an embodiment of the present invention, which integrates the vector objects of the high-precision map with the 2D detection object results, verifies the accuracy of the annotation with the help of RTK data, and performs optimization and adjustment.

[0088] Figure 7 It is a schematic block diagram of the structure of an object labeling device using a hardware implementation of a processing system according to an embodiment of the present disclosure.

[0089] Figure 8 It is a schematic block diagram of the structure of an object labeling device using a hardware implementation method of a processing system according to another embodiment of the present disclosure.

[0090] Description of Reference Numerals

[0091] 1000 Object Annotation Device

[0092] 1002 Sensor data acquisition module

[0093] 1004 Initial Alignment Module

[0094] 1006 Data Segmentation Module

[0095] 1008 RTK variance calculation / judgment module

[0096] 1010 Secondary Alignment Module

[0097] 1012 Data Accumulation Module

[0098] 1014 Object Annotation Module

[0099] 1100 Bus

[0100] 1200 processor

[0101] 1300 Memory

[0102] 1400 Other circuits. DETAILED DESCRIPTION

[0103] The present disclosure is further described in detail below in conjunction with the accompanying drawings and implementations. It is understood that the specific implementations described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0104] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0105] Unless otherwise specified, the exemplary embodiments / embodiments shown will be understood as providing exemplary features of various details of some ways in which the technical concept of the present disclosure can be implemented in practice. Therefore, unless otherwise specified, the features of the various embodiments / embodiments can be combined, separated, interchanged and / or rearranged without departing from the technical concept of the present disclosure.

[0106] The use of cross-hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise specified, the presence or absence of cross-hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the described order. In addition, the same figure numbers represent the same components.

[0107] When a component is referred to as being "on" or "over," "connected to," or "coupled to" another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being "directly on," "directly connected to," or "directly coupled to" another component, there are no intervening components. For this purpose, the term "connected" may refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0108] The terms used herein are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used herein, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0109] Combined with the following Figures 1 to 7 The object labeling method, object labeling device, electronic device and readable storage medium disclosed in the present invention are described in detail.

[0110] Figure 1 It is a flowchart of an object labeling method according to an embodiment of the present invention.

[0111] refer to Figure 1 The object labeling method S100 of this embodiment includes:

[0112] S110, acquiring data collected by multiple sensors, the data including image data, laser point cloud data, vehicle RTK posture sequence data, and vehicle IMU / wheel speed data;

[0113] S120, based on the timestamp of the image data, the timestamp of the laser point cloud data and the vehicle body RTK posture sequence data, perform initial spatiotemporal alignment on the image data and the laser point cloud data to obtain the laser point cloud data and image data after the initial spatiotemporal alignment;

[0114] S130, segmenting the laser point cloud data and image data after the initial spatiotemporal alignment according to a preset time length (e.g., 20 seconds) to obtain the RTK variance corresponding to each segmented data;

[0115] S140, based on the RTK variance corresponding to each segmented data, determine whether to use the vehicle IMU / wheel speed data to optimize the vehicle RTK posture sequence data to perform secondary spatiotemporal alignment on the image data and the laser point cloud data, and obtain the laser point cloud data and image data after secondary spatiotemporal alignment;

[0116] S160, perform 3D object detection and 2D object detection on the laser point cloud data and the image data respectively to perform object labeling (motor vehicles, non-motor vehicles, pedestrians, lane lines, traffic lights, traffic signs, etc.).

[0117] The object labeling method S100 of this embodiment can realize real-time object labeling, that is, single-frame labeling.

[0118] In some embodiments of the present disclosure, the object labeling method S100 of the present disclosure further includes:

[0119] S115, accumulating laser point cloud data and image data;

[0120] In step S160, the 3D object detection and the 2D object detection are performed on the accumulated laser point cloud data and the accumulated image data respectively.

[0121] In some embodiments of the present disclosure, the sensors used in the present disclosure include cameras, lidars, inertial measurement units (IMUs) and RTKs for vehicle positioning, which are respectively used for the image data, laser point cloud data, vehicle IMU / wheel speed data, and vehicle RTK pose sequence data described above.

[0122] Among them, the cameras can be divided into six cameras according to their installation positions on the vehicle, namely, front view camera, rear view camera, left front view camera, left rear view camera, right front view camera, and right rear view camera. Those skilled in the art can adjust the number and installation positions of the cameras, which all fall within the protection scope of the present disclosure.

[0123] In some embodiments of the present disclosure, relevant internal parameters and relevant external parameters of each sensor are also obtained.

[0124] Among them, the camera's external parameters use matrix T CW It can be decomposed into the rotation matrix R CW and the translation matrix t CW , which determine the relative position relationship between the camera coordinate system and the world coordinate system.

[0125] Camera coordinates P c With world coordinates P w The relationship between them is:

[0126] P c =TCW P w , that is, P c =R CW P w +t CW .

[0127] The relevant internal parameters of the camera can be expressed as a matrix K, which includes {f,κ,S x ,S y ,C x ,C y} six parameters, among which f is the focal length; κ represents the magnitude of radial distortion. If κ is a negative value, the distortion becomes barrel distortion; if κ is a positive value, the distortion becomes pincushion distortion; S x ,S y is the scaling factor, which represents the distance between adjacent pixels in the horizontal direction and the distance between adjacent pixels in the vertical direction on the sensor; C x ,C y It is the vertical projection of the projection center on the imaging plane and is also the center of radial distortion.

[0128] Among them, the laser radar disclosed in the present invention can be a solid-state laser radar, a total of four. Technical personnel in this field can select / adjust the type and quantity of the laser radar, all of which fall within the protection scope of the present disclosure.

[0129] The relevant external parameters of the laser radar are the rotation and translation transformation matrix T LW ; Its related configuration parameters may include: field of view angle, resolution, ranging range, refresh frequency, scanning frequency, laser wavelength, and maximum radiation power.

[0130] An inertial measurement unit (IMU) is used to provide relative positioning information. Its function is to measure the path of an object (such as a vehicle) relative to a starting point. An inertial measurement unit (IMU) may include a three-axis accelerometer and a three-axis gyroscope as sensitive elements, which output acceleration and angular velocity respectively.

[0131] RTK, or carrier phase differential technology, is a commonly used satellite positioning measurement technology in the prior art for vehicle positioning.

[0132] In the present disclosure, since the data collection process involves multiple sensors working together, but the collection frequency, own coordinate system, etc. of the sensors are not consistent, the present disclosure performs time-space alignment on the data before using the data.

[0133] In some embodiments of the present disclosure, the vehicle body coordinate system takes the center of the rear axle of the vehicle as the origin, the vehicle's forward direction is due forward, the x-axis points due forward, the y-axis points due left, and the z-axis points due upward; the camera coordinate system takes the optical center of the camera as the origin, the z-axis points to the front of the camera, the x-axis points to the right of the camera, and the y-axis points to the bottom of the camera; the lidar coordinate system takes the geometric center of the lidar as the origin, the z-axis points to the front of the lidar, the x-axis points to the right of the lidar, and the y-axis points to the bottom of the lidar; the coordinate system of the IMU is consistent with the vehicle body coordinate system, with the center of the rear axle of the vehicle as the origin, the vehicle's forward direction is due forward, the x-axis points to the front, the y-axis points to the left, and the z-axis points to the top.

[0134] Since data collection involves multiple sensors and the collected data all contain time stamps, the present disclosure performs time synchronization processing on the multi-sensor data.

[0135] Different sensors have different sensor timestamps and sensor frequencies are also different.

[0136] In some embodiments of the present disclosure, the six cameras used can be triggered simultaneously, and the camera transmits frames per second at 30fps, but the four solid-state laser radars cannot be triggered synchronously, and the transmission frame rate per second is 10fps.

[0137] Figure 2 A schematic diagram of triggering a camera and a lidar according to an embodiment of the present disclosure is shown.

[0138] Figure 2 In the figure, the gray bars represent the acquisition frequencies of the camera and lidar, and the horizontal axis represents the unified timestamp. Each sampling moment of the sensor is recorded in a unified time series. Since the acquisition rate of the lidar is slow, after the multi-sensor data is uniformly converted to the vehicle coordinate system, the sampling of the lidar is oriented towards the sampling of the camera, and the time-space alignment between the two sensor data is performed by finding the image data collected by the camera at the nearest time to the lidar acquisition time.

[0139] In some embodiments of the present disclosure, in step S110, the image data is transformed from the camera coordinate system to the vehicle body coordinate system, and the laser point cloud data is transformed from the laser radar coordinate system to the vehicle body coordinate system.

[0140] In some embodiments of the present disclosure, various types of data collected by multiple sensors (lidar, camera, IMU, RTK, etc.) are recorded on a unified timeline.

[0141] Figure 3 It is a schematic diagram of a process for performing initial spatiotemporal alignment of image data and laser point cloud data according to an embodiment of the present disclosure.

[0142] refer to Figure 3Preferably, S120, based on the timestamp of the image data, the timestamp of the laser point cloud data and the vehicle body RTK posture sequence data, the image data and the laser point cloud data are initially space-time aligned to obtain the laser point cloud data and the image data after the initial space-time alignment, including:

[0143] S1202. For a certain time t n,j The laser point cloud data is used to find the time t closest to this time. n,i Image data, get the time difference Δt between two moments n ;

[0144] S1204: extract the above two moments t from the vehicle RTK posture sequence data. n,j and t n,i RTK pose sequence in the world coordinate system (j and i represent different moments) To obtain the postures corresponding to the above two moments and (where subscript C represents camera, subscript L represents lidar, and superscript W represents world coordinate system), i.e., the vehicle posture when the camera collects image data and the vehicle posture when the lidar collects laser point cloud data;

[0145] S1206: Obtaining the posture difference in the world coordinate system corresponding to the time difference

[0146] S1208, based on posture difference The laser point cloud data in the vehicle body coordinate system is corrected to obtain laser point cloud data that is spatially and temporally aligned with the image data in the vehicle body coordinate system.

[0147] In this embodiment, preferably, the timestamp of the front-view camera in the camera is used as the reference. Where n represents the identification / number of the laser radar, n=1,2,3,4.

[0148] In this implementation, due to the acquisition frequency issue, the RTK acquisition time is not necessarily the same as the sampling time of the camera or lidar. In this implementation, it is assumed that the acquisition times do not overlap (if they overlap, the corresponding RTK pose data can be directly taken as the pose data at the camera or lidar sampling time).

[0149] Take the value containing t n,j With t n,i (Assume t n,j Later than t n,i ) based on the world coordinate system RTK pose sequence t n,i Should be located at With T1 W The corresponding time interval, t n,jShould be located at and Preferably, the camera at t is obtained by linear interpolation. n，i The position corresponding to the moment and LiDAR at t n,j The position corresponding to the moment

[0150] Preferably, based on the camera at t n,i The position corresponding to the moment and LiDAR at t n,j The position corresponding to the moment By formula Find the n The pose difference between the camera and the lidar sampling time based on the world coordinate system in time

[0151] Preferably, the laser point cloud data based on the laser radar coordinate system collected by the laser radar is Transform the space to the vehicle coordinate system. The superscript B represents the vehicle coordinate system. T BL is the transformation matrix.

[0152] More preferably, by t n,j The laser point cloud data at time t is aligned to n,i At this moment, the time alignment of the lidar to the camera is completed.

[0153] Preferably, the present disclosure also adjusts the relative pose sequence based on the variance condition of RTK and point cloud matching.

[0154] In some embodiments of the present disclosure, S120, based on the timestamp of the image data, the timestamp of the laser point cloud data and the vehicle body RTK posture sequence data, the image data and the laser point cloud data are initially space-time aligned to obtain the laser point cloud data and the image data after the initial space-time alignment, including:

[0155] S1202, when the laser point cloud data and the image data are accumulated (accumulated for a preset time period), for the laser point cloud data at a certain moment, the time difference between the moment and the initial radar image acquisition moment within the accumulation time period is obtained;

[0156] S1204, extracting the RTK pose sequence in the world coordinate system including the above two moments from the vehicle body RTK pose sequence data, to obtain the poses corresponding to the above two moments respectively;

[0157] S1206, obtaining a posture difference in a world coordinate system corresponding to the time difference;

[0158] S1208. Correct the laser point cloud data in the vehicle body coordinate system based on the posture difference to obtain laser point cloud data that is spatially and temporally aligned with the image data in the vehicle body coordinate system.

[0159] Figure 4 It is a flowchart of performing posture adjustment based on RTK variance according to an embodiment of the present disclosure.

[0160] refer to Figure 4 In S130, the laser point cloud data and image data after the initial spatiotemporal alignment are segmented with a preset time length (e.g., 20 seconds) to obtain the RTK variance corresponding to each segmented data, including:

[0161] When obtaining the RTK variance, the RTK variances of all degrees of freedom (ie, all data segments) are statistically analyzed separately.

[0162] According to a preferred embodiment of the present disclosure, in S140, based on the RTK variance corresponding to each segmented data, it is determined whether to use the vehicle IMU / wheel speed data to perform posture optimization on the vehicle RTK posture sequence data to perform secondary spatiotemporal alignment on the image data and the laser point cloud data, and obtain the laser point cloud data and image data after secondary spatiotemporal alignment, including:

[0163] When the RTK variance of each degree of freedom is less than or equal to the variance threshold, it is determined that the vehicle IMU / wheel speed data is not used to optimize the vehicle RTK pose sequence data; otherwise, it is determined that the vehicle IMU / wheel speed data is used to optimize the vehicle RTK pose sequence data.

[0164] refer to Figure 4 In some embodiments of the present disclosure, when the variance of RTK is less than or equal to the variance threshold, 3D laser point cloud accumulation is performed directly; when the variance of RTK is greater than the variance threshold, the posture is transformed with the help of IMU data and wheel speed meter data, and then 3D laser point cloud accumulation is performed.

[0165] In normal weather conditions and when satellite signals are not interfered with, the time-space alignment + RTK posture transformation described above in this disclosure can be used for accurate data processing, but there are still some situations, such as bad weather, vehicles passing through tall buildings or viaducts, etc., which may cause RTK to have a large variance in a certain time period, resulting in the actual position of the accumulated points of the laser point cloud being inaccurate. For this reason, this disclosure preferably uses the fusion of IMU data and wheel speed meter data to perform vehicle posture transformation instead of the posture transformation calculated by RTK described above, and combines the time-space alignment method described above to reprocess the laser point cloud data in the time period when the RTK variance is greater than the variance threshold.

[0166] For the object labeling method S100 of the present disclosure, preferably, in step S140, the vehicle body IMU / wheel speed data is used to perform posture optimization on the vehicle body RTK posture sequence data, including:

[0167] For a segment of data whose RTK variance exceeds the variance threshold, obtain the first IMU data closest to and earlier than the start time of the segment of data, and use the corresponding time as the starting point t start , obtain the second IMU data closest to and later than the end time of this segment of data, and use the corresponding time as the end point t end ;

[0168] Between the starting point and the end point, for each image data acquisition time t1 and each laser point cloud data acquisition time t2, the following processing is performed:

[0169] Based on the wheel speed data, the translation amount Δx of the vehicle body from the starting time to the end time is obtained; based on the IMU data, the rotation angle differences Δyaw, Δpitch and Δroll of the vehicle body in the x, y, and z directions (vehicle body coordinate system) from the starting time to the end time are obtained;

[0170] By integrating the rotation angle differences Δyaw, Δpitch and Δroll with the translation Δx, the translation Δy and the translation Δz of the vehicle body along the y-axis and z-axis during the time period from t1 to t2 are obtained, that is, the position change of the vehicle body during the time period from t1 to t2;

[0171] The pose change is used to replace the pose difference corresponding to the data whose RTK variance exceeds the variance threshold.

[0172] According to a preferred embodiment of the present disclosure, 3D object detection and 2D object detection are performed on laser point cloud data and image data based on a deep learning algorithm.

[0173] In some embodiments of the present disclosure, in step S160, 3D object detection and 2D object detection are performed on the laser point cloud data and the image data respectively to perform object annotation, including:

[0174] Detect 3D objects in laser point cloud data using a BEV-based target detection method, a range view-based target detection method, a point-wise feature-based target detection method, or a fusion feature-based target detection method;

[0175] Detect 2D objects in image data using Cascade / Haar / SVM-based object detection algorithms, R-CNN / Fast R-CNN-based candidate region / box + deep learning classification algorithms, or RRC detection / Deformable CNN-based deep learning detection methods;

[0176] Based on the weights of the feature maps of image data and laser point cloud data, Mixture of Experts is used to build a model, and feature fusion is performed through splicing operations to complete the automatic detection of 3D objects and 2D objects.

[0177] In some embodiments of the present disclosure, for static objects in the accumulation situation, point clouds are not merged, that is, object detection is not performed through point clouds. Multi-frame objects can be directly merged with the help of the outer frame lines of the detected objects. The object detection step after the point cloud merging can be omitted to reduce the amount of calculation, and the problem of unclear observation caused by a large amount of overlap of multiple point clouds can also be avoided.

[0178] Figure 5 It is a flowchart of performing 3D object detection and 2D object detection on laser point cloud data and image data respectively according to an embodiment of the present disclosure.

[0179] In step S160 of the object labeling method S100 of the present disclosure, 3D object detection and 2D object detection are performed on the laser point cloud data and the image data respectively to perform object labeling, including:

[0180] The 3D object is projected onto the corresponding 2D object, and the correctness of the detected object is determined based on the overlap between the 3D and 2D projections of the same object. If correct, the object is output; if incorrect, the object is not output, so as to perform object labeling.

[0181] If the projection coincidence of an object is greater than or equal to the coincidence threshold, the detected object is judged to be correct; if the projection coincidence of an object is lower than the coincidence threshold, further detection is performed through posture optimization.

[0182] If the projection overlap is low, it is considered that the object marking method / system of the present disclosure has certain errors. The errors may be caused by the unstable and inaccurate posture of the vehicle due to bumps during the movement of the vehicle, or may be caused by inaccurate detection of 3D objects or 2D objects.

[0183] In some embodiments of the present disclosure, preferably, an optimization method is used to find the vehicle posture with the smallest projection error between the observed 3D object and its corresponding 2D object, so as to optimize the projection relationship between the 3D object and the 2D object, and to find abnormal 3D object detection results and abnormal 2D object detection results. Preferably, the following method is used for optimization.

[0184] Since a 3D object may be observed by multiple frames of multiple cameras, that is, the projection error is the sum of multiple 3D objects and their multiple 2D projection errors, during the optimization process, the present invention performs local time-domain multi-frame optimization. Assume that there are m frames of data in a time period that needs to be reprojected, and m can be set to 5 frames (it can also be adjusted to other frame numbers).

[0185] The reprojection error (expressed using the Lie algebra ξ) is calculated using the geometric center points P of n 3D objects and the geometric center p of their 2D projections.

[0186] Assume that a space point P i =[X i ,Y i ,Z i ] T , the pixel coordinate of its projection is p i =[u i ,v i ] T , then the relationship between the pixel position and the spatial point position is as follows:

[0187] That is i p i =Kexp(ξ^)P i

[0188] Among them, K i represents the camera internal parameters, s i Represents the distance between pixels, and exp(ξ^) is used to transform the pose from the world coordinate system to the camera coordinate system.

[0189] However, due to the inaccurate camera pose and the noise of the observation points, there is an error in the above equation. Sum the multi-frame errors of all pixels of the observed object, construct a least squares problem, and then find a better vehicle pose to minimize it:

[0190]

[0191] Among them, since there are 6 cameras and 4 laser radars on the vehicle body, and a 3D object may be observed by multiple sensors in multiple frames, it is necessary to transform the position and posture of the laser radar object to the camera object for the sensor data observing a certain object. Here, E i Represents the external parameters from the vehicle coordinate system to the camera coordinate system to complete the transformation process.

[0192] Use Gauss-Newton method to solve P B , using the adjusted vehicle posture P B , combined with the internal and external reference relationship between the camera and the lidar, the relationship between 3D projection and 2D projection is established to determine whether the automatically labeled objects are correct or not. If there are individual inaccuracies, they will be labeled separately.

[0193] The present invention uses the above method to perform visual fusion of 3D and 2D, output all adjusted object detection results, and complete the automatic labeling of 3D category images such as motor vehicles, non-motor vehicles, pedestrians, traffic signs, traffic lights, etc.

[0194] According to the object labeling method S100 of the preferred embodiment of the present disclosure, step S160 further includes:

[0195] Based on the vehicle RTK posture sequence data, each vector object in the high-precision map is projected into the laser point cloud data and image data to update the laser point cloud data and image data for object annotation.

[0196] This embodiment is used together with the object annotation method described above to complete the automatic annotation process. In this embodiment, it mainly acts on objects such as ground traffic signs, as well as objects such as poles, traffic signs, and traffic lights on the ground. It is difficult to automatically detect these objects in 3D based on laser point clouds. Therefore, this disclosure only performs 2D object detection on image data. This embodiment merges the vector objects of the high-precision map with the 2D detection object results, verifies the accuracy of the annotation with the help of RTK data, and makes optimization adjustments. Figure 6 , preferably comprising the following steps:

[0197] S1602, based on the vehicle body RTK posture sequence data, obtaining the coordinates of each vector object in the high-precision map in the world coordinate system within the observation range of the laser radar and the coordinates of the vehicle body in the world coordinate system, wherein the observation range takes the center of the laser radar as the origin;

[0198] S1604, transforming the coordinates of each vector object in the world coordinate system into the vehicle body coordinate system;

[0199] S1606, projecting each vector object in the vehicle body coordinate system into the laser point cloud data;

[0200] S1608, transforming the coordinates of each vector object in the vehicle body coordinate system into the camera coordinate system, and forming a projection in the image data;

[0201] S1610. When the variance of the RTK data is less than or equal to the variance threshold, each vector object in the high-precision map is projected into the laser point cloud data and image data. When the variance of the RTK data is greater than the variance threshold, the point cloud data is optimized based on the image data for posture reprojection to complete the detection of 3D objects and 2D objects.

[0202] In this implementation, RTK is first used to obtain the coordinates of each vector object in the high-precision map based on the world coordinate system within the observation range of the laser radar and the coordinates of the vehicle body in the world coordinate system. The observation range is based on the center of the laser radar as the origin. The observation range can be 150m in front, and 50m on the left, right and rear.

[0203] In this implementation, the categories of these detected objects can be divided into categories with clear attributes (lights, signs, poles, lane lines, crosswalks, stop lines, arrows, manhole covers, etc.) and categories without clear attributes (building shapes, ground text markings, etc.) according to their contents.

[0204] Since each image contains a small number of objects, the corresponding object can be quickly found through the object category and the corresponding position of the bounding box, and its feature points can be used for detail matching.

[0205] Combined with the detection results, according to the relative position of the vector object and the vehicle body in the world coordinate system, the coordinates of each vector object based on the world coordinate system are converted to the vehicle body coordinate system:

[0206] The projection is formed in the laser radar, and then the coordinates of each vector object based on the vehicle coordinate system are converted to the camera coordinate system. And form a projection in the camera.

[0207] Calculate the variance of RTK data when projecting vector objects. When the RTK variance is less than or equal to the variance threshold, directly project each vector object in the high-precision map into the camera and lidar to complete the labeling of 3D objects and 2D objects.

[0208] When the RTK variance is greater than the variance threshold, the pose of each vector object data in the high-precision map is adjusted overall by reprojecting the minimum error, and then projecting. The adjustment and optimization steps are as follows:

[0209] Select each time period in which the RTK variance is greater than the variance threshold. Since there are multiple cameras and lidars, there are usually m (m>2) frames of images that can be detected and projected for the same object in the same time domain. Therefore, in the actual processing process, it is preferred to associate the mth frame image (m>2) with its previous frame (m-1, m-2, …, 2, 1) and match it according to the detection results associated with the object in the three-dimensional space and the current frame image.

[0210] Reproject the 2D and 3D objects after detection and matching. During the reprojection process, the detected objects in the 3D laser radar are projected into the 2D camera image, using the formula The pose given by RTK is used as the initial pose, and the Gauss-Newton method is used to solve it to obtain the optimized pose with the smallest reprojection error.

[0211] Furthermore, based on the optimized posture obtained from the reprojection result, the original RTK posture is updated by combining the vehicle's wheel speed odometer and IMU calculated in the time domain. 3D and 2D projections are directly performed based on the new posture to complete the labeling of vector objects.

[0212] In addition, when annotating vector objects in high-precision maps, if one of the camera projections is deviated or the overlap of an object projection in one projection is low, it is usually considered that there is a problem with the internal and external parameters of the camera or a problem with a particular object. In this case, manual adjustments can be made to the individual problem. After the adjustment is completed, 3D and 2D object projections can be performed.

[0213] According to a preferred embodiment of the present disclosure, step S160 of the object labeling method S100 of the present disclosure further includes:

[0214] The laser point cloud data within a preset time period is superimposed to mark static objects and dynamic objects separately.

[0215] This implementation improves accuracy by overlaying multiple frames, that is, taking a certain time as a reference, all point clouds within a preset time period are projected into the world coordinate system, so that there is sufficient data support during the automatic annotation process.

[0216] And preferably, the present disclosure performs differentiated processing on dynamic and static objects in the scene, that is, static objects are marked and processed once, and dynamic objects are supported to be marked and adjusted multiple times, wherein adjustments can be made in the 3D view or fine-tuned in the 2D image view.

[0217] Preferably, in order to enhance the applicability and accuracy of the object labeling method of the present disclosure, the present disclosure preferably improves the accuracy by accumulating multi-frame superimposed point clouds.

[0218] Take the laser point cloud data collected by the laser radar within a preset time period (for example, 20s, adjustable), that is, the "image". Take the origin of the vehicle coordinate system in the first frame of the image collected within the preset time period as the origin of the world coordinate system, and the time t corresponding to the first frame of the image as the initial time. The laser point cloud data of all other frames are aligned to time t and space to the world coordinate system through the time-space alignment steps described above, and the aligned laser point cloud data are accumulated.

[0219] For the accumulated static objects within the preset time period, including static objects that cannot move (road signs, etc.) and objects with motion attributes but stationary within the preset time period (roadside vehicles, etc.), a one-time object detection is performed:

[0220] Calculate RTK variance;

[0221] For data with RTK variance less than or equal to the variance threshold, it is judged that the vehicle's positioning accuracy is high and is directly marked; for data with RTK variance greater than the variance threshold, it is judged that it cannot be directly marked, and the relevant processing described in the spatiotemporal alignment method and the method of projecting various vector objects based on RTK data described above is required, and then the objects are reprojected and automatically marked.

[0222] In the case of point cloud accumulation, for static objects, all frame images within the above preset time period are counted, and the mean variance is used to detect the change in the position of the detected object after statistics. The statistical process includes the process of removing outliers, that is, when there are a few frames of abnormal situations in the multi-frame point cloud within the preset time period, they are not adjusted through reprojection or manual annotation, but directly removed as extreme values.

[0223] The above method can reduce the manpower required for labeling while ensuring accuracy.

[0224] In addition, for static objects, when accumulating multiple frames of images in the time domain, the multiple frames of objects can be merged directly with the help of the outer frame lines of the detected objects. This can omit the object detection step after the point cloud merging, reduce the amount of calculation, and avoid unclear observation problems caused by a large amount of overlap of multiple point clouds.

[0225] For dynamic objects accumulated within the preset time period, the accumulation is a phantom of the movement route within a period of time, which still needs to be detected frame by frame, and it is impossible to remove outliers like static objects. Since the movement of dynamic objects conforms to the motion model, all frame images within the preset time period are fitted to calculate whether the movement of the object conforms to the motion model. If it conforms, the detection is considered correct and can be directly marked; if it does not conform to the motion model after fitting (for example, the difference is large), it is marked and object detection is performed based on the deep learning method described above or manual adjustment is performed.

[0226] The present disclosure also provides an object labeling device, referring to Figure 7 , an object labeling device 1000 of an embodiment of the present disclosure includes:

[0227] The sensor data acquisition module 1002 acquires data collected by multiple sensors, including image data, laser point cloud data, vehicle RTK posture sequence data, and vehicle IMU / wheel speed data;

[0228] The initial alignment module 1004 performs initial spatiotemporal alignment on the image data and the laser point cloud data based on the timestamp of the image data, the timestamp of the laser point cloud data and the vehicle body RTK posture sequence data, and obtains the laser point cloud data and the image data after the initial spatiotemporal alignment;

[0229] The data segmentation module 1006 groups the laser point cloud data and the image data after the initial spatiotemporal alignment according to a preset time length;

[0230] The RTK variance calculation / judgment module 1008 obtains the RTK variance corresponding to each segmented data; based on the RTK variance corresponding to each segmented data, determines whether to use the vehicle body IMU / wheel speed data to perform posture optimization on the vehicle body RTK posture sequence data to perform secondary spatiotemporal alignment on the image data and the laser point cloud data;

[0231] A secondary alignment module 1010 uses the vehicle IMU / wheel speed data to perform posture optimization on the vehicle RTK posture sequence data to perform secondary spatiotemporal alignment on the image data and the laser point cloud data, and obtains the laser point cloud data and image data after secondary spatiotemporal alignment;

[0232] The object labeling module 1014 performs 3D object detection and 2D object detection on the laser point cloud data and the image data respectively to perform object labeling.

[0233] Figure 8 is a schematic structural diagram of an object labeling apparatus 1000 according to yet another embodiment of the present disclosure.

[0234] exist Figure 7 On the basis of, the object labeling device 1000 of this embodiment further includes a data accumulation module 1012, and the data accumulation module 1012 accumulates the laser point cloud data and the image data;

[0235] The object annotation module 1014 performs 3D object detection and 2D object detection on the accumulated laser point cloud data and the accumulated image data respectively.

[0236] The object labeling device disclosed in the present invention can be implemented based on a computer software program architecture.

[0237] The object annotation method disclosed in the present invention is an automatic annotation method. The technical solution disclosed in the present invention utilizes multiple data including 3D laser point cloud data with timestamps, video data of multiple cameras at different positions, RTK for vehicle positioning, IMU, wheel speed and related status data to improve the accuracy of automatic annotation. In some embodiments of the present invention, according to the RTK variance, the method of RTK, IMU+wheel speed fusion is progressively used to complete the time and space alignment of various data, reducing the impact of laser radar acquisition frequency asynchronization on accuracy. The technical solution disclosed in the present invention adopts a deep learning algorithm to perform automatic laser 3D object detection and image 2D object detection, and with the help of high-precision map RTK positioning data and reprojection, completes the automatic annotation and optimization adjustment process of fast 3D (laser point cloud 3D frame and video 3D frame) and image 2D frame, thereby improving the efficiency of automatic annotation. In some embodiments of the present invention, a multi-frame superposition method is adopted to accumulate point clouds, and static and dynamic objects in the scene are distinguished and annotated. The annotation and optimization adjustment of all static objects in multiple frames are completed through static one-time adjustment, thereby improving the annotation efficiency.

[0238] The technical solution disclosed in the present invention is particularly capable of completing high-precision and extensive automatic labeling of high-precision map scenes.

[0239] Figure 7 , Figure 8 It is a schematic block diagram of the structure of an object labeling device of the present disclosure using a hardware implementation of a processing system.

[0240] The device may include a corresponding module for performing each or several steps in the above flowchart. Therefore, each step or several steps in the above flowchart may be performed by a corresponding module, and the device may include one or more modules in these modules. The module may be one or more hardware modules specially configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination.

[0241] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0242] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.

[0243] Any process or method description in the flowchart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes other implementations, in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by a person skilled in the art of the present disclosure. The processor performs the various methods and processes described above. For example, the method implementation in the present disclosure can be implemented as a software program, which is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0244] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions on an instruction execution system, apparatus, or device), or in combination with such instruction execution systems, apparatuses, or devices.

[0245] For the purposes of this specification, a "readable storage medium" may be any device that can contain, store, communicate, propagate or transmit a program for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples of readable storage media (a non-exhaustive list) include the following: an electrical connection with one or more wirings (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, the readable storage medium may even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a memory.

[0246] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0247] A person skilled in the art may understand that all or part of the steps of implementing the above-mentioned implementation method may be completed by instructing related hardware through a program, and the program may be stored in a readable storage medium, which, when executed, includes one or a combination of the steps of the implementation method.

[0248] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a readable storage medium. The storage medium may be a read-only memory, a disk or an optical disk, etc.

[0249] The present disclosure also provides an electronic device, including: a memory, the memory storing execution instructions; and a processor or other hardware module, the processor or other hardware module executes the execution instructions stored in the memory, so that the processor or other hardware module executes the above-mentioned object labeling method.

[0250] The present disclosure also provides a readable storage medium, in which execution instructions are stored. When the execution instructions are executed by a processor, they are used to implement the above-mentioned object labeling method.

[0251] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms are not necessarily the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are contradictory.

[0252] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0253] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. An object labeling method, characterized in that: include: S110, acquiring data collected by multiple sensors, wherein the data includes image data, laser point cloud data, vehicle RTK posture sequence data, vehicle IMU data, and wheel speed data; S120, based on the timestamp of the image data, the timestamp of the laser point cloud data and the vehicle body RTK posture sequence data, perform initial spatiotemporal alignment on the image data and the laser point cloud data to obtain the laser point cloud data and image data after the initial spatiotemporal alignment; S130, grouping the laser point cloud data and image data after the initial spatiotemporal alignment according to a preset time length, and obtaining the RTK variance corresponding to each segmented data; S140, based on the RTK variance corresponding to each segmented data, determine whether to use the vehicle IMU data and wheel speed data to perform posture optimization on the vehicle RTK posture sequence data to perform secondary spatiotemporal alignment on the image data and the laser point cloud data, and obtain laser point cloud data and image data after secondary spatiotemporal alignment; as well as S160, performing 3D object detection and 2D object detection on the laser point cloud data and the image data respectively to perform object labeling; In step S140, the vehicle body IMU data and wheel speed data are used to optimize the vehicle body RTK pose sequence data, including: For a segment of data whose RTK variance exceeds the variance threshold, obtain the first IMU data closest to and earlier than the start time of the segment of data, and use the corresponding time as the starting point; obtain the second IMU data closest to and later than the end time of the segment of data, and use the corresponding time as the end point; Between the starting point and the end point, for each image data acquisition time t1 and each laser point cloud data acquisition time t2, the following processing is performed: The translation amount Δx of the vehicle body from the starting point to the end point is obtained based on the wheel speed data; the rotation angle differences Δyaw, Δpitch and Δroll of the vehicle body in the x, y and z directions from the starting point to the end point are obtained based on the IMU data; By integrating the rotation angle differences Δyaw, Δpitch and Δroll with the translation Δx, the translation Δy and translation Δz of the vehicle body along the y-axis and z-axis directions during the time period from t1 to t2 are obtained, that is, the position change of the vehicle body during the time period from t1 to t2.

2. The object labeling method according to claim 1, characterized in that: After step S110, the method further includes: S115, accumulating laser point cloud data and image data; In step S160, the 3D object detection and the 2D object detection are performed on the accumulated laser point cloud data and the accumulated image data respectively.

3. The object labeling method according to claim 1 or 2, characterized in that: S110, obtaining data collected by multiple sensors, including: The image data is transformed from the camera coordinate system to the vehicle body coordinate system, and the laser point cloud data is transformed from the lidar coordinate system to the vehicle body coordinate system.

4. The object labeling method according to claim 3, characterized in that: Various types of data collected by multiple sensors are recorded on a unified timeline.

5. The object labeling method according to claim 1, characterized in that: S120, based on the timestamp of the image data, the timestamp of the laser point cloud data and the vehicle body RTK posture sequence data, the image data and the laser point cloud data are initially space-time aligned to obtain the laser point cloud data and image data after the initial space-time alignment, including: S1202, for the laser point cloud data at a certain moment, find the image data at the moment closest to the moment, and obtain the time difference between the two moments; S1204, extracting the RTK pose sequence in the world coordinate system including the above two moments from the vehicle body RTK pose sequence data, to obtain the poses corresponding to the above two moments respectively; S1206, obtaining a posture difference in a world coordinate system corresponding to the time difference; and S1208. Correct the laser point cloud data in the vehicle body coordinate system based on the posture difference to obtain laser point cloud data that is spatially and temporally aligned with the image data in the vehicle body coordinate system.

6. The object labeling method according to claim 2, characterized in that: S120, based on the timestamp of the image data, the timestamp of the laser point cloud data and the vehicle body RTK posture sequence data, the image data and the laser point cloud data are initially space-time aligned to obtain the laser point cloud data and image data after the initial space-time alignment, including: S1202, for the laser point cloud data at a certain moment, obtaining the time difference between the moment and the initial radar image acquisition moment within the accumulation time period; S1204, extracting the RTK pose sequence in the world coordinate system including the above two moments from the vehicle body RTK pose sequence data, to obtain the poses corresponding to the above two moments respectively; S1206, obtaining a posture difference in a world coordinate system corresponding to the time difference; and S1208. Correct the laser point cloud data in the vehicle body coordinate system based on the posture difference to obtain laser point cloud data that is spatially and temporally aligned with the image data in the vehicle body coordinate system.

7. The object labeling method according to claim 5 or 6, characterized in that: S130, segmenting the laser point cloud data and image data after the initial spatiotemporal alignment according to a preset time length, and obtaining the RTK variance corresponding to each segmented data, including: When obtaining the RTK variance, statistics are collected on the RTK variances of all degrees of freedom.

8. The object labeling method according to claim 5 or 6, characterized in that: S140, based on the RTK variance corresponding to each segmented data, determining whether to use the vehicle body IMU data and wheel speed data to perform posture optimization on the vehicle body RTK posture sequence data to perform secondary spatiotemporal alignment on the image data and the laser point cloud data, and obtaining laser point cloud data and image data after secondary spatiotemporal alignment, including: When the RTK variance of each degree of freedom is less than or equal to the variance threshold, it is determined that the vehicle IMU / wheel speed data is not used to optimize the vehicle RTK pose sequence data; otherwise, it is determined that the vehicle IMU / wheel speed data is used to optimize the vehicle RTK pose sequence data.

9. The object labeling method according to claim 5 or 6, characterized in that: In step S140, the vehicle body IMU data and wheel speed data are used to optimize the vehicle body RTK pose sequence data, which also includes: The pose difference corresponding to the data whose RTK variance exceeds the variance threshold is replaced by the pose change.

10. The object labeling method according to claim 5 or 6, characterized in that: S160, performing 3D object detection and 2D object detection on the laser point cloud data and the image data respectively to perform object labeling, including: Detect 3D objects in laser point cloud data using a BEV-based target detection method, a range view-based target detection method, a point-wise feature-based target detection method, or a fusion feature-based target detection method; Detect 2D objects in image data using Cascade-based, Haar-based, or SVM-based object detection algorithms, R-CNN-based or Fast R-CNN-based candidate regions or boxes combined with deep learning classification algorithms, or RRC detection-based or DeformableCNN-based deep learning detection methods; and Based on the weights of the feature maps of image data and laser point cloud data, Mixture of Experts is used to build a model, and feature fusion is performed through splicing operations to complete the automatic detection of 3D objects and 2D objects.

11. The object labeling method according to claim 10, characterized in that: S160, performing 3D object detection and 2D object detection on the laser point cloud data and the image data respectively to perform object labeling, including: The 3D object is projected onto the corresponding 2D object, and the correctness of the detected object is determined based on the overlap between the 3D and 2D projections of the same object. If correct, the object is output; if incorrect, the object is not output, so as to perform object labeling.

12. The object labeling method according to claim 11, characterized in that: Projecting the 3D object onto the corresponding 2D object, judging whether the detected object is correct based on the overlap between the 3D and 2D projections of the same object, outputting the object if correct, and not outputting the object if incorrect, including: If the projection coincidence of an object is greater than or equal to the coincidence threshold, the detected object is judged to be correct; If the projection coincidence of an object is lower than the coincidence threshold, further detection is performed through pose optimization.

13. The object labeling method according to claim 12, characterized in that: Step S160 also includes: Based on the vehicle RTK posture sequence data, each vector object in the high-precision map is projected into the laser point cloud data and image data to update the laser point cloud data and image data for object annotation.

14. The object labeling method according to claim 13, characterized in that: Based on the vehicle RTK pose sequence data, each vector object in the high-precision map is projected into the laser point cloud data and image data to update the laser point cloud data and image data for object annotation, including: S1602, based on the vehicle body RTK posture sequence data, obtaining the coordinates of each vector object in the high-precision map in the world coordinate system within the observation range of the laser radar and the coordinates of the vehicle body in the world coordinate system, wherein the observation range takes the center of the laser radar as the origin; S1604, transforming the coordinates of each vector object in the world coordinate system into the vehicle body coordinate system; S1606, projecting each vector object in the vehicle body coordinate system into the laser point cloud data; S1608, transforming the coordinates of each vector object in the vehicle body coordinate system into the camera coordinate system to form a projection in the image data; and S1610. When the variance of the RTK data is less than or equal to the variance threshold, each vector object in the high-precision map is projected into the laser point cloud data and image data. When the variance of the RTK data is greater than the variance threshold, the point cloud data is optimized based on the image data for posture reprojection to complete the detection of 3D objects and 2D objects.

15. The object labeling method according to claim 14, characterized in that: Step S160 also includes: The laser point cloud data within a preset time period is superimposed to mark static objects and dynamic objects separately.

16. The object labeling method according to claim 15, characterized in that: Overlay the laser point cloud data within a preset time period to mark static objects and dynamic objects separately, including: Take the laser images collected by the laser radar within a preset time period, accumulate and overlay the point clouds, and obtain overlay data; For static objects in the overlay data, one-time object detection and labeling are performed; For dynamic objects in the superimposed data, the motion path of the dynamic object is extracted to determine whether the motion path conforms to the preset object motion model. If so, it is directly labeled based on the preset object motion model. If not, object detection and labeling are performed based on the deep learning algorithm.

17. An object labeling device, characterized in that: include: A sensor data acquisition module, wherein the sensor data acquisition module acquires data collected by multiple sensors, wherein the data includes image data, laser point cloud data, vehicle RTK posture sequence data, vehicle IMU data, and wheel speed data; A primary alignment module, wherein the primary alignment module performs a primary spatiotemporal alignment on the image data and the laser point cloud data based on the timestamp of the image data, the timestamp of the laser point cloud data, and the vehicle body RTK posture sequence data, and obtains the laser point cloud data and the image data after the primary spatiotemporal alignment; A data segmentation module, wherein the data segmentation module groups the laser point cloud data and the image data after the initial spatiotemporal alignment according to a preset time length; An RTK variance calculation / judgment module, wherein the RTK variance calculation / judgment module obtains the RTK variance corresponding to each segmented data; based on the RTK variance corresponding to each segmented data, it is determined whether to use the vehicle body IMU data and wheel speed data to perform posture optimization on the vehicle body RTK posture sequence data to perform secondary spatiotemporal alignment on the image data and the laser point cloud data; A secondary alignment module, wherein the secondary alignment module uses the vehicle body IMU data and the wheel speed data to perform posture optimization on the vehicle body RTK posture sequence data to perform secondary spatiotemporal alignment on the image data and the laser point cloud data, and obtains the laser point cloud data and image data after the secondary spatiotemporal alignment; as well as An object annotation module, wherein the object annotation module performs 3D object detection and 2D object detection on the laser point cloud data and the image data respectively to perform object annotation; The method of using the vehicle body IMU data and wheel speed data to perform posture optimization on the vehicle body RTK posture sequence data includes: For a segment of data whose RTK variance exceeds the variance threshold, obtain the first IMU data closest to and earlier than the start time of the segment of data, and use the corresponding time as the starting point; obtain the second IMU data closest to and later than the end time of the segment of data, and use the corresponding time as the end point; Between the starting point and the end point, for each image data acquisition time t1 and each laser point cloud data acquisition time t2, the following processing is performed: The translation amount Δx of the vehicle body from the starting point to the end point is obtained based on the wheel speed data; the rotation angle differences Δyaw, Δpitch and Δroll of the vehicle body in the x, y and z directions from the starting point to the end point are obtained based on the IMU data; By integrating the rotation angle differences Δyaw, Δpitch and Δroll with the translation Δx, the translation Δy and translation Δz of the vehicle body along the y-axis and z-axis directions during the time period from t1 to t2 are obtained, that is, the position change of the vehicle body during the time period from t1 to t2.

18. The object labeling device according to claim 17, characterized in that: Also includes: A data accumulation module, which accumulates laser point cloud data and image data; The object annotation module performs the 3D object detection and the 2D object detection on the accumulated laser point cloud data and the accumulated image data.

19. An electronic device, characterized in that: include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instruction stored in the memory, so that the processor executes the object labeling method according to any one of claims 1 to 16.

20. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which are used to implement the object labeling method according to any one of claims 1 to 16 when executed by a processor.

Images