Map tag generation method and device, computer equipment and readable storage medium

Abstract

The invention relates to a map tag generation method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring multiple frames of point cloud data acquired within a preset time period before the current moment, and determining a constraint relationship corresponding to the current frame of point cloud data based on position estimation, image data, map data and multiple inertial pose estimations corresponding to the current frame of point cloud data; selecting matched target point cloud data in a sliding manner from the multiple frames of point cloud data by adopting a window with a preset size, and performing pose optimization on the target point cloud data selected in the sliding manner currently based on a constraint relationship to obtain optimized poses corresponding to the multiple frames of point cloud data; and updating map data corresponding to the current moment based on each optimized pose, and mapping map element labels corresponding to the updated map data to each piece of image data to obtain map labels corresponding to each piece of image data. By adopting the method, the map tag generation efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of map construction technology, and in particular to a map label generation method, device, computer equipment, computer-readable storage medium and computer program product. Background Art

[0002] In the field of intelligent driving, perception algorithms based on deep learning play an important role in lane line and obstacle detection. Especially in recent years, with the practical application of BEV (Brids-eye-view) technology, the output space of its perception task has been transformed from 2D perspective image controls to 3D space, and the corresponding annotation space has also been transformed to 3D space. The current mainstream annotation solution is the camera solution, which annotates lane lines, obstacles, etc. on a single-frame point cloud generated by a lidar, and then projects the annotation box into the image. Subsequently, occlusion and visibility labels are manually added to the annotation results to complete the final annotation task. However, this method requires the annotator to annotate the laser frames corresponding to all image frames frame by frame, and the annotation efficiency is low. Therefore, there is a problem of low efficiency in map label generation. Summary of the invention

[0003] Based on this, it is necessary to provide a map label generation method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve the efficiency of map label generation in order to address the above technical problems.

[0004] In a first aspect, the present application provides a map label generation method, comprising:

[0005] Acquire multiple frames of point cloud data collected within a preset time period before the current moment, and obtain position estimates, image data, map data, and multiple inertial pose estimates obtained at the time of collection of each frame of point cloud data;

[0006] For each frame of point cloud data, determine the constraint relationship corresponding to the current frame of point cloud data based on the position estimation, image data, map data and multiple inertial pose estimations corresponding to the current frame of point cloud data;

[0007] A window of preset size is used to slide and select matching target point cloud data in multiple frames of point cloud data. For each slide, the pose of the target point cloud data selected by the current slide is optimized based on the constraint relationship to obtain the optimized pose corresponding to each frame of point cloud data.

[0008] The map data corresponding to the current moment is updated based on the optimized postures corresponding to each of the multiple frames of point cloud data, and the map element annotations corresponding to the updated map data are mapped to each image data to obtain the map labels corresponding to each image data.

[0009] In one embodiment, based on the position estimation, image data, map data and multiple inertial pose estimations corresponding to the current frame point cloud data, the constraint relationship corresponding to the current frame point cloud data is determined, including:

[0010] Based on multiple inertial pose estimates during the current frame and the adjacent previous frame, a first constraint relationship between the point cloud data of the current frame and the point cloud data of the adjacent previous frame is obtained;

[0011] Perform distortion correction on the point cloud data to obtain corrected point cloud data; perform incremental point cloud registration based on the corrected point cloud data and map data to obtain a second constraint relationship between the current frame point cloud data and the map data;

[0012] The position estimation corresponding to the current frame point cloud data is used as the third constraint relationship corresponding to the current frame point cloud data;

[0013] The constraint relationships corresponding to the current frame point cloud data include a first constraint relationship, a second constraint relationship, and a third constraint relationship.

[0014] In one embodiment, based on multiple inertial pose estimates during the current frame and the adjacent previous frame, a first constraint relationship between the point cloud data of the current frame and the point cloud data of the adjacent previous frame is obtained, including:

[0015] Pre-integrate multiple inertial pose estimates during the current frame and the adjacent previous frame to obtain a pre-integration result;

[0016] Based on the pre-integration result, a first constraint relationship between the current frame point cloud data and the adjacent previous frame point cloud data is obtained.

[0017] In one embodiment, a window of a preset size is used to slide and select matching target point cloud data in multiple frames of point cloud data, including:

[0018] Sort the multi-frame point cloud data according to the acquisition order to obtain a point cloud data sequence;

[0019] Obtain the pose change between the current frame point cloud data and the adjacent previous frame point cloud data. When the pose change is greater than the preset change, based on the first edge optimization rule, use a window of preset size to slide and select matching target point cloud data in the point cloud data sequence; the first edge optimization rule is that the first frame point cloud data in each window is the adjacent subsequent frame point cloud data of the first frame point cloud data in the adjacent previous window;

[0020] When the change in posture is not greater than the preset change, based on the second edge optimization rule, a window of preset size is used to slide in the point cloud data sequence to select matching target point cloud data; the second edge optimization rule is that the last frame point cloud data in each window is the adjacent rear frame point cloud data of the last frame point cloud data in the adjacent previous window, and the second last frame point cloud data in each window is the adjacent previous frame point cloud data of the last frame point cloud data in the adjacent previous window.

[0021] In one embodiment, the constraint relationship includes a first constraint relationship, a second constraint relationship, and a third constraint relationship; based on the constraint relationship, the pose of the target point cloud data currently selected by sliding is optimized to obtain the optimized poses corresponding to the multiple frames of point cloud data, including:

[0022] When the first edge optimization rule is used for sliding selection, the first constraint relationship is marginalized to obtain the fourth constraint relationship, and the third constraint relationship is marginalized to obtain the fifth constraint relationship;

[0023] Integrate the first constraint relationship, the second constraint relationship, the third constraint relationship, the fourth constraint relationship and the fifth constraint relationship into a constraint matrix;

[0024] Based on the constraint matrix and the target point cloud data selected by the current sliding, the optimized pose corresponding to the point cloud of the current frame is calculated;

[0025] Continue sliding the window until it reaches the last frame of point cloud data to obtain the optimized poses corresponding to each frame of point cloud data.

[0026] In one embodiment, the constraint relationship includes a first constraint relationship, a second constraint relationship, and a third constraint relationship; based on the constraint relationship, the pose of the target point cloud data currently selected by sliding is optimized to obtain the optimized poses corresponding to the multiple frames of point cloud data, including:

[0027] When the second edge optimization rule is used for sliding selection, the third constraint relationship is marginalized to obtain a sixth constraint relationship;

[0028] Integrate the first constraint relationship, the second constraint relationship, the third constraint relationship and the sixth constraint relationship into a constraint matrix;

[0029] Based on the constraint matrix and the target point cloud data currently selected by sliding, the optimized pose corresponding to the point cloud of the current frame is calculated;

[0030] Continue sliding the window until it reaches the last frame of point cloud data to obtain the optimized poses corresponding to each frame of point cloud data.

[0031] In a second aspect, the present application also provides a map label generation device, comprising:

[0032] An acquisition module is used to acquire multiple frames of point cloud data collected within a preset time period before the current moment, and to acquire position estimates, image data, map data, and multiple inertial pose estimates obtained at the time of acquisition of each frame of point cloud data;

[0033] A determination module, for determining, for each frame of point cloud data, a constraint relationship corresponding to the current frame of point cloud data based on a position estimate, image data, map data, and multiple inertial pose estimates corresponding to the current frame of point cloud data;

[0034] An optimization module is used to slide and select matching target point cloud data in multiple frames of point cloud data using a window of preset size. For each slide, based on the constraint relationship, the pose of the target point cloud data selected by the current slide is optimized to obtain the optimized pose corresponding to each of the multiple frames of point cloud data.

[0035] The mapping module is used to update the map data corresponding to the current moment based on the optimized postures corresponding to the multi-frame point cloud data, and map the map element annotations corresponding to the updated map data to each image data to obtain the map labels corresponding to each image data.

[0036] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0037] Acquire multiple frames of point cloud data collected within a preset time period before the current moment, and obtain position estimates, image data, map data, and multiple inertial pose estimates obtained at the time of collection of each frame of point cloud data;

[0038] For each frame of point cloud data, determine the constraint relationship corresponding to the current frame of point cloud data based on the position estimation, image data, map data and multiple inertial pose estimations corresponding to the current frame of point cloud data;

[0039] A window of preset size is used to slide and select matching target point cloud data in multiple frames of point cloud data. For each slide, the pose of the target point cloud data selected by the current slide is optimized based on the constraint relationship to obtain the optimized pose corresponding to each frame of point cloud data.

[0040] The map data corresponding to the current moment is updated based on the optimized postures corresponding to each of the multiple frames of point cloud data, and the map element annotations corresponding to the updated map data are mapped to each image data to obtain the map labels corresponding to each image data.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0042] Acquire multiple frames of point cloud data collected within a preset time period before the current moment, and obtain position estimates, image data, map data, and multiple inertial pose estimates obtained at the time of collection of each frame of point cloud data;

[0043] For each frame of point cloud data, determine the constraint relationship corresponding to the current frame of point cloud data based on the position estimation, image data, map data and multiple inertial pose estimations corresponding to the current frame of point cloud data;

[0044] A window of preset size is used to slide and select matching target point cloud data in multiple frames of point cloud data. For each slide, the pose of the target point cloud data selected by the current slide is optimized based on the constraint relationship to obtain the optimized pose corresponding to each frame of point cloud data.

[0045] The map data corresponding to the current moment is updated based on the optimized postures corresponding to each of the multiple frames of point cloud data, and the map element annotations corresponding to the updated map data are mapped to each image data to obtain the map labels corresponding to each image data.

[0046] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0047] Acquire multiple frames of point cloud data collected within a preset time period before the current moment, and obtain position estimates, image data, map data, and multiple inertial pose estimates obtained at the time of collection of each frame of point cloud data;

[0048] For each frame of point cloud data, determine the constraint relationship corresponding to the current frame of point cloud data based on the position estimation, image data, map data and multiple inertial pose estimations corresponding to the current frame of point cloud data;

[0049] A window of preset size is used to slide and select matching target point cloud data in multiple frames of point cloud data. For each slide, the pose of the target point cloud data selected by the current slide is optimized based on the constraint relationship to obtain the optimized pose corresponding to each frame of point cloud data.

[0050] The map data corresponding to the current moment is updated based on the optimized postures corresponding to each of the multiple frames of point cloud data, and the map element annotations corresponding to the updated map data are mapped to each image data to obtain the map labels corresponding to each image data.

[0051] The above-mentioned map label generation method, device, computer equipment, computer-readable storage medium and computer program product obtain multiple frames of point cloud data collected within a preset time period before the current moment, and obtain the position estimate, image data, map data and multiple inertial pose estimates obtained at the time of collection of each frame of point cloud data. For each frame of point cloud data, the position estimate, image data, map data and multiple inertial pose estimates at the corresponding collection time are used to jointly generate a constraint relationship for the current frame of point cloud data, and the constraint relationship is used to characterize the position and posture changes of the current frame of point cloud data; based on the constraint relationship, a sliding window selection method is adopted, and for each slide, the pose of the target point cloud data in the window is optimized, and the map data corresponding to the current moment is updated based on the optimized poses corresponding to each of the multiple frames of point cloud data, which is conducive to avoiding the pose angle divergence problem caused by constructing a map with a single frame of point cloud data, thereby improving the accuracy of the map data; map elements are annotated for the entire updated map data, and the map element annotations are mapped to each map data, effectively improving the efficiency of map label generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0053] FIG1 is a diagram of an application environment of a map label generation method according to an embodiment;

[0054] FIG2 is a schematic diagram of a flow chart of a method for generating a map label in one embodiment;

[0055] FIG3 is a schematic diagram of the principle of sliding window optimization in one embodiment;

[0056] FIG4 is a schematic diagram of the overall process of a map label generation method in one embodiment;

[0057] FIG5 is a schematic diagram of a process of sliding window optimization in one embodiment;

[0058] FIG6 is a structural block diagram of a map label generating device in one embodiment;

[0059] FIG. 7 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] The map label generation method provided in the embodiment of the present application can be applied to the application environment as shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. This embodiment is illustrated by the application of the method to the terminal 102. It can be understood that the method can also be applied to the server 104, and can also be applied to the system including the terminal 102 and the server 104, and is implemented through the interaction between the terminal 102 and the server 104. The terminal 102 obtains multiple frames of point cloud data collected within a preset time period before the current moment, and obtains the position estimation, image data, map data and multiple inertial pose estimations obtained at the time of collection of each frame of point cloud data; for each frame of point cloud data, based on the position estimation, image data, map data and multiple inertial pose estimations corresponding to the current frame of point cloud data, the constraint relationship corresponding to the current frame of point cloud data is determined; a window of preset size is used to slide and select matching target point cloud data in the multiple frames of point cloud data, and for each slide, the pose of the target point cloud data currently selected by the slide is optimized based on the constraint relationship to obtain the optimized pose corresponding to each of the multiple frames of point cloud data; based on the optimized pose corresponding to each of the multiple frames of point cloud data, the map data corresponding to the current moment is updated, and the map element annotation corresponding to the updated map data is mapped to each image data to obtain the map label corresponding to each image data. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0062] In an exemplary embodiment, as shown in FIG2 , a map label generation method is provided, which is described by taking the method applied to the terminal 102 in FIG1 as an example, and includes the following steps 202 to 208 . Among them:

[0063] Step 202, obtain multiple frames of point cloud data collected within a preset time period before the current moment, and obtain position estimation, image data, map data and multiple inertial pose estimations obtained at the time of collection of each frame of point cloud data.

[0064] The point cloud data may be the three-dimensional points and intensity information of obstacles collected by the ranging sensor. In the field of intelligent driving, the ranging sensor may be a radar sensor installed in a mobile platform, for example, a laser radar, illustratively, a 64-line laser radar or a 128-line laser radar. The radar sensor is installed on the top or around the mobile platform, and may be one or more, for collecting three-dimensional point cloud data. The data format of the three-dimensional point cloud data is: frame number, timestamp, and three-dimensional coordinates of obstacles scanned by the laser.

[0065] Position estimation refers to the position information collected by a positioning device, such as a GPS (Global Positioning System) device.

[0066] Image data refers to images acquired through image sensors. Exemplarily, the image sensors are installed on a mobile platform, including: a front telephoto camera, a front wide-angle camera, a rear mid-focus camera, a left front camera, a left rear camera, a front camera, a rear camera, a front fisheye camera, a rear fisheye camera, a left fisheye camera, and a right fisheye camera. 11V images can be acquired through the image sensors installed at different positions of the mobile platform.

[0067] Map data refers to a point cloud map obtained by stitching point clouds. Map data acquired by the terminal at the current moment refers to a point cloud map obtained by stitching multiple frames of point cloud data before the current moment.

[0068] Inertial pose estimation refers to the data collected by the Inertial Measurement Unit (IMU). The IMU is installed on the mobile platform. The data format of inertial pose estimation is: frame number, timestamp, linear acceleration of the mobile platform, and angular velocity of the mobile platform.

[0069] In some embodiments, the acquisition frequency of the radar sensor is 10 Hz, the acquisition frequency of the inertial measurement unit is 200 Hz, the acquisition frequency of the positioning device is 7 Hz, and the acquisition frequency of the image sensor is 10 Hz. Different sensors have different data acquisition frequencies. In order to facilitate data processing in subsequent steps, the data information of each sensor needs to be aligned.

[0070] Taking the point cloud data frame (i.e. the point cloud data frame collected by the radar sensor) as the node, all position estimates, image data, map data and multiple inertial pose estimates from the collection time of the current frame point cloud data to the collection time of the next frame point cloud data are integrated or interpolated and added to the current node to align the data information of each sensor.

[0071] Step 204 , for each frame of point cloud data, based on the position estimation, image data, map data and multiple inertial pose estimations corresponding to the current frame of point cloud data, determine the constraint relationship corresponding to the current frame of point cloud data.

[0072] Among them, since the data collected by multiple sensors are collected at the time of collecting each frame of point cloud data, the terminal can integrate the data collected by multiple sensors to obtain the constraint relationship between the data collected by multiple sensors and the current frame of point cloud data. The constraint relationship is used to characterize the position and posture changes of the current frame of point cloud data.

[0073] Step 206, using a window of preset size to slide and select matching target point cloud data in the multi-frame point cloud data, for each slide, based on the constraint relationship, the pose of the target point cloud data currently selected by the slide is optimized to obtain the optimized pose corresponding to each of the multi-frame point cloud data.

[0074] Among them, when using single-frame point cloud data to construct map data, serious attitude angle divergence problems will occur when the mobile platform runs long distances. Therefore, the embodiment of the present application proposes to use a sliding window method. Each time the sliding is performed, the target point cloud data that matches the window of preset size is selected from the multi-frame point cloud data, so that the target point cloud data in one window can be processed each time. There is often a certain constraint relationship between the target point cloud data of each frame in a window. Therefore, the sliding window method is conducive to avoiding the attitude angle divergence problem when the mobile platform runs long distances, thereby helping to improve the accuracy of the constructed map data.

[0075] Step 208 , based on the optimized postures corresponding to the multiple frames of point cloud data, the map data corresponding to the current moment is updated, and the map element annotations corresponding to the updated map data are mapped to each image data to obtain the map labels corresponding to each image data.

[0076] Among them, updating the map data with optimized posture is conducive to improving the accuracy of the map data. The terminal annotates the map elements of the entire updated map data to obtain the map element annotations of the entire updated map data. Since the annotation is not performed frame by frame, but only once, the annotation efficiency is greatly improved. Map element annotation refers to the map element annotation results corresponding to the updated map data, including lane lines, curbs and other map elements.

[0077] The mapping method of map element annotation is that the terminal can read the posture information, lidar external parameters, and image external parameters of each 11V image data, so as to project the map element annotations into each image data and complete the generation of map labels in the image.

[0078] In the above-mentioned map label generation method, by obtaining multiple frames of point cloud data collected within a preset time period before the current moment, and obtaining the position estimate, image data, map data and multiple inertial pose estimates obtained at the time of collection of each frame of point cloud data, for each frame of point cloud data, the position estimate, image data, map data and multiple inertial pose estimates at the corresponding time of collection are used to jointly generate a constraint relationship for the current frame of point cloud data, and the constraint relationship is used to characterize the position and posture changes of the current frame of point cloud data; based on the constraint relationship, a sliding window selection method is adopted, and for each slide, the pose of the target point cloud data in the window is optimized, and the map data corresponding to the current moment is updated based on the optimized poses corresponding to each of the multiple frames of point cloud data, which is conducive to avoiding the attitude angle divergence problem caused by constructing a map with a single frame of point cloud data, thereby improving the accuracy of the map data; map elements are annotated for the entire updated map data, and the map element annotations are mapped to each map data, effectively improving the efficiency of map label generation.

[0079] In an exemplary embodiment, based on the position estimate, image data, map data and multiple inertial pose estimates corresponding to the current frame point cloud data, the constraint relationship corresponding to the current frame point cloud data is determined, including: obtaining a first constraint relationship between the current frame point cloud data and the adjacent previous frame point cloud data based on multiple inertial pose estimates during the current frame and the adjacent previous frame; performing distortion correction on the point cloud data to obtain corrected point cloud data; performing incremental point cloud registration based on the corrected point cloud data and map data to obtain a second constraint relationship between the current frame point cloud data and the map data; using the position estimate corresponding to the current frame point cloud data as the third constraint relationship corresponding to the current frame point cloud data; the constraint relationships corresponding to the current frame point cloud data include a first constraint relationship, a second constraint relationship and a third constraint relationship.

[0080] The first constraint relationship is used to characterize the pose and position change between the current frame point cloud data and the adjacent previous frame point cloud data. Since the acquisition frequency of the inertial measurement unit is high, the terminal can obtain multiple inertial pose estimates during the acquisition of a single frame of point cloud data, and further obtain the first constraint relationship based on multiple inertial pose estimates during the current frame and the adjacent previous frame.

[0081] The second constraint relationship is used to characterize the change in the pose and position of the current frame point cloud data relative to the map data. The second constraint relationship can be obtained by performing point cloud registration on the current frame point cloud data and the map data. In order to further improve the accuracy of the second constraint relationship, the terminal can first perform distortion correction on the current frame point cloud data to obtain the corrected point cloud data, and then perform point cloud registration on the corrected point cloud data and the map data to obtain the registration result, and use the configuration result as the second constraint relationship. Among them, the point cloud registration can adopt the incremental point cloud registration method.

[0082] Among them, since the current frame point cloud data is collected within a preset time period (for example, 0.1 seconds), and the mobile platform is moving within the preset time period, there is a slight deviation in the posture of each point in the point cloud data. It may be necessary to perform distortion correction on the point cloud data to improve the accuracy of the point cloud data.

[0083] Distortion correction can be performed using the following formula:

[0084]

[0085] in, is the i-th point of the point cloud data before correction, is the i-th point of the corrected point cloud data, and They respectively represent the rotation matrix and translation vector of the acquisition pose of the i-th point before correction in the acquisition pose coordinate system of the 0-th point.

[0086] The third constraint relationship is used to characterize the position of the current frame point cloud data. Exemplarily, the terminal may use the position estimate corresponding to the current frame point cloud data as the third constraint relationship.

[0087] The terminal may take the first constraint relationship, the second constraint relationship and the third constraint relationship together as a constraint relationship, or may perform weighted summation on the first constraint relationship, the second constraint relationship and the third constraint relationship and take the weighted summation result as the constraint relationship.

[0088] In this embodiment, three constraint relationships are determined respectively through inertial posture estimation, map data and position estimation, and the three constraint relationships are used as the final constraint relationships. In this way, the point cloud data is constrained through the collection of multiple sensors, which is beneficial to improve the accuracy of the posture and position corresponding to the point cloud data.

[0089] In an exemplary embodiment, based on multiple inertial pose estimates during a current frame and an adjacent previous frame, a first constraint relationship between point cloud data of a current frame and point cloud data of an adjacent previous frame is obtained, including: pre-integrating multiple inertial pose estimates during the current frame and an adjacent previous frame to obtain a pre-integration result; based on the pre-integration result, obtaining a first constraint relationship between point cloud data of a current frame and point cloud data of an adjacent previous frame.

[0090] Among them, the inertial pose estimation includes position, attitude, angular velocity and linear velocity. The terminal can interpolate multiple inertial pose estimates during the current frame and the adjacent previous frame, thereby pre-integrating multiple interpolated inertial pose estimates. Exemplarily, the original inertial pose estimation measured by the inertial measurement unit only contains linear acceleration and angular velocity information, and the inertial measurement unit has inherent errors. Therefore, the IMU observation model based on inherent errors is shown in the following formula:

[0091]

[0092]

[0093] in: (t) is the original angular velocity measured by the inertial measurement unit at time t, (t) is the original linear acceleration measured by the inertial measurement unit at time t, (t) and They are the angular velocity bias and linear acceleration bias of the inertial measurement unit data at time t, (t) and They are the random walk of angular velocity and linear acceleration of the inertial measurement unit data at time t, is the acceleration due to gravity, (t) is the rotation matrix of the world reference coordinate system in the IMU coordinate system. This rotation matrix is ​​the variable to be solved. (t) and They are the real angular velocity and linear acceleration of the mobile platform reflected by the IMU data at time t. The terminal further pre-integrates the real angular velocity and real linear acceleration that truly reflect the motion situation, and uses the pre-integration result as the first constraint relationship. Therefore, the first constraint relationship can be used to characterize the changes in position, posture, and linear velocity between the adjacent previous frame and the current frame.

[0094] In this embodiment, observations based on inherent errors are performed on the original data measured by the inertial measurement unit to obtain the true angular velocity and true linear acceleration, and then pre-integration is performed on multiple true inertial pose estimates during the current frame and the adjacent previous frame to obtain the first constraint relationship. This can eliminate the interference of errors and is conducive to obtaining an accurate first constraint relationship.

[0095] In an exemplary embodiment, a window of preset size is used to slide and select matching target point cloud data in multiple frames of point cloud data, including: sorting the multiple frames of point cloud data in the order of acquisition to obtain a point cloud data sequence; obtaining the pose change between the current frame point cloud data and the adjacent previous frame point cloud data, and when the pose change is greater than the preset change, based on the first edge optimization rule, using a window of preset size to slide and select matching target point cloud data in the point cloud data sequence; the first edge optimization rule is that the first frame point cloud data in each window is the adjacent subsequent frame point cloud data of the first frame point cloud data in the adjacent previous window; when the pose change is not greater than the preset change, based on the second edge optimization rule, using a window of preset size to slide and select matching target point cloud data in the point cloud data sequence; the second edge optimization rule is that the last frame point cloud data in each window is the adjacent subsequent frame point cloud data of the last frame point cloud data in the adjacent previous window, and the second last frame point cloud data in each window is the adjacent previous frame point cloud data of the last frame point cloud data in the adjacent previous window.

[0096] Among them, the terminal performs sliding window optimization on multiple frames of point cloud data. First, each frame of point cloud data should be sorted according to the acquisition order to obtain a point cloud data sequence. Each time the window slides, the target point cloud data that matches the window of the preset size is selected from the point cloud data sequence. For example, if the window size is 20, each time the window slides, 20 frames of point cloud data are selected from the point cloud data sequence as the target point cloud data.

[0097] At the same time, the target point cloud data in each window can adopt the first edge optimization rule or the second edge optimization rule, and which rule to adopt is determined based on whether the change in posture between the current frame point cloud data and the adjacent previous frame point cloud data exceeds the preset change. Among them, for the first edge optimization rule, if the window size is 20, then after adopting the first edge optimization rule, the target point cloud data obtained by the first window sliding includes 0-19 frames of point cloud data, the target point cloud data obtained by the second window sliding includes 1-20 frames of point cloud data, and the target point cloud data obtained by the third window sliding includes 2-21 frames of point cloud data, and so on. That is, the first edge optimization rule is equivalent to removing the oldest frame from the window each time.

[0098] For the second edge optimization rule, if the window size is 20, after the second edge optimization rule is adopted, the target point cloud data obtained by the first window sliding includes the point cloud data of frames 0-19, the target point cloud data obtained by the second window sliding includes the point cloud data of frames 0-18 and 20, the target point cloud data obtained by the second window sliding includes the point cloud data of frames 0-18 and 21, the target point cloud data obtained by the third window sliding includes the point cloud data of frames 0-18 and 22, and so on. That is, the second edge optimization rule is equivalent to removing the newest frame from the window each time.

[0099] In this embodiment, as the window slides, if some point cloud frames are not eliminated, the point cloud data in the window will continue to increase, seriously affecting the system operation efficiency. Based on this, it is proposed to eliminate the oldest frame or the second newest frame from the point cloud data in the window each time the window is slid to ensure that the point cloud data within the sliding remains the same, which is beneficial to improving the system operation efficiency.

[0100] In an exemplary embodiment, the constraint relationship includes a first constraint relationship, a second constraint relationship and a third constraint relationship; based on the constraint relationship, the pose of the target point cloud data currently selected by sliding is optimized to obtain the optimized pose corresponding to each of the multiple frames of point cloud data, including: marginalizing the first constraint relationship to obtain a fourth constraint relationship, marginalizing the third constraint relationship to obtain a fifth constraint relationship; integrating the first constraint relationship, the second constraint relationship, the third constraint relationship, the fourth constraint relationship and the fifth constraint relationship into a constraint matrix; based on the constraint matrix and the target point cloud data currently selected by sliding, the optimized pose corresponding to the current frame point cloud is calculated; the sliding window is continued until it slides to the last frame of point cloud data to obtain the optimized pose corresponding to each of the multiple frames of point cloud data.

[0101] Among them, when the window is sliding, directly eliminating the oldest frame or the newest frame will lose the constraint relationship between the eliminated frame node and the node in the window, resulting in the lack of anchor points in the sliding window, causing all optimized nodes in the sliding window to drift. To solve this problem, the embodiment of the present application proposes to marginalize the eliminated nodes, retain the necessary constraint relationship in the last sliding window, and provide anchor points for the current sliding window optimization, thereby improving the accuracy of mapping and positioning. As shown in Figure 3, it is a schematic diagram of the principle of sliding window optimization in an embodiment. Among them, the figure shows the point cloud data frame nodes of 9 consecutive moments. In the actual program, 20 consecutive laser frame nodes are set to form a window. There is a first constraint relationship provided by IMU between the point cloud data frame nodes to constrain the position change between adjacent nodes. If there is GPS information, each node also has a global position constraint provided by GPS, and the third constraint relationship is used to characterize the position of the current frame point cloud data. Define the oldest laser frame node or the second newest laser frame node as a sliding window, and optimize all the node poses, node velocities, and IMU biases in the window with all the constraints mentioned above. Add the optimized results and point cloud data to the map data to update the map data. With the addition of point cloud data frame nodes, the sliding window is moved, and then the pose, linear velocity, and angular velocity of the latest frame node are obtained, and the updated map data is obtained at the same time.

[0102] Different posture optimization methods will be adopted for different optimization rules. Specifically, when the first edge optimization rule is adopted for sliding selection, the terminal can marginalize the first constraint relationship to obtain the fourth constraint relationship, marginalize the third constraint relationship to obtain the fifth constraint relationship, and then integrate the first constraint relationship, the second constraint relationship, the third constraint relationship, the fourth constraint relationship and the fifth constraint relationship into a matrix, namely the constraint matrix, and use the constraint matrix to marginalize each frame of target point cloud data of the current sliding selection.

[0103] The marginalization method can be adopted as follows:

[0104]

[0105]

[0106] in, represents the point cloud data frame node to be marginalized, is the node that has a constraint relationship with the node to be marginalized. are the data expressions of the constraint relations respectively. By eliminating the formulas, we can get the For example, is the point cloud data of the first frame, then It can be the point cloud data of the second frame.

[0107] In this embodiment, when the first edge optimization rule is used for sliding selection, the posture of each frame of target point cloud data of the current sliding selection is optimized based on the constraint matrix obtained by integrating five constraint relationships. This method of jointly determining the optimized posture through five constraint relationships is conducive to improving the accuracy of the optimized posture.

[0108] In an exemplary embodiment, the constraint relationship includes a first constraint relationship, a second constraint relationship and a third constraint relationship; based on the constraint relationship, the pose of the target point cloud data currently selected by sliding is optimized to obtain the optimized pose corresponding to each of the multiple frames of point cloud data, including: when the second edge optimization rule is used for sliding selection, the third constraint relationship is marginalized to obtain the sixth constraint relationship; the first constraint relationship, the second constraint relationship, the third constraint relationship and the sixth constraint relationship are integrated into a constraint matrix; based on the constraint matrix and the target point cloud data currently selected by sliding, the optimized pose corresponding to the current frame point cloud is calculated; the sliding window is continued until it slides to the last frame of point cloud data to obtain the optimized pose corresponding to each of the multiple frames of point cloud data.

[0109] Among them, when the second edge optimization rule is used for sliding selection, since the change in posture between the second-newest frame and the latest frame is small and the inertial posture estimation of the second-newest frame is not lost, there is no need to marginalize the first constraint relationship obtained based on the inertial posture estimation. Specifically, the terminal can marginalize the third constraint relationship to obtain the sixth optimized posture, integrate the first constraint relationship, the second constraint relationship, the third constraint relationship and the sixth constraint relationship into a constraint matrix, and then marginalize the target point cloud data of the current sliding selection based on the constraint matrix to obtain the optimized posture corresponding to the current frame point cloud, and continue to slide the window to obtain the optimized posture corresponding to each of the multiple frames of point cloud data. Among them, the marginalization method can refer to the marginalization method when the first edge optimization rule is used for sliding selection.

[0110] In this embodiment, when the second edge optimization rule is used for sliding selection, the third constraint relationship can be marginalized to obtain the sixth optimized pose, and then the four constraint relationships are integrated to obtain the constraint matrix. Finally, the constraint matrix is ​​used to marginalize the target point cloud data. This edge optimization method marginalizes the eliminated nodes, retains the necessary constraint relationships in the last sliding window, provides an anchor point for each sliding window optimization, and improves the accuracy of mapping and positioning.

[0111] To explain the map label generation method and effect in detail, the following is a most detailed example:

[0112] The map label generation method can be applied to the field of intelligent driving. Among them, the map label generation device includes a mobile platform, a ranging sensor, an inertial measurement unit, and an image acquisition device. Among them, the IMU can provide 6-degree-of-freedom inertial pose estimation, the GPS can provide x, y, and z position estimation, the radar sensor can provide 6-degree-of-freedom point cloud pose estimation, and the image sensor is installed on the mobile platform, including: front telephoto camera, front wide-angle camera, rear mid-focus camera, left front camera, left rear camera, front camera, rear camera, front fisheye camera, rear fisheye camera, left fisheye camera, and fisheye camera. Through the above-mentioned image sensors installed at different positions on the mobile platform, 11V images can be collected.

[0113] As shown in FIG4 , it is a schematic diagram of the overall process of a map label generation method in an embodiment. In step 401, the terminal obtains multiple frames of point cloud data collected within a preset time period before the current moment, and obtains the position estimate, image data, map data and multiple inertial pose estimates obtained at the time of collection of each frame of point cloud data. In which, the point cloud data collected by the radar sensor is used as a node, and the collected data of the IMU, GPS and image sensor can be timestamped. In step 402, the collected inertial pose estimate is pre-integrated to obtain the first constraint relationship between the current frame point cloud data and the adjacent previous frame point cloud data. In step 403, the current frame point cloud data is distorted by using the inertial pose estimate, and the corrected point cloud data is incrementally aligned with the map data to obtain the second constraint relationship between the current frame point cloud data and the map data. In step 404, the position estimate corresponding to the current frame point cloud data is used as the third constraint relationship corresponding to the current frame point cloud data. Step 405: Based on the constraint relationship, the multi-frame point cloud data is subjected to sliding window optimization, and the optimized posture is added to the map data, and the constraint relationship includes a first constraint relationship, a second constraint relationship, and a third constraint relationship. Step 406: The multi-frame corrected point cloud data is spliced ​​to obtain a spliced ​​point cloud, and the map data is updated according to the spliced ​​point cloud and the optimized posture, and the map element annotation corresponding to the updated map data is mapped to each image data to obtain the map label corresponding to each image data.

[0114] FIG5 is a schematic diagram of a process of sliding window optimization in an embodiment.

[0115] Step 501, add laser frame nodes at the acquisition frame rate of the ranging sensor, and the node data includes: point cloud data collected at the current moment, IMU pre-integration collected from the previous moment to the current moment, and GPS data at the current moment. Step 502, optimization, specifically, using all the above-mentioned constraint relationships and constraints obtained by marginalization, optimize all node poses, node velocities, and IMU biases in the window. Step 503, use the pose change of the current frame node pose relative to the adjacent previous frame node pose to determine whether to marginalize the oldest frame or the marginalize the newest frame. In the actual program, the rotation angle threshold in the preset change is set to 0.017 radians, and the translation threshold is 0.5 meters. Step 504, marginalize the oldest frame node. Step 505, marginalize the newest frame node. Step 506, marginalize the IMU pre-integration constraint connected to the oldest frame node. When marginalizing one of the nodes, the constraint is converted into a separate constraint for the remaining nodes. After sliding the window, the separate constraint is retained, thereby providing an anchor point for optimization. Step 507, marginalize the GPS constraints connected to the oldest frame node. Step 508, marginalize the GPS constraints connected to the next-to-newest frame node. Step 509, add the IMU data of the next-to-newest frame to the latest frame, and perform IMU pre-integration. Specifically: Unlike marginalizing the oldest frame, there is no marginalization of IMU constraints in marginalizing the next-to-newest frame. The reason is that the change in posture between the next-to-newest frame and the latest frame is small, and the IMU of the next-to-newest frame is not lost. Step 510, integrate the marginalized constraints obtained from marginalizing the oldest frame or marginalizing the next-to-newest frame, and perform a new sliding window optimization for the input of the next frame node.

[0116] The above-mentioned map label generation method obtains multiple frames of point cloud data collected within a preset time period before the current moment, and obtains the position estimate, image data, map data and multiple inertial pose estimates obtained at the time of collection of each frame of point cloud data. For each frame of point cloud data, the position estimate, image data, map data and multiple inertial pose estimates at the corresponding time of collection are used to jointly generate a constraint relationship for the current frame of point cloud data, and the constraint relationship is used to characterize the position and posture changes of the current frame of point cloud data; based on the constraint relationship, a sliding window selection method is adopted, and for each slide, the pose of the target point cloud data in the window is optimized, and the map data corresponding to the current moment is updated based on the optimized poses corresponding to each of the multiple frames of point cloud data, which is conducive to avoiding the attitude angle divergence problem caused by constructing a map with a single frame of point cloud data, thereby improving the accuracy of the map data; map elements are annotated for the entire updated map data, and the map element annotations are mapped to each map data, effectively improving the efficiency of map label generation. In addition, the method of performing one annotation on the spliced ​​point cloud and multiplexing to realize 11V image projection at different times and complete the preliminary results of all image annotations greatly improves the efficiency of map label generation. At the same time, this annotation method can ensure the consistency of annotation results between adjacent frames. The embodiment of the present application can realize the annotation of static elements in scenes such as high-speed elevated roads, urban scenes, and parking lots.

[0117] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0118] Based on the same inventive concept, the embodiment of the present application also provides a map label generation device for implementing the map label generation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more map label generation device embodiments provided below can refer to the limitations of the map label generation method above, and will not be repeated here.

[0119] In an exemplary embodiment, as shown in FIG6 , a map label generating device 600 is provided, comprising: an acquisition module 620 , a determination module 640 , an optimization module 660 and a mapping module 680 , wherein:

[0120] An acquisition module 620 is used to acquire multiple frames of point cloud data collected within a preset time period before the current moment, and to acquire position estimates, image data, map data, and multiple inertial pose estimates obtained at the moment of collection of each frame of point cloud data;

[0121] A determination module 640 is used to determine, for each frame of point cloud data, a constraint relationship corresponding to the current frame of point cloud data based on a position estimate, image data, map data, and multiple inertial pose estimates corresponding to the current frame of point cloud data;

[0122] The optimization module 660 is used to slide and select matching target point cloud data in the multi-frame point cloud data using a window of a preset size, and optimize the pose of the target point cloud data selected by the current sliding based on the constraint relationship for each sliding, so as to obtain the optimized pose corresponding to each of the multi-frame point cloud data;

[0123] The mapping module 680 is used to update the map data corresponding to the current moment based on the optimized postures corresponding to the multi-frame point cloud data, and map the map element annotations corresponding to the updated map data to each image data to obtain the map labels corresponding to each image data.

[0124] The above-mentioned map label generation device obtains multiple frames of point cloud data collected within a preset time period before the current moment, and obtains the position estimate, image data, map data and multiple inertial pose estimates obtained at the time of collection of each frame of point cloud data. For each frame of point cloud data, the position estimate, image data, map data and multiple inertial pose estimates at the corresponding time of collection are used to jointly generate a constraint relationship for the current frame of point cloud data, and the constraint relationship is used to characterize the position and posture changes of the current frame of point cloud data; based on the constraint relationship, a sliding window selection method is adopted, and for each slide, the pose of the target point cloud data in the window is optimized, and the map data corresponding to the current moment is updated based on the optimized poses corresponding to each of the multiple frames of point cloud data, which is conducive to avoiding the attitude angle divergence problem caused by constructing a map with a single frame of point cloud data, thereby improving the accuracy of the map data; map elements are annotated for the entire updated map data, and the map element annotations are mapped to each map data, effectively improving the efficiency of map label generation.

[0125] In one embodiment, based on the position estimate, image data, map data and multiple inertial pose estimates corresponding to the current frame point cloud data, the constraint relationship corresponding to the current frame point cloud data is determined, and the determination module 640 is also used to: obtain a first constraint relationship between the current frame point cloud data and the adjacent previous frame point cloud data based on multiple inertial pose estimates during the current frame and the adjacent previous frame; perform distortion correction on the point cloud data to obtain corrected point cloud data; perform incremental point cloud registration based on the corrected point cloud data and map data to obtain a second constraint relationship between the current frame point cloud data and the map data; use the position estimate corresponding to the current frame point cloud data as the third constraint relationship corresponding to the current frame point cloud data; the constraint relationship corresponding to the current frame point cloud data includes a first constraint relationship, a second constraint relationship and a third constraint relationship.

[0126] In one embodiment, based on multiple inertial pose estimates during the current frame and the adjacent previous frame, a first constraint relationship between the point cloud data of the current frame and the point cloud data of the adjacent previous frame is obtained, and the determination module 640 is also used to: pre-integrate the multiple inertial pose estimates during the current frame and the adjacent previous frame to obtain a pre-integration result; based on the pre-integration result, obtain the first constraint relationship between the point cloud data of the current frame and the point cloud data of the adjacent previous frame.

[0127] In one embodiment, a window of preset size is used to slide and select matching target point cloud data in multiple frames of point cloud data, and the optimization module 660 is also used to: sort the multiple frames of point cloud data according to the acquisition order to obtain a point cloud data sequence; obtain the pose change between the current frame point cloud data and the adjacent previous frame point cloud data, and when the pose change is greater than the preset change, based on the first edge optimization rule, use a window of preset size to slide and select matching target point cloud data in the point cloud data sequence; the first edge optimization rule is that the first frame point cloud data in each window is the adjacent subsequent frame point cloud data of the first frame point cloud data in the adjacent previous window; when the pose change is not greater than the preset change, based on the second edge optimization rule, use a window of preset size to slide and select matching target point cloud data in the point cloud data sequence; the second edge optimization rule is that the last frame point cloud data in each window is the adjacent subsequent frame point cloud data of the last frame point cloud data in the adjacent previous window, and the second last frame point cloud data in each window is the adjacent previous frame point cloud data of the last frame point cloud data in the adjacent previous window.

[0128] In one embodiment, the constraint relationship includes a first constraint relationship, a second constraint relationship and a third constraint relationship; based on the constraint relationship, the pose of the target point cloud data currently selected by sliding is optimized to obtain the optimized pose corresponding to each of the multiple frames of point cloud data. The optimization module 660 is also used for: when the first edge optimization rule is used for sliding selection, the first constraint relationship is marginalized to obtain the fourth constraint relationship, and the third constraint relationship is marginalized to obtain the fifth constraint relationship; the first constraint relationship, the second constraint relationship, the third constraint relationship, the fourth constraint relationship and the fifth constraint relationship are integrated into a constraint matrix; based on the constraint matrix and the target point cloud data currently selected by sliding, the optimized pose corresponding to the current frame point cloud is calculated; the sliding window is continued until it slides to the last frame of point cloud data to obtain the optimized pose corresponding to each of the multiple frames of point cloud data.

[0129] In one embodiment, the constraint relationship includes a first constraint relationship, a second constraint relationship and a third constraint relationship; based on the constraint relationship, the pose of the target point cloud data currently selected by sliding is optimized to obtain the optimized pose corresponding to each of the multiple frames of point cloud data. The optimization module 660 is also used for: when the second edge optimization rule is used for sliding selection, the third constraint relationship is marginalized to obtain a sixth constraint relationship; the first constraint relationship, the second constraint relationship, the third constraint relationship and the sixth constraint relationship are integrated into a constraint matrix; based on the constraint matrix and the target point cloud data currently selected by sliding, the optimized pose corresponding to the current frame point cloud is calculated; the sliding window is continued until it slides to the last frame of point cloud data to obtain the optimized pose corresponding to each of the multiple frames of point cloud data.

[0130] Each module in the above-mentioned map label generation device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0131] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG7. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for generating map labels is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0132] Those skilled in the art will understand that the structure shown in FIG. 7 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0133] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0134] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0135] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0136] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0137] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0138] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0139] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for generating map labels, characterized in that: The method comprises: Acquire multiple frames of point cloud data collected within a preset time period before the current moment, and obtain position estimates, image data, map data, and multiple inertial pose estimates obtained at the time of collection of each frame of point cloud data; For each frame of point cloud data, determine the constraint relationship corresponding to the current frame of point cloud data based on the position estimation, image data, map data and multiple inertial pose estimations corresponding to the current frame of point cloud data; A window of preset size is used to slide and select matching target point cloud data in multiple frames of point cloud data, and for each slide, based on the constraint relationship, the pose of the target point cloud data currently selected by the slide is optimized to obtain the optimized pose corresponding to each of the multiple frames of point cloud data; The map data corresponding to the current moment is updated based on the optimized postures corresponding to each of the multiple frames of point cloud data, and the map element annotations corresponding to the updated map data are mapped to each image data to obtain the map labels corresponding to each image data.

2. The method according to claim 1, characterized in that: The determining of the constraint relationship corresponding to the current frame point cloud data based on the position estimation, image data, map data and multiple inertial pose estimations corresponding to the current frame point cloud data includes: Based on a plurality of inertial pose estimates during the current frame and the adjacent previous frame, a first constraint relationship between the point cloud data of the current frame and the point cloud data of the adjacent previous frame is obtained; Performing distortion correction on the point cloud data to obtain corrected point cloud data; performing incremental point cloud registration based on the corrected point cloud data and the map data to obtain a second constraint relationship between the current frame point cloud data and the map data; The position estimation corresponding to the current frame point cloud data is used as the third constraint relationship corresponding to the current frame point cloud data; The constraint relationships corresponding to the current frame point cloud data include the first constraint relationship, the second constraint relationship, and the third constraint relationship.

3. The method according to claim 2, characterized in that The step of obtaining a first constraint relationship between the point cloud data of the current frame and the point cloud data of the adjacent previous frame based on multiple inertial pose estimates during the current frame and the adjacent previous frame includes: Pre-integrating a plurality of inertial pose estimates during the current frame and an adjacent previous frame to obtain a pre-integration result; Based on the pre-integration result, a first constraint relationship between the current frame point cloud data and the adjacent previous frame point cloud data is obtained.

4. The method according to claim 1, characterized in that The method of using a window of a preset size to slide and select matching target point cloud data in multiple frames of point cloud data includes: Sort the multi-frame point cloud data according to the acquisition order to obtain a point cloud data sequence; Obtaining the pose change amount between the current frame point cloud data and the adjacent previous frame point cloud data, and when the pose change amount is greater than a preset change amount, based on a first edge optimization rule, using a window of a preset size to slide in the point cloud data sequence to select matching target point cloud data; the first edge optimization rule is that the first frame point cloud data in each window is the adjacent subsequent frame point cloud data of the first frame point cloud data in the adjacent previous window; When the change in posture is not greater than a preset change, based on the second edge optimization rule, a window of preset size is used to slide in the point cloud data sequence to select matching target point cloud data; the second edge optimization rule is that the last frame point cloud data in each window is the adjacent rear frame point cloud data of the last frame point cloud data in the adjacent front window, and the second last frame point cloud data in each window is the adjacent front frame point cloud data of the last frame point cloud data in the adjacent front window.

5. The method according to claim 4, characterized in that The constraint relationship includes a first constraint relationship, a second constraint relationship and a third constraint relationship; based on the constraint relationship, the pose optimization of the target point cloud data currently selected by sliding is performed to obtain the optimized poses corresponding to the multiple frames of point cloud data, including: When the first edge optimization rule is used for sliding selection, the first constraint relationship is marginalized to obtain a fourth constraint relationship, and the third constraint relationship is marginalized to obtain a fifth constraint relationship; Integrate the first constraint relationship, the second constraint relationship, the third constraint relationship, the fourth constraint relationship and the fifth constraint relationship into a constraint matrix; Based on the constraint matrix and the target point cloud data currently selected by sliding, the optimized pose corresponding to the point cloud of the current frame is calculated; Continue sliding the window until it reaches the last frame of point cloud data to obtain the optimized poses corresponding to each frame of point cloud data.

6. The method according to claim 4, characterized in that The constraint relationship includes a first constraint relationship, a second constraint relationship and a third constraint relationship; based on the constraint relationship, the pose optimization of the target point cloud data currently selected by sliding is performed to obtain the optimized poses corresponding to the multiple frames of point cloud data, including: When the second edge optimization rule is used for sliding selection, the third constraint relationship is marginalized to obtain a sixth constraint relationship; Integrate the first constraint relationship, the second constraint relationship, the third constraint relationship and the sixth constraint relationship into a constraint matrix; Based on the constraint matrix and the target point cloud data currently selected by sliding, the optimized pose corresponding to the point cloud of the current frame is calculated; Continue sliding the window until it reaches the last frame of point cloud data to obtain the optimized poses corresponding to each frame of point cloud data.

7. A map label generating device, characterized in that: The device comprises: An acquisition module is used to acquire multiple frames of point cloud data collected within a preset time period before the current moment, and to acquire position estimates, image data, map data, and multiple inertial pose estimates obtained at the time of acquisition of each frame of point cloud data; A determination module, for determining, for each frame of point cloud data, a constraint relationship corresponding to the current frame of point cloud data based on a position estimate, image data, map data, and multiple inertial pose estimates corresponding to the current frame of point cloud data; An optimization module, used to slide and select matching target point cloud data in multiple frames of point cloud data using a window of preset size, and for each slide, based on the constraint relationship, perform pose optimization on the target point cloud data currently selected by the slide, so as to obtain the optimized pose corresponding to each of the multiple frames of point cloud data; The mapping module is used to update the map data corresponding to the current moment based on the optimized postures corresponding to the multi-frame point cloud data, and map the map element annotations corresponding to the updated map data to each image data to obtain the map labels corresponding to each image data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images