Ground element labeling method, storage medium, vehicle-mounted equipment and vehicle
By generating mixed layers, combining ground mesh images and fusion point clouds, the problems of low efficiency and poor accuracy of ground element labeling are solved, efficient and automatic ground element labeling are achieved, complex road construction needs are met, and high Real-time and reliability of fine maps.
Patent Information
- Application Number
- CN202410889909.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-04
- Publication Date
- 2024-11-08
AI Technical Summary
In the prior art, the ground element labeling efficiency is low and the accuracy is poor, making it difficult to meet the needs of complex road construction, and the reliance on manual labeling leads to high costs and low update rates, making it difficult to ensure real-time and reliability.
By obtaining the original ground image and point cloud data, ground reconstruction and point cloud fusion are carried out, mixed layers are generated, combined with ground mesh images and fused point clouds, labeling, and using the information in the mixed layer to improve the accuracy of labeling and efficiency.
It realizes more accurate and efficient labeling of ground elements, reduces manual labeling costs, improves the degree of automation of labeling, meets the needs of complex road construction, and enhances the real-time and reliability of high-precision maps.
Smart Images

Figure CN118918182A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent driving technology, and in particular to a ground element labeling method, a storage medium, an on-board device, and a vehicle. Background Art
[0002] In intelligent driving, high-precision maps can provide rich ground element information including road network topology, lane lines, traffic signs and traffic lights, so that the intelligent driving system can achieve more efficient and accurate driving. However, in the production process of high-precision maps, a large number of ground elements need to be manually labeled, and the labels need to be continuously updated and maintained to adapt to various changes in road traffic. As a result, the cost of high-precision maps is generally high, the update rate is low, and it is difficult to popularize and apply them, and it is difficult to ensure real-time and reliability.
[0003] To improve the efficiency of labeling ground elements in high-precision maps, the existing solutions mainly include the following three:
[0004] The first method is to project the input image onto the ground to obtain an AVM (Around View Monitor) image, and then manually mark the ground elements on the AVM. However, in this scheme, since the projected AVM image has a very small range, the range of the marked ground elements is also very small, which is difficult to meet the usage requirements. In addition, this scheme is highly dependent on the accuracy of the camera's internal and external parameters, and the disturbance of the internal and external parameters will affect the marking accuracy.
[0005] The second method is to use the different reflection intensity of ground elements and other locations to directly mark ground elements on the point cloud. However, in this solution, the marking of ground elements is easily affected by environmental interference. For example, changes in ground temperature and humidity, changes in the age of road elements, and changes in light will affect the reflection intensity measured by the sensor. It is necessary to repeatedly adjust the reflection intensity range to see the road elements clearly. The marking efficiency is low, and some road elements formed by special materials have the same or similar reflectivity as vehicles, pedestrians, and obstacles that may appear around them, so they cannot be effectively identified and marked.
[0006] The third method is to directly use the results of image and / or point cloud segmentation as the annotation of ground elements, such as datasets such as NuScenes and Argoverse, or deep learning network models such as HDMapNet, VectorMapNet, MapTR and GeMap trained on NuScenes and Argoverse datasets. Images and / or point clouds are segmented through these datasets and deep learning network models, and then the segmentation results are annotated. However, the quality of this annotation scheme is very dependent on the segmentation results, and it can only be used for simple regular annotations. For example, there are only three categories in the NuScenes dataset: pedestrian crossing, lane divider and road boundary, which cannot meet the needs of complex road mapping. Summary of the invention
[0007] The present application mainly provides a ground element labeling method, storage medium, vehicle-mounted equipment and vehicle, aiming to solve the technical problems of low efficiency and poor accuracy of ground element labeling.
[0008] In order to solve the above technical problems, the technical solution adopted by the present application is: to provide a ground element annotation method. The ground element annotation method includes: obtaining original ground images, original ground point clouds and self-vehicle posture data at different times; reconstructing the ground based on the original ground image and the self-vehicle posture data at the corresponding time to obtain a ground mesh image; fusing point clouds based on the original ground point cloud and the self-vehicle posture data at the corresponding time to obtain a ground fused point cloud; converting the ground mesh image and the ground fused point cloud to a preset coordinate system, and compressing the ground fused point cloud in the preset coordinate system and the ground mesh image in the preset coordinate system to the same horizontal plane to obtain a mixed layer; annotating the ground elements in the mixed layer.
[0009] In some embodiments, the labeling of the ground elements in the mixed layer includes: obtaining the labeling distance range at different times, and projecting the labeling distance range into the mixed layer based on the vehicle posture data at the corresponding time to obtain a labeling area; labeling the ground elements in the labeling area.
[0010] In some embodiments, the labeling of the ground elements in the labeling area includes: labeling the ground mesh image of the ground elements represented by the ground mesh image; labeling the projection points of the ground fusion point cloud of the ground elements that cannot be represented by the ground mesh image; and smoothly transitioning the same ground element at the junction of the projection points of the ground mesh image and the ground fusion point cloud.
[0011] In some embodiments, the original ground image at each moment includes a sequence of surround images from different perspectives; the original ground image, the original ground point cloud and the vehicle posture data correspond one-to-one at each moment.
[0012] In some embodiments, the original ground image also includes corresponding camera parameters; the ground reconstruction based on the original ground image and the ego-vehicle posture data at the corresponding moment to obtain a ground mesh image includes: generating an initialization mesh image, the initialization mesh image including coordinate data and initial color data of multiple grid points; rendering the initialization mesh image to obtain two-dimensional images of multiple perspectives; sampling the grid points to obtain multiple sampling points; based on the ego-vehicle posture data at the corresponding moment, determining the color data true value and semantic segmentation true value of the sampling point at the corresponding position in the original ground image, and based on the vehicle posture data at the corresponding moment and the camera parameters, determining the sampled color data and semantic segmentation result of the corresponding position in the two-dimensional image; calculating the color loss of each of the two-dimensional images based on the sampled color data and the color data true value to obtain a color loss value; calculating the semantic segmentation loss of each of the two-dimensional images based on the semantic segmentation result and the semantic segmentation true value to obtain a segmentation loss value; when the color loss value is less than a first loss threshold and the segmentation loss value is less than a second loss threshold, using the initialization mesh image as the ground mesh image.
[0013] In some embodiments, after obtaining the color loss value and obtaining the segmentation loss value, it also includes: when the color loss value is not less than the first loss threshold, and / or the segmentation loss value is not less than the second loss threshold, optimizing the camera parameters based on the color loss value and the segmentation loss value; updating the initialized mesh image based on the optimized camera parameters.
[0014] In some embodiments, the color loss of each of the two-dimensional images is calculated based on the sampled color data and the true value of the color data to obtain a color loss value, including: based on the formula Calculate the color loss of each sampling point, where L color is the color loss value, C j is the true value of the color data, For the sampled color data, Mask j is the area of the non-ground element under the jth two-dimensional image, and N is the total number of the two-dimensional images under different viewing angles; the semantic segmentation loss of each of the two-dimensional images is calculated based on the semantic segmentation result and the semantic segmentation true value to obtain the segmentation loss value, including: based on the formula Calculate the semantic segmentation loss of each of the two-dimensional images, where L sem is the segmentation loss value, S j is the semantic segmentation truth value, is the semantic segmentation result, For S j and The cross entropy loss between Mask j is the area of the non-ground element in the jth two-dimensional image, and N is the total number of the two-dimensional images under different viewing angles.
[0015] In some embodiments, the point cloud fusion based on the original ground point cloud and the self-vehicle posture data at the corresponding moment to obtain the ground fused point cloud includes: dedistorting each of the original ground point clouds; based on the self-vehicle posture data, converting each of the dedistorted original ground point clouds to the preset coordinate system at the same moment, and performing point cloud stitching to obtain multiple frames of stitched point clouds; removing the stitched point cloud outside a preset range of the original ground point cloud to obtain multiple frames of the ground fused point cloud, wherein the preset range includes at least one of a marked distance range, a preset height range, and a preset reflection intensity range.
[0016] In some embodiments, the ground mesh image and the ground fusion point cloud are converted into a preset coordinate system, and the ground fusion point cloud in the preset coordinate system and the ground mesh image in the preset coordinate system are compressed to the same horizontal plane to obtain a mixed layer, including: converting the ground mesh image and the ground fusion point cloud into the preset coordinate system; compressing the ground fusion point cloud into the horizontal plane of the ground mesh image, and adjusting the reflectivity interval according to the reflection intensity of the ground fusion point cloud to obtain the mixed layer.
[0017] In some embodiments, after labeling the ground elements in the mixed layer, the method further includes: training a target mapping model based on the labeled mixed layer.
[0018] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a storage medium, on which program data is stored, characterized in that when the program data is executed by a processor, the steps of the ground element labeling method as described above are implemented.
[0019] To solve the above technical problems, another technical solution adopted in the present application is: to provide a vehicle-mounted device, which includes a processor and a memory connected to each other, the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the ground element labeling method as mentioned above.
[0020] In order to solve the above technical problem, another technical solution adopted in the present application is: providing a vehicle, which includes the storage medium as mentioned above or the vehicle-mounted device as mentioned above.
[0021] The beneficial effects of the present application are as follows: Different from the prior art, the present application discloses a ground element labeling method, a storage medium, a vehicle-mounted device and a vehicle. The present application processes the original ground image and the original ground point cloud to obtain a hybrid layer that combines the information of the ground mesh image and the ground fusion point cloud, so that the ground elements can be characterized more accurately and easily identified, so as to label the ground elements in the hybrid layer, thereby improving the accuracy and efficiency of ground element labeling. At the same time, the method combines point cloud features with image features, has a wider range of labeling, is less affected by environmental factors, realizes the labeling of multiple ground elements, has strong flexibility, and can meet more complex road mapping needs. In addition, the method can automatically obtain the hybrid layer for labeling without additional labor costs, reduces the workload of manual labeling, is easy to deploy and maintain and update, and is easy to popularize and apply, which is conducive to obtaining more real-time and reliable high-precision maps, thereby improving the effectiveness and safety of intelligent driving. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:
[0023] Figure 1 It is a flowchart of an embodiment of a ground element labeling method provided by the present application;
[0024] Figure 2 yes Figure 1 A schematic diagram of a flow chart of an embodiment of step 20 in the embodiment;
[0025] Figure 3 yes Figure 1 A schematic diagram of a flow chart of an embodiment of step 30 in the embodiment;
[0026] Figure 4 yes Figure 1 A schematic diagram of a flow chart of an embodiment of step 40 in the embodiment;
[0027] Figure 5 yes Figure 1 A schematic diagram of a flow chart of an embodiment of step 50 in the embodiment;
[0028] Figure 6 yes Figure 5 A schematic diagram of a flow chart of an embodiment of step 52 in the embodiment;
[0029] Figure 7 It is a structural schematic diagram of an embodiment of a storage medium provided by the present application;
[0030] Figure 8 It is a structural schematic diagram of an embodiment of the vehicle-mounted device provided by the present application;
[0031] Fig. 9 It is a structural schematic diagram of a vehicle embodiment provided by the present application. DETAILED DESCRIPTION
[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0033] The terms "first", "second", "third" in the embodiments of the present application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the feature defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically limited. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.
[0034] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0035] This application provides a ground element labeling method, see Figure 1 , Figure 1 : is a flow chart of an embodiment of a ground element labeling method provided by the present application, the ground element labeling method comprising:
[0036] Step 10: Obtain the original ground image, original ground point cloud and ego-vehicle pose data at different times.
[0037] The original ground image, that is, the image containing ground elements, can be obtained by a camera, scanner or other image capture device. The original ground image can display the color, texture, shape, spatial relationship and other characteristics of the ground elements, which facilitates the identification or labeling of the corresponding ground elements and provides a data basis for the corresponding image processing and application.
[0038] Optionally, the original ground image also includes corresponding camera parameters. The camera parameters may include parameters such as focal length, principal point coordinates, pixels, distortion coefficients, rotation matrix, and translation vector of the image capture device used to obtain the original ground image. These camera parameters can effectively reflect the state and quality of the image, and accurate control and optimization of the image can be achieved by adjusting the parameters.
[0039] The original ground point cloud is the ground point cloud data collected by lidar, depth camera or other three-dimensional sensors. These data contain the spatial position information of ground elements and can reflect the geometric shape and spatial structure of ground elements.
[0040] The ego vehicle pose data (Pose) refers to the position and attitude data of the autonomous driving vehicle itself, which can be obtained through the vehicle's positioning system and sensors such as the inertial measurement unit, magnetometer and wheel encoder. The ego vehicle pose data is used to represent the precise position coordinates of the vehicle in the set world coordinate system, as well as attitude information such as yaw angle, pitch angle and roll angle. Based on the ego vehicle pose data, the position and attitude of the vehicle when it captures the original ground image and the original ground point cloud can be determined. Combining the image, point cloud and the ego vehicle pose data and the corresponding image coordinate system, point cloud coordinate system and ego vehicle coordinate system, the spatial alignment of the image, point cloud and vehicle can be achieved.
[0041] This step obtains the original ground image, original ground point cloud and self-vehicle posture data at different times. These data at different times can be obtained from pre-collected sample data sets, or they can be continuously collected and recorded by the autonomous driving vehicle during driving, so as to obtain a series of data sets at different times. Among them, for the case of obtaining from pre-collected sample data sets, these sample data sets may contain ground element information and self-vehicle posture data at multiple different times, different locations and different environmental conditions. By analyzing and processing them, more comprehensive and accurate ground element features can be extracted and used for subsequent annotation and map construction. For the case of real-time collection by the autonomous driving vehicle during driving, the latest ground element information and self-vehicle posture data can be obtained in real time, and the ground elements can be annotated and updated based on this information to achieve real-time construction and updating of high-precision maps.
[0042] Optionally, the original ground image at each moment includes a sequence of surround images at different perspectives; the original ground image, the original ground point cloud and the ego-vehicle position and posture data correspond one to one at each moment.
[0043] Surround view image sequences are usually a set of images taken from different angles at the same time by multiple cameras installed around the vehicle, or a single image data obtained by further stitching, or image data directly captured by a panoramic camera or a surround view camera. Surround view image sequences of different perspectives can be used to observe ground elements from different perspectives, so as to obtain richer ground element information. By matching the original ground image, the original ground point cloud and the vehicle posture data one by one at a time, the synchronization of these data in time can be ensured, which facilitates subsequent data processing and annotation work.
[0044] Step 20: Reconstruct the ground based on the original ground image and the vehicle posture data at the corresponding time to obtain a ground mesh image.
[0045] Mesh image is a representation of a three-dimensional model, which shows the surface morphology of an object in a gridded form. In this step, ground reconstruction mainly uses the self-vehicle posture data to spatially align the original ground image, and extracts the geometric morphology and spatial structure of the ground elements by combining the visual features of the image and the three-dimensional spatial features of the mesh image, thereby constructing a three-dimensional model of the ground elements and obtaining a ground mesh image. The ground reconstruction can be implemented based on computer vision and three-dimensional reconstruction technology, or by optimizing the image-related camera parameters through a specific algorithm, or by extracting and matching features of the ground image through methods such as deep learning to achieve more accurate ground reconstruction.
[0046] The reconstructed ground mesh image not only contains the spatial position information of the ground elements, but also reflects the surface morphology and detailed features of the ground elements, providing a more accurate and rich data basis for subsequent element annotation. In addition, since the ground mesh image is obtained from the original ground image, and the original ground image includes multiple frames of images from multiple perspectives, when the original ground image is acquired in real time, the ground mesh image can also be used to observe the real road conditions of all the places where the vehicle has passed. In the context of big data, it can effectively improve the utilization value of the image and reduce the subsequent update cost of the corresponding high-precision map.
[0047] Optionally, see Figure 2 , based on the original ground image and the vehicle posture data at the corresponding time, ground reconstruction is performed to obtain a ground mesh image, which can be performed as follows:
[0048] Step 21: Generate an initialization mesh image, where the initialization mesh image includes coordinate data of multiple grid points and initial color data.
[0049] The initialization mesh image is composed of multiple grid points, each of which contains its coordinate data and initial color data, where the initial color data can be blank data or random data generated according to the data format of the mesh image, or image data obtained based on preset color values or prediction algorithms. The coordinate data is the value in the three-dimensional coordinate system, usually expressed as (x, y, z) coordinates, while the initial color data can be the value in the RGB color system, LAB color system, HSV color system or other color space. The initialization mesh image generated by this step provides a data basis for subsequent and refined ground reconstruction. By optimizing the relevant parameters of the initialization mesh image, the required ground mesh image can be obtained.
[0050] Step 22: Render the initialized mesh image to obtain two-dimensional images of multiple perspectives.
[0051] Rendering is the step of converting the 3D initialized mesh image into a 2D image. Rendering can be implemented through differentiable renderers such as PyTorch3D, OpenDR, and mesh_renderer, or based on a rasterizer, or through a rendering model based on deep learning. The 2D image obtained after rendering can reflect the appearance of the mesh image from different perspectives, and realize multi-angle observation of the mesh image, thereby more comprehensively presenting the characteristic information and data characteristics of the ground elements, and providing a basis for the subsequent calculation of color loss and semantic segmentation loss.
[0052] Step 23: Sampling the grid points to obtain a plurality of sampling points.
[0053] Sampling is the process of randomly or regularly selecting grid points, aiming to select some representative grid points from the entire initialized mesh image and the corresponding two-dimensional image, reducing the amount of data required for subsequent processing. In the sampling process, you can choose a suitable sampling method according to specific needs and conditions, such as random sampling, uniform sampling, sampling based on grid density, etc., to ensure the representativeness and uniformity of the sampling points. This step preferably uses the waypoint sampling method for sampling, wherein the waypoint sampling method is an efficient sampling method based on the principle of equidistant sampling, which can ensure that the sampling points are evenly distributed in the mesh image and have stronger representativeness, avoiding information loss or deviation problems caused by uneven sampling or insufficient number of sampling points.
[0054] Step 24: Based on the vehicle posture data at the corresponding moment, determine the true value of the color data and the true value of the semantic segmentation of the sampling point at the corresponding position in the original ground image, and based on the vehicle posture data and camera parameters at the corresponding moment, determine the sampling color data and semantic segmentation results of the corresponding position in the two-dimensional image.
[0055] The vehicle pose data provides the precise position and orientation information of the vehicle relative to the ground elements in the image, so that the sampling points can be aligned with the original ground image through coordinate transformation mapping, and the sampling points can be mapped to the corresponding pixel positions on the original ground image, so that the color data of these pixel positions can be extracted as the true value of the color data. At the same time, the semantic segmentation true value of these pixel positions can also be obtained through a pre-trained high-precision semantic segmentation model or manual annotation method, so as to distinguish the categories of ground elements, such as lane lines, turn markings, and crosswalks.
[0056] In the two-dimensional image, it is necessary to determine the corresponding positions of the sampling points in the two-dimensional image based on the vehicle posture data and camera parameters, and extract the sampled color data and semantic segmentation results of these positions. First, the sampling points are mapped to the vehicle coordinate system through the vehicle posture data, and then the sampling points mapped to the vehicle coordinate system are mapped to the two-dimensional image according to the coordinate relationship between the vehicle and the corresponding image capture device that can be reflected by the camera parameters. The sampled color data reflects the color characteristics of the initialized mesh image at the corresponding two-dimensional perspective, and the semantic segmentation result is obtained by applying the semantic segmentation algorithm or model to the two-dimensional image, which identifies the semantic category of each sampling point in the image. On this basis, the color loss and semantic segmentation loss can be calculated. The loss value provides a basis for optimizing the mesh image and semantic segmentation model, and through continuous iteration and adjustment, the accuracy and precision of ground reconstruction can be improved.
[0057] Step 25: Calculate the color loss of each two-dimensional image based on the sampled color data and the true value of the color data to obtain a color loss value.
[0058] The color loss value is an indicator to measure the difference between the color data of the sampling point and the real color data. This difference can be quantified by the distance metric in the color space, such as the Euclidean distance, cosine similarity, color value difference in the color space, and a specific loss algorithm. According to the color loss values of all sampling points, a total or average color loss value can be calculated to evaluate the color accuracy of the entire ground mesh image. This total color loss value can be used as an indicator to optimize the ground mesh image. By continuously iterating and adjusting the parameters and structure of the mesh image, the color loss value can be gradually reduced, thereby obtaining a more accurate ground mesh image.
[0059] By calculating the color loss value, the accuracy of the initialized mesh image and its corresponding two-dimensional image can be evaluated. Subsequently, the method of obtaining the color data of the corresponding sampling points can be continuously optimized based on the color loss value, such as optimizing the camera parameters of the image capture device, optimizing the parameters and hyperparameters of the model or algorithm, and optimizing the architecture of the model or algorithm, so that the color loss value is gradually reduced, thereby obtaining a more accurate ground mesh image.
[0060] Optionally, calculating the color loss of each two-dimensional image based on the sampled color data and the true value of the color data to obtain a color loss value includes:
[0061] Based on the formula Calculate the color loss of each sampling point, where L color is the color loss value, C j is the true value of color data, To sample color data, Mask j is the area of non-ground elements in the jth two-dimensional image, and N is the total number of two-dimensional images at different viewing angles;
[0062] The formula first calculates the color difference of the sampling points under each 2D image viewing angle, that is, Then multiply by the corresponding "Mask j ", where the area consisting of non-ground elements is not the ground element of concern in the subsequent annotation, so this part of the area can be given a lower weight when calculating the color loss, so by multiplying the corresponding different "Mask j " value, the impact of these areas on the color loss value can be reduced. Finally, the formula sums the color loss of all sampling points and divides it by the total number of sampling points "N" and the sum of the areas of non-ground elements in all two-dimensional images to obtain the color loss value "L color ”. The color loss value reflects the average level of color accuracy of the entire ground mesh image and can be used as an indicator for optimizing corresponding models, algorithms and related equipment parameters.
[0063] Step 26: Calculate the semantic segmentation loss of each two-dimensional image based on the semantic segmentation result and the semantic segmentation true value to obtain a segmentation loss value.
[0064] Semantic segmentation loss is an important indicator to measure the performance of semantic segmentation models or algorithms. It reflects the difference between the model's prediction of the semantic category of pixels in an image and the actual result. When calculating semantic segmentation loss, commonly used loss algorithms include cross entropy loss, Dice loss, IoU (intersection over union) loss, etc. These loss functions can measure the degree of inconsistency between the predicted results and the actual results, and select different loss functions or their combinations to optimize the performance of the model according to the requirements of specific tasks. For example, cross entropy loss is often used for multi-classification problems, while Dice loss and IoU loss focus more on evaluating the segmentation accuracy of the model at the pixel level.
[0065] For each sampling point, its semantic segmentation result is compared with the corresponding semantic segmentation true value, and the selected loss algorithm is applied to calculate its semantic segmentation loss to obtain the corresponding segmentation loss value. The segmentation loss value can reflect the accuracy of the entire ground mesh image after semantic segmentation, and can be used to evaluate the performance of the semantic segmentation algorithm or model on the ground mesh image, and optimize the corresponding algorithm or model accordingly. For example, the parameters and structure of the algorithm or model can be adjusted, or a more advanced semantic segmentation algorithm or model can be used to reduce the segmentation loss value and improve the recognition accuracy of ground elements, which helps to more accurately extract the information and data characteristics of ground elements and provide strong support for subsequent applications.
[0066] Optionally, the semantic segmentation loss of each two-dimensional image is calculated based on the semantic segmentation result and the semantic segmentation true value to obtain a segmentation loss value, including:
[0067] Based on the formula Calculate the semantic segmentation loss of each two-dimensional image, where L sem is the segmentation loss value, S j is the semantic segmentation truth value, is the semantic segmentation result, For S j and The cross entropy loss between Mask j is the area of non-ground elements in the jth two-dimensional image, and N is the total number of two-dimensional images at different viewing angles.
[0068] In this formula, the cross entropy loss function It measures the semantic segmentation truth value "S j "The prediction results of semantic segmentation with the model or algorithm The difference between the two. Cross entropy loss is often used in classification problems, especially when the output is a probability distribution. In the semantic segmentation task, each pixel is assigned a category label, so this problem can be regarded as a classification problem for each pixel, and the cross entropy loss is used to measure the prediction performance of the model. j " and "N" are the same as "Mask j " and "N" have similar functions and will not be described here. By calculating the segmentation loss value, an average semantic segmentation loss value "L sem ", which reflects the overall level of semantic segmentation accuracy of the entire ground mesh image to evaluate the performance of the corresponding algorithm or model under different perspectives and serves as an indicator for optimizing the corresponding model, algorithm and related equipment parameters.
[0069] Step 27: When the color loss value is less than the first loss threshold and the segmentation loss value is less than the second loss threshold, the initialized mesh image is used as the ground mesh image.
[0070] After obtaining the color loss value and the segmentation loss value as described above, this solution sets a loss threshold to determine whether the prediction result of the model meets the preset accuracy requirements. The setting of the first loss threshold and the second loss threshold is usually based on a comprehensive consideration of multiple factors such as the needs of the actual application, model performance, data characteristics, and practical experience. For example, if the application scenario has high requirements for color accuracy, the first loss threshold can be set relatively low. Similarly, if the accuracy of semantic segmentation is critical to subsequent tasks, a lower second loss threshold can be selected. Compare the calculated color loss value with the first loss threshold, and compare the segmentation loss value with the second loss threshold. Only when the color loss value is less than the first loss threshold and the segmentation loss value is less than the second loss threshold, is it considered that the current initialized mesh image has met the preset accuracy requirements in terms of color and semantic segmentation, and can meet the needs of subsequent applications or analysis and processing.
[0071] Step 28: When the color loss value is not less than the first loss threshold, and / or the segmentation loss value is not less than the second loss threshold, optimize the camera parameters based on the color loss value and the segmentation loss value.
[0072] Step 29: Update the initialized mesh image based on the optimized camera parameters.
[0073] When the color loss value is not less than the first loss threshold, or the segmentation loss value is not less than the second loss threshold, it means that the current initialized mesh image fails to meet the preset accuracy requirements in terms of color or semantic segmentation. Therefore, it is necessary to optimize the parameters of the corresponding model, algorithm or related equipment. The optimization method may include adjusting the architecture of the model, increasing or decreasing the number of network layers, changing the size and number of convolution kernels, adjusting the parameters or hyperparameters of the model, algorithm or related equipment, etc.
[0074] Preferably, the present application uses a gradient descent method to optimize camera parameters.
[0075] Gradient descent is an iterative optimization algorithm that calculates the gradient of the loss function with respect to the camera parameters and updates the parameters in the opposite direction of the gradient to minimize the loss function. When using the gradient descent method to optimize the camera parameters, it is first necessary to calculate the gradient of the loss function with respect to the camera parameters. This usually involves the steps of deriving the loss function and propagating the gradient to the camera parameters according to the chain rule. After that, the camera parameters can be updated according to a certain learning rate based on the calculated gradient. By iterating this process, the value of the loss function can be gradually reduced, thereby optimizing the performance of the corresponding image capture device. During the optimization process, multiple iterations may be required to achieve a satisfactory result. After each iteration, the loss value needs to be recalculated and checked whether it meets the preset threshold conditions. If the loss value still does not meet the requirements, the gradient descent optimization process continues until the preset number of iterations is reached, the loss value converges, or other stop conditions are met.
[0076] Optimizing the camera parameters means adjusting the focal length, principal point coordinates, pixels and distortion coefficients, rotation matrix and translation vector of the collected image, so as to optimize the effectiveness and reliability of the coordinate transformation and projection mapping based on the original mesh image, so as to obtain more accurate two-dimensional images and ground mesh images, improve the quality and accuracy of the ground mesh image, and help to further improve the quality and efficiency of the ground reconstruction process. Based on the optimized camera parameters, more accurate and effective rendering, projection and coordinate transformation steps can be achieved, and then the initialization mesh image can be updated. By adjusting the camera's focal length, principal point coordinates, pixels and distortion coefficients, the color value and semantic category of the pixel in the mesh image can be changed to make it closer to the real scene, and by adjusting the rotation matrix and translation vector, the corresponding original ground image can be better projected into the mesh image and the corresponding two-dimensional image, which is conducive to obtaining a more accurate ground mesh image, which can better reflect the coordinates, color and semantic category of the corresponding sampling points, and provide more valuable data for subsequent applications or analysis. This optimization process can be repeated until the color loss value and the segmentation loss value meet the preset threshold requirements. By continuously optimizing camera parameters and updating mesh images, the quality and accuracy of mesh images can be gradually improved to meet the needs of various intelligent driving tasks.
[0077] Step 30: Perform point cloud fusion based on the original ground point cloud and the vehicle posture data at the corresponding time to obtain a ground fused point cloud.
[0078] Since the acquired ground point cloud data often comes from different perspectives and positions, it is necessary to fuse these point cloud data to generate a complete and accurate ground model. This application performs point cloud fusion on the original ground point cloud and the self-vehicle posture data at the corresponding moment to obtain a ground fused point cloud. Among them, point cloud fusion is used to integrate point cloud data under different perspectives and positions into a unified coordinate system to generate a complete and continuous point cloud data. The implementation of point cloud fusion usually involves algorithms and techniques such as coordinate transformation and data registration, which can be implemented by methods based on deep learning, methods based on feature matching, methods based on grid processing, and methods based on modeling. For example, each point cloud data can be converted from a local coordinate system to a global coordinate system using the self-vehicle posture data and the corresponding rotation matrix and translation vector, so that the point cloud data under different perspectives can be correctly aligned and fused. In the fusion process, factors such as the density and accuracy of the point cloud may also need to be considered to ensure the quality and accuracy of the fused point cloud data.
[0079] After point cloud fusion, the corresponding ground fused point cloud can be obtained. The fused point cloud contains the information of multiple original point cloud data, and has been processed by coordinate transformation and registration, which has higher accuracy and completeness, laying a solid data foundation for the subsequent hybrid layer, which is more effective and reliable.
[0080] Optionally, see Figure 3 , point cloud fusion is performed based on the original ground point cloud and the vehicle posture data at the corresponding time to obtain the ground fused point cloud, which can be performed as follows:
[0081] Step 31: Dedistort each original ground point cloud.
[0082] Step 32: Based on the vehicle posture data, the dedistorted original ground point clouds are converted into a preset coordinate system at the same time, and the point clouds are stitched to obtain a multi-frame stitched point cloud.
[0083] Step 33: remove the stitched point cloud outside the preset range of the original ground point cloud to obtain a multi-frame ground fusion point cloud, wherein the preset range includes at least one of a marked distance range, a preset height range and a preset reflection intensity range.
[0084] When acquiring ground point cloud data, the influence of equipment, environment and other factors may cause distortion in the point cloud data, such as twisting, stretching and compression. These distortions will affect the accuracy and integrity of the point cloud data, so it needs to be dedistorted. The dedistortion processing method can correct and repair the point cloud data based on a known distortion model, or automatically dedistort the point cloud data through methods based on deep learning and machine learning technology, or dedistort the point cloud data through methods based on statistical analysis and filtering technology to eliminate the influence of distortion on the point cloud data, so as to make the point cloud data closer to the real scene.
[0085] Based on the vehicle's posture data, the point cloud data from different perspectives can be aligned to the same coordinate system through coordinate transformation, and the point clouds in the same coordinate system can be spliced. Splicing can be achieved based on feature registration, adjacent search, translation and rotation search, or based on specific models and algorithms. During the splicing process, factors such as the density and accuracy of the point cloud can be considered, and fine adjustments and optimizations can be made to ensure the quality and accuracy of the spliced point cloud data. The spliced point cloud data will contain information from multiple original point clouds and have a unified coordinate system, which is more conducive to obtaining the corresponding ground fusion point cloud.
[0086] The point cloud data obtained after stitching usually includes a large amount of data, and this large amount of data also contains a large amount of redundant point cloud data such as roadside billboards, plants in green belts, pedestrians and vehicles passing by, which cannot be used to obtain subsequent mixed layers, as well as point cloud data that cannot be used in intelligent driving tasks in a short time, such as long-distance road sections, overhead bridges, roads and tunnels below. Therefore, in order to obtain a more streamlined, effective and reliable ground fusion point cloud, it is necessary to remove this part of redundant and unusable point cloud data.
[0087] This application removes this part of the point cloud data by setting a preset range. The preset range can be set according to the actual application requirements. For example, the annotation distance range, preset height range and preset reflection intensity range can be set to screen and filter the spliced point cloud. Among them, the preset distance range can correspond to the distance range involved in the aforementioned original ground image, and be adjusted according to specific scenarios and needs. The preset height range can be set with appropriate upper and lower limits based on the characteristics of the ground point cloud to remove point cloud data that is too high or too low. The preset reflection intensity range can be set according to the reflection characteristics of the point cloud to remove abnormal or irrelevant point cloud data. After these screening and filtering operations, a multi-frame ground fused point cloud can be obtained. The fused point cloud has higher accuracy and completeness, is more streamlined while ensuring the validity and reliability of the data, and has a smaller amount of data, which is convenient for better extraction of the required ground elements, and provides high-quality data support with lower data volume for subsequent intelligent driving tasks.
[0088] Step 40: Convert the ground mesh image and the ground fused point cloud to a preset coordinate system, and compress the ground fused point cloud in the preset coordinate system and the ground mesh image in the preset coordinate system to the same horizontal plane to obtain a mixed layer.
[0089] The preset coordinate system can be any pre-set coordinate system, as long as the corresponding ground mesh image and ground fused point cloud can be converted to the preset coordinate system based on the vehicle posture data and the corresponding image coordinate system and point cloud coordinate system. The preset coordinate system is preferably the coordinate system where the vehicle, image or point cloud is located at a certain moment when the corresponding original ground image and original ground point cloud are collected. It can be the coordinate system corresponding to the middle moment, the starting moment or the last moment, so that the corresponding ground fused point cloud and ground mesh image can fall into the middle part of the coordinate system or the part close to the origin on one side after the coordinate conversion, which is more convenient for corresponding observation and annotation.
[0090] The ground fusion point cloud in the preset coordinate system and the ground mesh image in the preset coordinate system are compressed to the same horizontal plane. This is to eliminate the height difference between the point cloud data and the image data so that the two can be better integrated. During the compression process, a suitable horizontal plane can be determined by calculating the average height, minimum height and majority height of the point cloud data, or the horizontal plane where the preset coordinate origin is located can be directly used as the horizontal plane, and then both the point cloud data and the image data are projected onto the horizontal plane, so as to achieve the superposition and fusion of the ground fusion point cloud and the ground mesh image on the same horizontal plane.
[0091] Finally, the compressed ground fusion point cloud and ground mesh image are mixed to obtain a mixed layer. In the mixing process, different fusion strategies can be adopted considering the different characteristics of point cloud data and image data. For example, the ground mesh image can be adjusted and optimized accordingly according to the density and distribution characteristics of the point cloud data, so that the two are closer and more consistent in color and texture. At the same time, the point cloud data can also be colored and rendered using the color, texture and other information in the image data to make the mixed layer more realistic and vivid.
[0092] The hybrid layer combines the information of the ground mesh image and the ground fused point cloud, and contains both rich color and texture information and precise geometry and location information, providing important data support for subsequent annotation tasks and intelligent driving tasks based on high-precision maps, and making it easier to identify and annotate corresponding ground elements. At the same time, since the hybrid layer is generated in a preset coordinate system, it is also easy to integrate and fuse with other sensor data and map data to achieve more comprehensive environmental perception and decision-making.
[0093] Optionally, see Figure 4 , convert the ground mesh image and the ground fusion point cloud to the preset coordinate system, and compress the ground fusion point cloud in the preset coordinate system and the ground mesh image in the preset coordinate system to the same horizontal plane to obtain a mixed layer, which can be performed as follows:
[0094] Step 41: Convert the ground mesh image and the ground fused point cloud into a preset coordinate system.
[0095] Step 42: compress the ground fusion point cloud into the horizontal plane of the ground mesh image, and adjust the reflectivity range according to the reflection intensity of the ground fusion point cloud to obtain a mixed layer.
[0096] On the basis of the above steps, in order to make the mixed layer more realistic and vivid, the reflectivity interval can also be adjusted according to the reflection intensity of the ground fused point cloud, so that the point cloud data can present richer color and texture information in the mixed layer. In the process of adjusting the reflectivity interval, the range and change rules of the reflectivity can be set according to the actual application requirements and scene characteristics. For example, different reflectivity intervals can be set according to the reflection intensity distribution characteristics of the point cloud data, and different colors or texture information can be assigned to each interval. In this way, the point cloud data in the mixed layer can present different colors and texture effects according to the reflection intensity, thereby more realistically reflecting the actual situation of the ground.
[0097] Step 50: Label the ground elements in the mixed layer.
[0098] After obtaining the mixed layer, the ground elements in the mixed layer can be annotated. When annotating, it can be done manually, that is, manually observing the mixed layer, identifying and annotating the ground elements to ensure the accuracy and precision of the annotation. It can also be done by using automatic annotation algorithms or models, and automatically identifying and annotating the ground elements in the mixed layer through computer vision and machine learning technologies to improve the efficiency of annotation and reduce manual intervention and costs. Regardless of the annotation method used, the ground elements need to be classified and described in detail during the annotation process. For example, the ground elements can be divided into different types such as lanes, crosswalks, and turn signs, and corresponding labels and attributes can be set for each type. At the same time, the geometric information such as the position, size, and shape of the ground elements, as well as the visual information such as color and texture, can also be recorded to facilitate the subsequent accurate identification and positioning of the ground elements.
[0099] Labeling is the process of identifying and classifying ground elements. Through labeling, detailed information about ground elements can be provided to autonomous vehicles. After labeling, the labeling information can be associated and stored with the hybrid layer. In this way, in subsequent autonomous driving vehicle tasks, ground elements can be identified and responded to based on the labeling information, improving the perception and decision-making level of autonomous driving vehicles. At the same time, the labeling information can also be used to build and update high-precision maps, providing more powerful data support for intelligent driving, which is conducive to improving the effectiveness and safety of intelligent driving.
[0100] Optionally, see Figure 5 To mark the ground elements in the mixed layer, follow the steps below:
[0101] Step 51: Obtain the annotated distance range at different times, and project the annotated distance range into the mixed layer based on the vehicle posture data at the corresponding time to obtain the annotated area.
[0102] Step 52: Mark the ground elements within the marked area.
[0103] During the driving process of an autonomous vehicle, the position and posture of the vehicle will constantly change, so it is necessary to obtain the annotation distance range at different times to ensure that the annotated area always matches the current position and viewing angle of the vehicle. The annotation distance range can be set according to actual application requirements and scene characteristics. For example, it can be set to an area within a certain range in front of the vehicle, or the annotation distance range can be dynamically adjusted according to the vehicle's driving speed and direction.
[0104] Different annotation distance ranges can be set for different annotation tasks. For example, when the application of the annotation results involves driving tasks such as autonomous driving decision-making and path planning, a larger annotation distance range can be set, such as 0.5km, 1km, 2km, etc., to cover a wider range and better enable the vehicle to make decisions on subsequent roads. When the application of the annotation results involves driving tasks such as lane selection, parking space selection and road obstacle avoidance, a smaller annotation distance can be set, such as 10m, 30m, 100m, etc., to reduce the amount of annotated data and reduce the complexity of the corresponding model, algorithm or equipment parameter optimization, thereby reducing costs and improving efficiency.
[0105] After obtaining the annotation distance range, it can be projected into the hybrid layer to obtain the annotation area. During the projection process, the vehicle posture data at the corresponding moment needs to be used to convert the annotation distance range from the vehicle coordinate system to the preset coordinate system where the hybrid layer is located. In this way, the annotation area can accurately reflect the current position and viewing angle of the vehicle in the hybrid layer, which facilitates the subsequent annotation work.
[0106] In the annotation area, the ground elements in the annotation area can be annotated by manual or automatic annotation methods. During the annotation process, appropriate annotation methods and tools can be used to annotate according to the type, size, shape and other characteristics of the ground elements. For example, for continuous ground elements such as roads and sidewalks, line segments or curves can be used for annotation; for discrete ground elements such as road turn signs and roadblocks, rectangular boxes or circular boxes can be used for annotation. At the same time, the attribute information of the ground elements, such as type, location, size, etc., can also be recorded to facilitate the subsequent identification and positioning of the ground elements. After the annotation is completed, the annotation information can be associated and stored with the mixed layer for subsequent use and query.
[0107] Through the above steps, the ground elements in the mixed layer can be annotated. The annotation results can provide detailed information about ground elements for autonomous vehicles, helping vehicles to better understand and cope with complex road environments. At the same time, the annotation information can also be used to build and update high-precision maps, providing more accurate and comprehensive environmental information for autonomous vehicles, which is conducive to obtaining more real-time and reliable high-precision maps, thereby improving the effectiveness and safety of intelligent driving.
[0108] Optionally, see Figure 6 To mark the ground elements in the marked area, follow the steps below:
[0109] Step 521: labeling the ground mesh image of the ground elements represented by the ground mesh image.
[0110] Step 522: Annotate the projection points of the ground fusion point cloud of the ground elements that cannot be represented by the ground mesh image.
[0111] Step 523: smoothly transition and connect the same ground element at the intersection of the projection point of the ground mesh image and the ground fusion point cloud.
[0112] In a mixed layer, ground elements may be presented in various forms, some of which can be represented by ground mesh images, while others cannot be fully presented by ground mesh images alone due to occlusion, height changes, etc. Therefore, different annotation methods need to be used for different situations when annotating.
[0113] For ground elements that can be represented by ground mesh images, the ground mesh images can be annotated directly. This usually involves identifying, classifying, and labeling pixels or areas in the image to represent different types of ground elements. During the annotation process, professional image processing software or tools can be used to accurately annotate according to the characteristics of the ground elements, such as using different colors or labels to represent different ground types.
[0114] However, for ground elements that cannot be represented by the ground mesh image alone, such as parts that cannot be clearly presented in the mesh image due to occlusion or height change, it is necessary to combine the projection points of the ground fused point cloud for annotation. The ground fused point cloud contains rich three-dimensional information, which can make up for the shortcomings of the ground mesh image in representing ground elements. By projecting the ground fused point cloud onto a two-dimensional plane, the point cloud projection points corresponding to the ground mesh image can be obtained, and annotations can be made on these projection points. In this way, the information of ground elements can be obtained more comprehensively, and the accuracy and completeness of annotations can be improved.
[0115] However, since the ground mesh image and the ground fused point cloud may have differences or overlaps in representing ground elements, there may be discontinuities or jumps at their junctions. Therefore, when annotating the ground elements in the ground mesh image and the projection points of the ground fused point cloud corresponding to the ground elements, it is also necessary to smooth the junctions to ensure the continuity and consistency of the annotation results. The smoothing process can be achieved by interpolating, fusing or smoothing the pixels or point cloud projection points in the boundary area. In manual annotation, the same ground element can be connected by a smooth curve, or the same ground element or related ground elements can be annotated with the same label or related labels, so that the annotation results are more natural and accurate.
[0116] Through the above annotation steps, comprehensive, accurate and continuous annotation of ground elements in the mixed layer can be achieved. This annotation method combines the advantages of ground mesh images and ground fused point clouds, and can make full use of the information of both to improve the quality and efficiency of annotation. The annotation results can provide more abundant and accurate environmental information for autonomous driving vehicles, which helps vehicles better understand and cope with complex road environments. At the same time, the annotation information can also be used to build and update high-precision maps, providing more powerful data support for intelligent driving tasks.
[0117] Optionally, after the ground elements in the mixed layer are labeled, the method further includes: training a target mapping model based on the labeled mixed layer.
[0118] After the ground elements in the hybrid layer are labeled, the hybrid layer can be used to train the target mapping model. The target mapping model is a learning model used to generate or update high-precision maps. It can learn and construct detailed information such as the geometry of the road, traffic rules, and obstacle locations based on inputs such as sensor data and environmental information. By training the target mapping model, it can be given stronger recognition and understanding capabilities to cope with various complex road environments and driving scenarios. During the training process, the labeled hybrid layer is used as the input data of the model. By setting a suitable loss function and optimization algorithm, the model can learn the characteristics, classification, and location of the ground elements, and automatically perform labeling and high-precision map construction. By training the target mapping model, the automatic recognition and labeling of the ground elements in the hybrid layer can be realized, thereby greatly reducing the workload and time cost of manual labeling. The trained model can also be used to generate and update high-precision maps in real time, provide real-time and accurate environmental information for autonomous driving vehicles, and improve the safety and reliability of vehicles.
[0119] See also Figure 7 , Figure 7 It is a structural diagram of an embodiment of a storage medium provided by the present application.
[0120] The storage medium 60 stores program data 61. When the program data 61 is executed by the processor, the following is achieved: Figures 1 to 6 Described method for labeling ground elements.
[0121] The program data 61 is stored in a storage medium 60, and includes a number of instructions for enabling a network device (such as a router, a personal computer, a server, or other network device) or a processor to execute all or part of the steps of the method described in each embodiment of the present application.
[0122] Optionally, the storage medium 60 may be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, or other medium that can store the program data 61 .
[0123] See also Figure 8 , Figure 8 It is a structural schematic diagram of an embodiment of the vehicle-mounted device provided by the present application.
[0124] The vehicle-mounted device 70 includes a processor 72 and a memory 71 connected to each other. The memory 71 stores a computer program. When the processor 72 executes the computer program, the following is achieved: Figures 1 to 6 The ground element labeling method described herein. The memory 71 may include the storage medium 60, or may be other independently developed memory.
[0125] See also Fig. 9 , Fig. 9 It is a structural schematic diagram of a vehicle embodiment provided by the present application.
[0126] The vehicle 80 includes Figure 8 The vehicle 80 may be a gasoline vehicle, an electric vehicle, or a hybrid vehicle, etc., and the present application does not impose any specific restrictions on this. In addition, the vehicle 80 may be equipped with a vehicle-mounted device 70 having a storage medium 60, or may be equipped with a vehicle-mounted device 70 without a storage medium 60, or the vehicle 80 may be a type of vehicle that can directly read the storage medium 60.
[0127] Different from the prior art, the present application discloses a ground element labeling method, storage medium, vehicle-mounted equipment and vehicle. By processing the original ground image and the original ground point cloud to obtain a hybrid layer that combines the information of the ground mesh image and the ground fused point cloud, the ground elements can be characterized more accurately and easily identified, so as to label the ground elements in the hybrid layer, thereby improving the accuracy and efficiency of ground element labeling. At the same time, the method combines point cloud features with image features, has a wider range of labeling, is less affected by environmental factors, and can achieve the labeling of multiple ground elements. It has strong flexibility and can meet more complex road mapping needs. In addition, the method can automatically obtain the hybrid layer for labeling without additional labor costs, reduces the workload of manual labeling, is easy to deploy and maintain and update, and is easy to popularize and apply, which is conducive to obtaining more real-time and reliable high-precision maps, thereby improving the effectiveness and safety of intelligent driving.
[0128] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium embodiment, the vehicle-mounted device embodiment and the vehicle embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0129] The present application can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.
[0130] In the several embodiments provided in this application, it should be understood that the disclosed methods, storage media, vehicle-mounted devices, and vehicles can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
[0131] The above description is only an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.
Claims
1. A ground element labeling method, It is characterized in that include: Obtain original ground images, original ground point clouds and vehicle posture data at different times; Reconstructing the ground based on the original ground image and the vehicle posture data at the corresponding moment to obtain a ground mesh image; Performing point cloud fusion based on the original ground point cloud and the vehicle posture data at the corresponding time to obtain a ground fused point cloud; Converting the ground mesh image and the ground fused point cloud to a preset coordinate system, and compressing the ground fused point cloud in the preset coordinate system and the ground mesh image in the preset coordinate system to the same horizontal plane to obtain a mixed layer; The ground elements in the mixed layer are labeled.
2. The ground element labeling method according to claim 1, It is characterized in that The marking of the ground elements in the mixed layer includes: Acquire the annotated distance range at different times, and project the annotated distance range into the mixed layer based on the vehicle posture data at the corresponding time to obtain the annotated area; The ground elements within the marked area are marked.
3. According to the ground element labeling method of claim 2, It is characterized in that The marking of the ground elements in the marking area includes: annotating the ground mesh image of ground elements represented by the ground mesh image; Annotating projection points of the ground fused point cloud of ground elements that cannot be represented by the ground mesh image; A smooth transition connection is performed on the same ground element at the intersection of the projection point of the ground mesh image and the ground fusion point cloud.
4. The ground element labeling method according to claim 1, It is characterized in that The original ground image at each moment includes a sequence of surround images at different viewing angles; the original ground image, the original ground point cloud and the vehicle posture data correspond one to one at each moment.
5. The ground element labeling method according to claim 1, It is characterized in that The original ground image also includes corresponding camera parameters; The ground reconstruction based on the original ground image and the vehicle posture data at the corresponding moment to obtain a ground mesh image includes: Generate an initialization mesh image, wherein the initialization mesh image includes coordinate data of a plurality of grid points and initial color data; Rendering the initialized mesh image to obtain two-dimensional images of multiple viewing angles; Sampling the grid points to obtain a plurality of sampling points; Based on the vehicle posture data at the corresponding moment, determine the color data true value and semantic segmentation true value of the sampling point at the corresponding position in the original ground image, and based on the vehicle posture data at the corresponding moment and the camera parameters, determine the sampling color data and semantic segmentation result of the corresponding position in the two-dimensional image; Calculating the color loss of each of the two-dimensional images based on the sampled color data and the true value of the color data to obtain a color loss value; Calculating the semantic segmentation loss of each of the two-dimensional images based on the semantic segmentation result and the semantic segmentation true value to obtain a segmentation loss value; When the color loss value is less than a first loss threshold and the segmentation loss value is less than a second loss threshold, the initialized mesh image is used as the ground mesh image.
6. The ground element labeling method according to claim 5, It is characterized in that After obtaining the color loss value and the segmentation loss value, the method further includes: When the color loss value is not less than the first loss threshold, and / or the segmentation loss value is not less than the second loss threshold, optimizing the camera parameters based on the color loss value and the segmentation loss value; The initialization mesh image is updated based on the optimized camera parameters.
7. The ground element labeling method according to claim 5, It is characterized in that The calculating the color loss of each of the two-dimensional images based on the sampled color data and the true value of the color data to obtain a color loss value includes: Based on the formula Calculate the color loss of each sampling point, where L color is the color loss value, C j is the true value of the color data, For the sampled color data, Mask j is the area of the non-ground element in the j-th two-dimensional image, and N is the total number of the two-dimensional images under different viewing angles; The calculating the semantic segmentation loss of each of the two-dimensional images based on the semantic segmentation result and the semantic segmentation true value to obtain a segmentation loss value includes: Based on the formula Calculate the semantic segmentation loss of each of the two-dimensional images, where L sem is the segmentation loss value, S j is the semantic segmentation truth value, is the semantic segmentation result, For S j and The cross entropy loss between Mask j is the area of the non-ground element in the jth two-dimensional image, and N is the total number of the two-dimensional images under different viewing angles.
8. The ground element labeling method according to claim 1, It is characterized in that The step of performing point cloud fusion based on the original ground point cloud and the vehicle posture data at a corresponding time to obtain a ground fused point cloud includes: Performing dedistortion processing on each of the original ground point clouds; Based on the vehicle posture data, each of the original ground point clouds after dedistortion is converted into the preset coordinate system at the same time, and point clouds are stitched to obtain a multi-frame stitched point cloud; The stitched point cloud outside a preset range of the original ground point cloud is removed to obtain multiple frames of the ground fused point cloud, wherein the preset range includes at least one of a marked distance range, a preset height range, and a preset reflection intensity range.
9. The ground element labeling method according to claim 1, It is characterized in that The converting the ground mesh image and the ground fused point cloud into a preset coordinate system, and compressing the ground fused point cloud in the preset coordinate system and the ground mesh image in the preset coordinate system onto the same horizontal plane to obtain a mixed layer, includes: Converting the ground mesh image and the ground fused point cloud into the preset coordinate system; The ground fusion point cloud is compressed into the horizontal plane of the ground mesh image, and the reflectivity interval is adjusted according to the reflection intensity of the ground fusion point cloud to obtain the mixed layer.
10. The ground element labeling method according to claim 1, It is characterized in that After marking the ground elements in the mixed layer, the method further includes: The target mapping model is trained based on the labeled mixed layer.
11. A storage medium having program data stored thereon, It is characterized in that When the program data is executed by a processor, the steps of the ground element labeling method according to any one of claims 1 to 10 are implemented.
12. A vehicle-mounted device, It is characterized in that The method comprises a processor and a memory connected to each other, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the ground element labeling method according to any one of claims 1 to 10 are implemented.
13. A vehicle, It is characterized in that The vehicle includes the storage medium according to claim 11 or the vehicle-mounted device according to claim 12.
Citation Information
Cited By
3D data labeling method for intelligent robot with body
CN119495099A
Lightweight method and system for 3D asset model
CN119762701A
