Road static truth value generation method and device, storage medium and electronic equipment

Abstract

The invention discloses a road static truth value generation method and device, a storage medium and electronic equipment. The method comprises the steps that a three-dimensional point cloud map obtained by a true value vehicle is projected to a two-dimensional plane, a two-dimensional intensity map and a two-dimensional elevation map are obtained, the value of each pixel in the two-dimensional intensity map is used for indicating intensity information of a corresponding position, and the value of each pixel in the two-dimensional elevation map is used for indicating height information of the corresponding position; marking the two-dimensional intensity diagram to obtain a mark of at least one road static element, and back-projecting the mark of the road static element to a three-dimensional point cloud map to obtain a three-dimensional true value, the mark of the road static element being used for indicating a point set and a category corresponding to the road static element; and converting the three-dimensional truth value into a camera image according to the camera parameter to obtain a two-dimensional truth value. According to the invention, the technical problem of low efficiency of a road static truth value generation method provided by the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and more specifically, to a method and device for generating a static true value of a road, a storage medium, and an electronic device. Background Art

[0002] In intelligent driving systems, the perception system is the basis for subsequent decision-making and control, including detection, classification, positioning and tracking of surrounding objects. The evaluation of perception algorithms is a key step in evaluating algorithm performance and ensuring algorithm security and reliability. The evaluation process relies on true values, and the evaluation results of the model are obtained by comparing the difference between the true value and the model prediction value.

[0003] When generating the true value of the static elements of the road for the evaluation model, the related art usually uses a dynamic matching algorithm to determine the shape and position of the two-dimensional marking elements in the three-dimensional virtual feature map to generate the annotated three-dimensional feature map. However, the process of annotating point clouds in three-dimensional space is relatively cumbersome and requires more resources. In other words, the method for generating the static true value of the road provided by the related art has the problem of low efficiency.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a method and device for generating a static truth value of a road, a storage medium, and an electronic device, so as to at least solve the technical problem that the method for generating a static truth value of a road provided by the related art has low efficiency.

[0006] According to one aspect of an embodiment of the present application, a method for generating a static truth value of a road is provided, comprising: projecting a three-dimensional point cloud map acquired by a truth value vehicle onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional elevation map, wherein the value of each pixel in the two-dimensional intensity map is used to indicate the intensity information of the corresponding position, and the value of each pixel in the two-dimensional elevation map is used to indicate the height information of the corresponding position; annotating the two-dimensional intensity map to obtain an annotation of at least one static element of the road, and back-projecting the annotation of the static element of the road into the three-dimensional point cloud map to obtain a three-dimensional truth value, wherein the annotation of the static element of the road is used to indicate the point set and category corresponding to the static element of the road; converting the three-dimensional truth value into a camera image according to camera parameters to obtain a two-dimensional truth value, wherein the camera image and the three-dimensional point cloud map are data collected at the same time.

[0007] According to another aspect of an embodiment of the present application, a device for generating a static truth value of a road is also provided, including: a two-dimensional plane generation unit, used to project a three-dimensional point cloud map obtained by a truth value vehicle onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional elevation map, wherein the value of each pixel in the two-dimensional intensity map is used to indicate the intensity information of the corresponding position, and the value of each pixel in the two-dimensional elevation map is used to indicate the height information of the corresponding position; a three-dimensional truth value generation unit, used to annotate the two-dimensional intensity map to obtain an annotation of at least one static element of the road, and back-project the annotation of the static element of the road into the three-dimensional point cloud map to obtain a three-dimensional truth value, wherein the annotation of the static element of the road is used to indicate the point set and category corresponding to the static element of the road; a two-dimensional truth value generation unit, used to convert the three-dimensional truth value into a camera image according to camera parameters to obtain a two-dimensional truth value, wherein the camera image and the three-dimensional point cloud map are data collected at the same time.

[0008] Optionally, the above-mentioned two-dimensional plane generation unit includes: performing the following operations on any point cloud on the three-dimensional point cloud map: a two-dimensional horizontal coordinate determination module, used to determine the difference between the horizontal coordinate of the point cloud and the horizontal coordinate of the minimum coordinate point cloud, and determine the ratio between the difference between the horizontal coordinates and the scaling factor as the horizontal coordinate of the corresponding two-dimensional pixel point; a two-dimensional vertical coordinate determination module, used to determine the difference between the vertical coordinate of the point cloud and the vertical coordinate of the minimum coordinate point cloud, and determine the ratio between the difference between the vertical coordinates and the scaling factor as the vertical coordinate of the corresponding two-dimensional pixel point; an information determination module, used to determine the intensity information and height information corresponding to the two-dimensional pixel point; a two-dimensional plane generation module, used to generate a two-dimensional intensity map based on the horizontal coordinates, vertical coordinates and intensity information of all two-dimensional pixel points, and generate a two-dimensional elevation map based on the horizontal coordinates, vertical coordinates and height information of all two-dimensional pixel points.

[0009] Optionally, the above-mentioned information determination module is also used to: determine the three-dimensional point cloud set corresponding to the two-dimensional pixel point according to the zoom ratio, and determine the reference intensity information set and the reference height information set corresponding to the three-dimensional point cloud set; calculate the intensity standard deviation and the intensity mean corresponding to the reference intensity information set, and the height standard deviation and the height mean corresponding to the reference height information set; filter the reference intensity information set according to the intensity standard deviation and the intensity mean to obtain the target intensity information set, and filter the reference height information set according to the height standard deviation and the height mean to obtain the target height information set; determine the average value of the target intensity information set as the intensity information corresponding to the two-dimensional pixel point, and determine the average value of the target height information set as the height information corresponding to the two-dimensional pixel point.

[0010] Optionally, the above-mentioned three-dimensional truth value generation unit includes: a splitting module, which is used to split the two-dimensional intensity map to obtain multiple two-dimensional intensity sub-maps; a labeling module, which is used to label the two-dimensional intensity sub-maps using a labeling model to obtain the labeling of at least one road static element; a merging module, which is used to merge the labeling of the road static elements corresponding to all two-dimensional intensity sub-maps to obtain the labeling of all road static elements; a three-dimensional truth value generation module, which is used to back-project the labeling of all road static elements into a three-dimensional point cloud map to obtain a three-dimensional truth value.

[0011] Optionally, the above-mentioned annotation module is also used to: perform feature extraction on the two-dimensional intensity sub-map to obtain a feature map; input the feature map into a linear element annotation model to obtain an annotation of at least one first road static element, wherein the first road static element is used to indicate the linear static element of the road; input the feature map into an icon element annotation model to obtain an annotation of at least one second road static element, wherein the second road static element is used to indicate the icon static element of the road; merge the annotation of the first road static element and the annotation of the second road static element to obtain the annotation result corresponding to the two-dimensional intensity sub-map.

[0012] Optionally, the above-mentioned three-dimensional truth value generation module is also used to: perform the following operations on the pixel points in the point set corresponding to the static elements of the road: determine the abscissa of the corresponding three-dimensional point cloud by the product of the horizontal coordinate corresponding to the pixel point and the zoom factor and the sum of the horizontal coordinate of the minimum coordinate point cloud; determine the vertical coordinate of the corresponding three-dimensional point cloud by the product of the vertical coordinate corresponding to the pixel point and the zoom factor and the sum of the vertical coordinate of the minimum coordinate point cloud; determine the height information corresponding to the pixel point according to the two-dimensional elevation map, and determine the height information as the z-axis coordinate of the corresponding three-dimensional point cloud; determine the annotation of the corresponding three-dimensional point cloud according to the annotation of the static elements of the road corresponding to the pixel point.

[0013] Optionally, the above-mentioned road static truth generation device also includes: a point cloud map generation unit, which is used to extract features from the point cloud data obtained by the true value vehicle and remove the point cloud data corresponding to dynamic obstacles; perform pre-integration processing and forward optimization on the true value vehicle to obtain the posture change of the true value vehicle; and generate a three-dimensional point cloud map based on the posture change of the true value vehicle and the point cloud data.

[0014] Optionally, the above-mentioned road static true value generation device also includes: a two-dimensional evaluation unit, used to obtain the two-dimensional prediction value, and draw the two-dimensional predicted lane lines and the two-dimensional real lane lines according to the two-dimensional prediction value and the two-dimensional true value respectively; calculate the intersection and union ratio between each pair of two-dimensional predicted lane lines and the two-dimensional real lane lines, and remove the two-dimensional predicted lane lines and two-dimensional real lane lines corresponding to the intersection and union ratio less than the two-dimensional intersection and union ratio threshold; re-match the remaining two-dimensional predicted lane lines and the two-dimensional real lane lines to obtain a new matching relationship; and obtain the two-dimensional evaluation result of the prediction model according to the new matching relationship between the two-dimensional predicted lane lines and the two-dimensional real lane lines.

[0015] Optionally, the above-mentioned road static true value generation device also includes: a three-dimensional evaluation unit, used to obtain three-dimensional prediction values, and perform linear sampling according to the three-dimensional prediction values ​​and the three-dimensional true values ​​according to a preset segmented distance to obtain point cloud pairs; calculate the distance between the three-dimensional prediction values ​​and the three-dimensional true values ​​between the point cloud pairs, and remove the three-dimensional prediction values ​​and three-dimensional true values ​​corresponding to the distance greater than the distance threshold; re-match the remaining three-dimensional prediction values ​​and three-dimensional true values ​​to obtain a new matching relationship; and obtain the three-dimensional evaluation result of the prediction model based on the new matching relationship between the three-dimensional prediction values ​​and the three-dimensional true values.

[0016] According to another aspect of the embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method for generating a static true value of a road when running.

[0017] According to another aspect of the embodiment of the present application, a computer program product is provided, the computer program product comprising a computer program / instruction, the computer instruction being stored in a computer-readable storage medium. A processor of a computer device reads the computer program / instruction from the computer-readable storage medium, and the processor executes the computer program / instruction, so that the computer device executes the above method for generating a static truth value of a road.

[0018] According to another aspect of an embodiment of the present application, there is also provided an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the method for generating a static true value of a road through the computer program.

[0019] In an embodiment of the present application, the three-dimensional point cloud map obtained by the truth value vehicle is projected onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional elevation map, wherein the value of each pixel in the two-dimensional intensity map is used to indicate the intensity information of the corresponding position, and the value of each pixel in the two-dimensional elevation map is used to indicate the height information of the corresponding position; the two-dimensional intensity map is annotated to obtain the annotation of at least one road static element, and the annotation of the road static element is back-projected into the three-dimensional point cloud map to obtain a three-dimensional truth value, wherein the annotation of the road static element is used to indicate the point set and category corresponding to the road static element; the three-dimensional truth value is converted into a camera image according to the camera parameters to obtain a two-dimensional truth value, wherein the camera image and the three-dimensional point cloud map are data collected at the same time. The three-dimensional point cloud map can be projected onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional height map, and annotated on the two-dimensional intensity map, which reduces the difficulty of annotation, can automatically and efficiently realize static element recognition, and can naturally obtain a two-dimensional truth value by transforming the three-dimensional truth value through a coordinate system, which is simple and efficient, and improves the annotation efficiency of static truth value data. The above method solves the technical problem that the method for generating road static truth value provided by the related art has low efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0021] Figure 1 is a schematic diagram of a hardware environment of an optional method for generating a static true value of a road according to an embodiment of the present application;

[0022] Figure 2 is a flow chart of an optional method for generating a static true value of a road according to an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of an optional two-dimensional intensity map according to an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of an optional two-dimensional elevation map according to an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of the effect of an optional point cloud screening method according to an embodiment of the present application;

[0026] Figure 6 is a flow chart of an optional two-dimensional intensity map annotation method according to an embodiment of the present application;

[0027] Figure 7 is a schematic diagram of an optional annotation result of a two-dimensional intensity map according to an embodiment of the present application;

[0028] Figure 8 is a schematic diagram of an optional three-dimensional truth value according to an embodiment of the present application;

[0029] Fig. 9 is a schematic diagram of an optional two-dimensional truth value according to an embodiment of the present application;

[0030] Fig.10 is a schematic diagram of an optional three-dimensional point cloud data according to an embodiment of the present application;

[0031] Fig.11 is a schematic diagram of an optional point cloud map according to an embodiment of the present application;

[0032] Fig.12 is a flowchart of another optional method for generating a static true value of a road according to an embodiment of the present application;

[0033] Fig.13 is a schematic diagram of an optional device for generating a static true value of a road according to an embodiment of the present application;

[0034] Fig.14 is a schematic diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] According to one aspect of the embodiments of the present application, a method for generating a static truth value of a road is provided. As an optional implementation, the method for generating a static truth value of a road can be applied to, but is not limited to, Figure 1 The system for generating the road static truth value in the hardware environment shown. Optionally, the above method for generating the road static truth value can be applied to a vehicle terminal. Figure 1 A side view of a vehicle terminal 101 is shown, which is disposed on a travel surface 113 and is capable of traversing the travel surface 113. The vehicle terminal 101 includes an onboard navigation system 103, a computer-readable storage device or medium (memory) 102 including a digitized road map 104, a spatial monitoring system 117, a vehicle controller 109, a GPS (global positioning system) sensor 110, an HMI (human / machine interface) device 111, and also includes an autonomous controller 112 and a telematics controller 114. The vehicle terminal 101 may include, but is not limited to, commercial vehicles, industrial vehicles, agricultural vehicles, passenger vehicles, all-terrain vehicles, personal mobile devices, robots, and similar forms of mobile platforms to achieve the purpose of the present application.

[0038] In one embodiment, the spatial monitoring system 117 includes: one or more spatial sensors and systems arranged to monitor the visible area 105 in front of the vehicle terminal 101; and a spatial monitoring controller 118. The spatial sensors used to monitor the visible area 105 include, for example, a lidar sensor 106, a radar sensor 107, a camera 108, and the like. The placement of the spatial sensors allows the spatial monitoring controller 118 to monitor traffic flow, including approaching vehicles, intersections, lane markings, and other objects around the vehicle terminal 101. The spatial sensors of the spatial monitoring system 117 may include object positioning sensing devices. The lidar sensor 106 uses pulsed and reflected laser beams to measure the range or distance to an object. The radar sensor 107 uses radio waves to determine the range, angle, and / or speed of an object. The camera 108 includes an image sensor, a lens, and a camera controller.

[0039] The camera 108 is advantageously mounted and positioned on the vehicle terminal 101 in a position that allows capturing images of the visible area 105, wherein at least a portion of the visible area 105 includes a portion of the travel surface 113 in front of the vehicle terminal 101 and including the trajectory of the vehicle terminal 101. The visible area 105 may also include the surrounding environment. Other cameras (not shown) may also be used, for example, including a second camera disposed on the rear portion or side portion of the vehicle terminal 101 to monitor the rear of the vehicle terminal 101 and one of the right or left sides of the vehicle terminal 101. The camera 108 can collect road information to generate a road image sequence (such as a camera image) during the driving of the vehicle terminal 101.

[0040] The autonomous controller 112 is configured to implement autonomous driving or advanced driver assistance system (ADAS) vehicle functionality. Such functionality may include a vehicle onboard control system capable of providing a certain level of driving automation. Driving automation may include a series of dynamic driving and vehicle operations. Driving automation may include simultaneous automatic control of vehicle driving functions (including steering, acceleration and braking), wherein the driver abandons control of the vehicle for a period of time during the trip. Driving automation may include simultaneous automatic control of vehicle driving functions (including steering, acceleration and braking), wherein the driver abandons control of the vehicle terminal 101 for the entire trip. Driving automation includes hardware and a controller, which is configured to monitor the spatial environment in various driving modes to perform various driving tasks during dynamic vehicle operations. By way of non-limiting example, autonomous vehicle functions include adaptive cruise control (ACC) operations, lane guidance and lane keeping operations, lane change operations, steering assistance operations, object avoidance operations, parking assistance operations, vehicle braking operations, vehicle speed and acceleration operations, vehicle lateral motion operations, for example, as part of lane guidance, lane keeping and lane change operations, etc.

[0041] The autonomous controller can be equipped with an operating system and an automatic driving system. Among them, the operating system can be responsible for the management of the hardware resources (including sensors, system buses, networks, etc.) of the vehicle terminal 101 and the scheduling of computing resources. The automatic driving system can implement various algorithms required for automatic driving, including positioning, environmental perception, path planning and control, etc., and can make decisions in situations such as driving on curves, driving in a straight line, driving on complex roads, changing lanes, etc. The autonomous controller can perform step S1 when the vehicle is driving: obtaining a three-dimensional point cloud map and camera images at consecutive moments.

[0042] The vehicle terminal 101 may include a telematics controller 114 including a wireless telematics communication system capable of performing off-vehicle communications (including communicating with a communication network 115 having wireless and wired communication capabilities). Alternatively or additionally, the telematics controller 114 directly performs off-vehicle communications by communicating with an off-board server 116 via the communication network 115. The server side can execute steps S2-S4: project the three-dimensional point cloud map obtained by the truth value vehicle onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional elevation map, wherein the value of each pixel in the two-dimensional intensity map is used to indicate the intensity information of the corresponding position, and the value of each pixel in the two-dimensional elevation map is used to indicate the height information of the corresponding position; annotate the two-dimensional intensity map to obtain an annotation of at least one static element of the road, and back-project the annotation of the static element of the road into the three-dimensional point cloud map to obtain a three-dimensional truth value, wherein the annotation of the static element of the road is used to indicate the point set and category corresponding to the static element of the road; convert the three-dimensional truth value into the camera image according to the camera parameters to obtain a two-dimensional truth value, wherein the camera image and the three-dimensional point cloud map are data collected at the same time.

[0043] In an optional embodiment, the vehicle terminal 101 can collect road images in real time using sensors such as cameras 108 through the space monitoring system 117. The road images can contain surrounding road condition data, such as movement information of surrounding vehicles, pedestrians and other traffic participants, and location information of static obstacles such as roadblocks and road fences. The collected road image sequence can be stored in the memory 102. Since the storage space of the memory 102 is limited, based on storage efficiency considerations, images with higher priority can be stored in the memory 102, and a large amount of low-priority data can be stored in the non-record server 116 via the communication network 115 using the telematics controller 114.

[0044] In an optional embodiment, the method for generating the static truth value of the road of the present application can be applied to the quantitative evaluation of the model, and the truth value of the static elements of the road of the evaluation model can be generated, and data collection and data annotation can be automatically realized. The three-dimensional point cloud map and camera image obtained by the truth value vehicle can be annotated with the three-dimensional truth value and the two-dimensional truth value. The three-dimensional point cloud map obtained by the truth value vehicle is projected onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional elevation map, and the two-dimensional intensity map is annotated to obtain the annotation of the static elements of the road, and the two-dimensional map is back-projected into the three-dimensional point cloud map to obtain the three-dimensional truth value, and the three-dimensional truth value is converted into the camera image according to the camera parameters to obtain the two-dimensional truth value. Then, the obtained three-dimensional truth value and two-dimensional truth value are used to evaluate the model to be evaluated to determine the performance of the model.

[0045] In an embodiment of the present application, the three-dimensional point cloud map obtained by the truth value vehicle is projected onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional elevation map, wherein the value of each pixel in the two-dimensional intensity map is used to indicate the intensity information of the corresponding position, and the value of each pixel in the two-dimensional elevation map is used to indicate the height information of the corresponding position; the two-dimensional intensity map is annotated to obtain the annotation of at least one road static element, and the annotation of the road static element is back-projected into the three-dimensional point cloud map to obtain a three-dimensional truth value, wherein the annotation of the road static element is used to indicate the point set and category corresponding to the road static element; the three-dimensional truth value is converted into a camera image according to the camera parameters to obtain a two-dimensional truth value, wherein the camera image and the three-dimensional point cloud map are data collected at the same time. The three-dimensional point cloud map can be projected onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional height map, and annotated on the two-dimensional intensity map, which reduces the difficulty of annotation, can automatically and efficiently realize static element recognition, and can naturally obtain a two-dimensional truth value by transforming the three-dimensional truth value through a coordinate system, which is simple and efficient, and improves the annotation efficiency of static truth value data. The above method solves the technical problem that the method for generating road static truth value provided by the related art has low efficiency.

[0046] In an optional embodiment, if Figure 2 As shown, the method for generating the static true value of the road comprises the following steps:

[0047] S202, projecting the three-dimensional point cloud map obtained by the truth vehicle onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional elevation map, wherein the value of each pixel in the two-dimensional intensity map is used to indicate the intensity information of the corresponding position, and the value of each pixel in the two-dimensional elevation map is used to indicate the height information of the corresponding position;

[0048] It should be noted that the Ground Truth Vehicle refers to a vehicle that is specifically used to collect environmental data and generate "ground truth" (i.e., an accurate record of the actual environmental status) in the field of autonomous driving or intelligent vehicle technology. It is equipped with high-precision sensors such as LiDAR, cameras, GNSS (Global Navigation Satellite System), IMU (Inertial Measurement Unit), etc. These sensors work together to obtain detailed information about the surrounding environment, including but not limited to the three-dimensional structure of the road, lane lines, traffic signs, pedestrians, vehicles, etc. The Ground Truth Vehicle can collect accurate data sets to verify and optimize the perception system of autonomous driving vehicles. By comparing the perception results of autonomous driving vehicles with the data collected by the Ground Truth Vehicle, researchers can accurately evaluate the perception performance of autonomous driving vehicles, identify errors and deviations, and adjust and optimize the algorithm accordingly.

[0049] It should be noted that a 3D point cloud map is a collection of points collected by a lidar or other 3D scanning device in the environment. Each point has three coordinates (x, y, z), representing the position in space. These point cloud data can form a 3D model of the environment, including the shape and position information of roads, buildings, vegetation, etc. The 3D point cloud map provides the geometric structure of the environment, which is essential for vehicle positioning, environmental perception, and path planning. Through point cloud data, vehicles can identify and understand the details of the complex environment around them, such as road edges, traffic signs, obstacles, etc., so as to make safer and more accurate decisions.

[0050] It should be noted that the 2D Intensity Map is a projection of the LiDAR point cloud data on a 2D plane, and uses the average intensity value at each position as the grayscale value of that position. The LiDAR emits laser pulses and receives reflections. The values ​​on the intensity map reflect the intensity of the reflected laser, which can indicate the surface characteristics and position of the object. For example, road markings, vehicles, buildings, etc. will have different intensity values ​​in the 2D intensity map.

[0051] It should be noted that the 2D Elevation Map is a two-dimensional representation of height information extracted from 3D point cloud data, which records the height of the ground at different locations. The value of each 2D pixel represents the average height at the actual spatial position corresponding to that point, usually the vertical distance from the earth's surface to that point. The 2D elevation map provides information about the ground terrain, which helps vehicles understand the ups and downs and slopes of the road. The elevation map is also used to create a more accurate road network model, especially when dealing with complex terrain changes, such as hills, bridges, tunnels, etc., to ensure that the autonomous driving algorithm can correctly understand and respond to these environmental changes.

[0052] In an optional implementation, generating a 2D intensity map and a 2D elevation map from the 3D point cloud data collected by the ground truth vehicle is a key step in training and evaluating the environment perception model in autonomous driving technology. It simplifies the 3D complexity to 2D, which is convenient for algorithm understanding and processing, while retaining key information in the environment, providing a solid foundation for optimizing the autonomous driving system. The projection process will be described in detail later, so it will not be repeated here.

[0053] S204, annotating the two-dimensional intensity map to obtain an annotation of at least one road static element, and back-projecting the annotation of the road static element into the three-dimensional point cloud map to obtain a three-dimensional true value, wherein the annotation of the road static element is used to indicate a point set and a category corresponding to the road static element;

[0054] It should be noted that static road elements refer to objects and features that do not move or change in the road environment, and constitute an important part of the road infrastructure in a certain area. These elements may include but are not limited to: Lane lines: including solid lines, dashed lines, dashed and solid lines, etc., used to divide lanes, which are the basis for vehicle positioning and path planning. Curb: The edge of the road, which helps vehicles identify the road boundary and prevent them from deviating from the lane. Traffic signs: such as speed limit signs, warning signs, and direction signs, which are used to convey traffic rules and information to drivers. Pedestrian crossings: White lines across the road, indicating areas for pedestrians to cross the street. Stop lines: Usually located at intersections or the entrances to parking areas, indicating that vehicles should stop before this line. Pavement text: Such as "Stop", "Give way", etc., directly mark traffic rules with text on the road surface. Road texture: Such as the texture of paving materials, although not directly used for navigation, it can provide additional environmental information in visual recognition.

[0055] In an optional embodiment, the labeling of static road elements refers to the process of accurately marking the location, shape and category of static road elements in the collected data (such as images, point cloud data). Labeling can be achieved by the model, or it can be initially labeled by an automatic algorithm and then manually reviewed and corrected.

[0056] In an optional embodiment, the labeling process may include: determining the boundaries of each static element: using polygons, straight lines or curves to label the shape of the element, such as the continuous or discontinuous characteristics of a lane line. Classifying static elements: assigning a category to each element, such as a lane line, traffic sign, sidewalk, etc. Recording element attributes: such as the line type of the lane line (solid line, dashed line), the specific meaning of the traffic sign, etc. The specific labeling process will be described in detail later and will not be repeated here. Labeling provides standard reference data, i.e., true value data, for subsequent model training and performance evaluation.

[0057] It should be noted that 2D Ground Truth is the result of labeling static elements of the road in a 2D image space, marking the position, shape and category of static elements in the camera image. 2D Ground Truth is usually used when processing image data from a camera. 2D Ground Truth data is a performance criterion for evaluating perception models in image recognition tasks, such as the accuracy of detecting lane lines or traffic signs.

[0058] It should be noted that 3D Ground Truth is the precise annotation of static road elements in three-dimensional space, which contains the position information of static elements in the three-dimensional coordinate system. It is the annotation result used when processing data from three-dimensional sensors such as lidar. 3D ground truth data is essential for evaluating the performance of autonomous driving perception systems in three-dimensional space, such as the accuracy and stability of detecting and tracking three-dimensional objects. In autonomous driving technology, the generation and application of 3D ground truth data is a key step to ensure that vehicles can safely and accurately understand the surrounding environment.

[0059] S206, converting the three-dimensional true value into the camera image according to the camera parameters to obtain the two-dimensional true value, wherein the camera image and the three-dimensional point cloud map are data collected at the same time.

[0060] In an optional embodiment, the process of converting three-dimensional true value data into a camera image captured at the same time to obtain a two-dimensional true value is intended to map the annotation information in the three-dimensional space to the two-dimensional image, thereby evaluating the performance of the model in the image recognition task.

[0061] It should be noted that the camera parameters include internal and external parameters, which are used to describe the properties of the camera and its position and orientation in three-dimensional space. The internal parameters mainly include focal length (f_x, f_y), image center (u_0, v_0) and pixel size, which describe the characteristics of the camera sensor itself. The external parameters describe the position and attitude of the camera relative to the world coordinate system, including rotation and translation information. Camera parameters are crucial for converting three-dimensional coordinates into two-dimensional image coordinates.

[0062] In an optional implementation, the three-dimensional truth value is converted into the camera image, and the camera parameters need to be used for coordinate conversion. The specific process is as follows: Determine the relative position and posture of the camera and the three-dimensional point cloud map data, and ensure that the two are collected at the same time to facilitate accurate coordinate conversion. Apply the camera external parameter matrix (K2) to convert the static elements of the road in the three-dimensional truth value from the world coordinate system to the camera coordinate system. Then use the camera internal parameter matrix (K1) to project the converted three-dimensional coordinates onto the two-dimensional image coordinates to generate a two-dimensional projection of the element. In this process, the coordinates of each marked point need to be converted to ensure that important information is not lost during the conversion process. During the conversion process, the camera's field of view and resolution can also be taken into account to determine which three-dimensional elements can be projected into a specific camera image.

[0063] In an optional embodiment, through the above conversion process, the two-dimensional position information of the static elements of the road in the camera image, that is, the two-dimensional true value, can be obtained. The two-dimensional true value contains the two-dimensional position, shape and category information of the static elements of the road corresponding to the original three-dimensional true value. The three-dimensional true value can be the point set and annotation corresponding to the static elements of the road. Therefore, the conversion process is to convert the point cloud contained in the static elements of the road into the camera image, rather than converting the entire three-dimensional point cloud map into the camera image.

[0064] After determining the 3D and 2D ground truth, the ground truth can be used to evaluate image-based perception models, such as lane detection models based on convolutional neural networks (CNNs). By comparing the difference between the model's predicted output and the ground truth, various performance metrics such as accuracy, recall, F1 score, etc. can be calculated to quantify the performance of the model.

[0065] In an embodiment of the present application, the three-dimensional point cloud map obtained by the truth value vehicle is projected onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional elevation map, wherein the value of each pixel in the two-dimensional intensity map is used to indicate the intensity information of the corresponding position, and the value of each pixel in the two-dimensional elevation map is used to indicate the height information of the corresponding position; the two-dimensional intensity map is annotated to obtain the annotation of at least one road static element, and the annotation of the road static element is back-projected into the three-dimensional point cloud map to obtain a three-dimensional truth value, wherein the annotation of the road static element is used to indicate the point set and category corresponding to the road static element; the three-dimensional truth value is converted into a camera image according to the camera parameters to obtain a two-dimensional truth value, wherein the camera image and the three-dimensional point cloud map are data collected at the same time. The three-dimensional point cloud map can be projected onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional height map, and annotated on the two-dimensional intensity map, which reduces the difficulty of annotation, can automatically and efficiently realize static element recognition, and can naturally obtain a two-dimensional truth value by transforming the three-dimensional truth value through a coordinate system, which is simple and efficient, and improves the annotation efficiency of static truth value data. The above method solves the technical problem that the method for generating road static truth value provided by the related art has low efficiency.

[0066] In an optional embodiment, the above-mentioned three-dimensional point cloud map obtained by the truth vehicle is projected onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional elevation map, including: performing the following operations on any point cloud on the three-dimensional point cloud map: determining the difference between the horizontal coordinate of the point cloud and the horizontal coordinate of the minimum coordinate point cloud, and determining the ratio between the difference between the horizontal coordinates and the zoom factor as the horizontal coordinate of the corresponding two-dimensional pixel point; determining the difference between the vertical coordinate of the point cloud and the vertical coordinate of the minimum coordinate point cloud, and determining the ratio between the difference between the vertical coordinates and the zoom factor as the vertical coordinate of the corresponding two-dimensional pixel point; determining the intensity information and height information corresponding to the two-dimensional pixel point; generating a two-dimensional intensity map based on the horizontal coordinates, vertical coordinates and intensity information of all two-dimensional pixel points, and generating a two-dimensional elevation map based on the horizontal coordinates, vertical coordinates and height information of all two-dimensional pixel points.

[0067] It should be noted that the three-dimensional point cloud map obtained by the truth vehicle is projected onto a two-dimensional plane to generate a two-dimensional intensity map and a two-dimensional elevation map, which can convert the information of the three-dimensional space into a two-dimensional image, facilitating subsequent image processing and feature extraction.

[0068] In an optional embodiment, a two-dimensional coordinate system can be determined to represent the projection of the point cloud data on a two-dimensional plane. The minimum coordinate point cloud of the three-dimensional point cloud map is selected as the origin (upper left corner) of the two-dimensional coordinate system. Usually, the minimum coordinate is the minimum x and y coordinate value of all points in the three-dimensional point cloud map. This provides a starting point for subsequent coordinate conversion.

[0069] For any point on the 3D point cloud map (denoted as point), its coordinates can be converted to a 2D plane by the following steps: Calculate the difference between the x-coordinate of point and the x-coordinate of the minimum coordinate point cloud, and then divide this difference by the scaling factor (resolution) to get the horizontal coordinate of the corresponding 2D pixel. Similarly, calculate the difference between the y-coordinate of point and the y-coordinate of the minimum coordinate point cloud, and divide this difference by the scaling factor to get the vertical coordinate of the corresponding 2D pixel.

[0070] The formula is:

[0071] Horizontal coordinate: x = int((point[0]-min_coord[0]) / resolution),

[0072] Vertical coordinate: y = int((point[1]-min_coord[1]) / resolution)

[0073] Among them, (point[0], point[1]) are the horizontal and vertical coordinates of the point cloud in three-dimensional space, (min_coord[0], min_coord[1]) are the horizontal and vertical coordinates of the point cloud with the minimum coordinates in the three-dimensional point cloud map, resolution is the defined zoom factor, and (x, y) are the pixel coordinates on the two-dimensional plane.

[0074] In an optional embodiment, the two-dimensional intensity map records the intensity information of each pixel on the two-dimensional plane, reflecting the density and reflection intensity of the point cloud at that position. The intensity information can be determined by calculating the average reflection intensity of all point clouds at that position. Specifically, for each pixel, if there is a corresponding point cloud in the three-dimensional point cloud map, the gray value (intensity) of the point is the average reflection intensity of these point clouds, and then normalized to ensure that the intensity value distribution is within a reasonable range, which is convenient for image processing.

[0075] Figure 3 is a schematic diagram of an optional two-dimensional intensity map according to an embodiment of the present application; Figure 3 As shown, it can be a two-dimensional image obtained by projecting a three-dimensional point cloud map. It is not an image formed in the camera coordinate system, but is projected based on the coordinate system where the three-dimensional point cloud map is located. The intensity information of the corresponding point cloud data area can be indicated by the intensity of each point in the image.

[0076] In an optional embodiment, the two-dimensional elevation map records the height information corresponding to each pixel on the two-dimensional plane. The height information can be determined by calculating the average height of all point clouds at that location, but in order to eliminate possible outliers (for example, abnormal height values ​​caused by dynamic objects or sensor noise), a screening method of positive and negative double standard deviations is usually used, that is, those point clouds whose height values ​​are within the range of ±2 times the standard deviation of the average value are retained, and then the average height of these point clouds is calculated as the height information of the pixel. The two-dimensional elevation map generated in this way can more accurately reflect the height changes of the ground.

[0077] Figure 4 is a schematic diagram of an optional two-dimensional elevation map according to an embodiment of the present application; Figure 4 As shown in , it can be a color two-dimensional image, because it indicates the height information of the corresponding three-dimensional point cloud data area. Therefore, for a two-dimensional elevation map of a structured road, from an intuitive visual point of view, it can be seen that the height information is stable. Figure 4 As shown, the horizontal and vertical coordinates can indicate the position information of the pixel, and different colors can indicate the height information corresponding to the pixel.

[0078] In an optional embodiment, through the above steps, for each point cloud in the three-dimensional point cloud map, the corresponding pixel coordinates, intensity information and height information on the two-dimensional plane are determined. Next, this information is aggregated into a two-dimensional image to generate the final two-dimensional intensity map and two-dimensional elevation map. The two-dimensional intensity map can be expanded into a three-channel image to facilitate input into a common deep learning model for processing, while the elevation map remains as a grayscale image to provide height data.

[0079] Through the above implementation of the present application, complex three-dimensional environmental information can be converted into two-dimensional images, which is convenient for the understanding and processing of visual algorithms. The two-dimensional intensity map can be used to identify static elements of the road, such as lane lines, traffic signs, etc., while the two-dimensional elevation map is crucial for understanding the terrain and road conditions, such as slopes, potholes, etc., in preparation for subsequent back projection.

[0080] In an optional embodiment, the above-mentioned determination of the intensity information and height information corresponding to the two-dimensional pixel point includes: determining the three-dimensional point cloud set corresponding to the two-dimensional pixel point according to the zoom ratio, and determining the reference intensity information set and the reference height information set corresponding to the three-dimensional point cloud set; calculating the intensity standard deviation and the intensity mean corresponding to the reference intensity information set, and the height standard deviation and the height mean corresponding to the reference height information set; filtering the reference intensity information set according to the intensity standard deviation and the intensity mean to obtain a target intensity information set, and filtering the reference height information set according to the height standard deviation and the height mean to obtain a target height information set; determining the average value of the target intensity information set as the intensity information corresponding to the two-dimensional pixel point, and determining the average value of the target height information set as the height information corresponding to the two-dimensional pixel point.

[0081] It should be noted that there may be dynamic obstacles such as vehicles on the road that hinder the recognition of static road elements. Therefore, in the process of generating a two-dimensional image, the obstacles can be screened out by screening the point cloud. Optionally, the specific implementation method is as follows:

[0082] Determine the 3D point cloud set corresponding to the 2D pixel point; when the 3D point cloud map is projected onto a 2D plane, each 2D pixel point actually corresponds to an area in the 3D point cloud map, and the corresponding area can be determined according to the zoom factor. Through the determined area, all point clouds within this area can be extracted from the 3D point cloud map, and these point clouds constitute the 3D point cloud set corresponding to the 2D pixel point.

[0083] In an optional embodiment, after determining the multiple point cloud data corresponding to the two-dimensional pixel point, a reference intensity information set and a reference height information set can be determined. The intensity value (reflection intensity) of each point cloud can be obtained from the extracted three-dimensional point cloud set, and these intensity values ​​constitute the reference intensity information set. The z coordinate (height) of each point cloud is obtained from the extracted three-dimensional point cloud set, and these height values ​​constitute the reference height information set.

[0084] Different from the calculation process above, after determining the information set, the corresponding standard deviation and mean need to be calculated. For example, to calculate the intensity standard deviation and intensity mean: calculate the mean (average intensity) and standard deviation (the degree of dispersion of intensity) for all intensity values ​​in the reference intensity information set. To calculate the height standard deviation and height mean: calculate the mean (average height) and standard deviation (the degree of dispersion of height) for all height values ​​in the reference height information set.

[0085] In an optional embodiment, after calculating the standard deviation and mean, interference objects such as static vehicles, underground points that reflect light in rainy days, and various static objects on the roadside can be removed, and the intensity information and height information can be filtered separately. The reference intensity information set can be filtered by the calculated intensity mean and standard deviation. Specifically, point clouds whose intensity values ​​are within the range of ±2 times the standard deviation of the mean are retained, and these point clouds constitute the target intensity information set. The reference height information set can be filtered by the calculated height mean and standard deviation. Similar to the processing of intensity information,

[0086] Then the average value of the target intensity information set is calculated as the intensity information of the two-dimensional pixel point. This can reduce the impact of noise and ensure the accuracy of the intensity information. Standardize and then copy it into a three-channel image. At the same time, the average value of the target height information set can be calculated as the height information of the two-dimensional pixel point. This can reduce abnormal height values ​​caused by dynamic objects or sensor noise and ensure the accuracy of height information.

[0087] Figure 5 is a schematic diagram of the effect of an optional point cloud screening method according to an embodiment of the present application; Figure 5 As shown in (a), it can be a two-dimensional intensity map obtained by directly projecting the three-dimensional point cloud data in the three-dimensional point cloud map. It can be seen that there are still static interference objects 502 (such as Figure 5 (a) The white circle in the figure). Figure 5 As shown in (b), it can be a two-dimensional intensity map obtained after screening. It can be seen that Figure 5 Compared with (a), the static interferer 502 at the position selected by the white circle has been screened out.

[0088] In an optional embodiment, in the process of generating the 2D intensity map and the elevation map, some locations may not have direct 3D point cloud data due to lack of point cloud coverage or occlusion by dynamic targets, etc. In order to obtain intensity information and elevation information at these locations, a distance range search method may be used.

[0089] The specific steps are as follows: Starting from the current two-dimensional pixel position, search for intensity values ​​at fixed distance intervals outward. The initial search range can be set to (0,2), which means searching within a range of 2 pixel units around the current pixel. If the intensity value is not found within the initial search range, expand the search range, such as increasing it to (0,4), that is, searching within a range of 4 pixel units around the current pixel. If the intensity value is still not found, continue to expand the search range, such as (0,16), and so on, until a pixel containing an intensity value is found. By gradually expanding the search range in this way, the intensity information of the position adjacent to or close to the current pixel can be found within a certain range to fill the missing information of the current pixel and ensure the integrity of the two-dimensional image.

[0090] Once the pixels containing intensity values ​​are found within the search range, the intensity values ​​need to be filtered to remove possible outliers or abnormal values. Outliers are usually caused by sensor noise, dynamic target occlusion, or errors in the point cloud data, which will interfere with the accuracy of the intensity information and height information. The filtering process is as follows: For the set of intensity values ​​found, the average value (mean) and standard deviation of the intensity values ​​are calculated. Then, the set of intensity values ​​is filtered to retain those points whose intensity values ​​are within ±2 times the standard deviation of the average value. For the height values, the same method is used for filtering as for the intensity values, that is, those points within the range of ±2 times the standard deviation of the height of the average height are retained. After the filtering is completed, the average values ​​are calculated for the retained intensity values ​​and height values ​​respectively. The average value represents the typical intensity and height characteristics of the point cloud data within the search range, eliminating the influence of outliers, thereby providing more accurate information for pixels that currently have no direct point cloud data. The average value is regarded as the intensity information and height information of the current pixel.

[0091] Through the above steps, even in areas where 3D point cloud data is sparse or missing, relatively accurate 2D intensity and elevation maps can be generated, ensuring the consistency of information and the integrity of the image. This is crucial for model training and performance evaluation in autonomous driving systems, as data integrity directly affects the model's ability to understand and predict the environment.

[0092] Through the above-mentioned implementation mode of the present application, the information in the three-dimensional point cloud map can be accurately projected onto a two-dimensional plane, and a two-dimensional intensity map and a two-dimensional elevation map can be generated. This process ensures that the generated two-dimensional intensity map and two-dimensional elevation map can accurately reflect the information in the three-dimensional point cloud map, and provides a high-quality data foundation for subsequent image processing and model training. When generating a two-dimensional intensity map and an elevation map, for pixels that do not directly correspond to point cloud data, the intensity information of adjacent positions is found by gradually expanding the search range, and outliers are eliminated by a screening method of positive and negative twice the standard deviation, and finally the average value is calculated to fill in the missing information. This process ensures the integrity of the image and the consistency of the information, and is very effective in processing missing or abnormal situations in point cloud data, providing more reliable data support for autonomous driving systems.

[0093] In an optional embodiment, the above-mentioned annotation of the two-dimensional intensity map is performed to obtain the annotation of at least one road static element, and the annotation of the road static element is back-projected into the three-dimensional point cloud map to obtain the three-dimensional truth value, including: splitting the two-dimensional intensity map to obtain multiple two-dimensional intensity sub-maps; annotating the two-dimensional intensity sub-maps using the annotation model to obtain the annotation of at least one road static element; merging the annotations of the road static elements corresponding to all the two-dimensional intensity sub-maps to obtain the annotations of all the road static elements; back-projecting the annotations of all the road static elements into the three-dimensional point cloud map to obtain the three-dimensional truth value.

[0094] It should be noted that after extracting and annotating the static elements of the road from the two-dimensional intensity map, these annotation information can be back-projected into the three-dimensional point cloud map to generate three-dimensional true value data.

[0095] In an optional embodiment, the two-dimensional intensity map can be split first. The two-dimensional intensity map usually contains a bird's-eye view image of the entire acquisition scene and contains a lot of information. In order to improve the efficiency and accuracy of annotation, the two-dimensional intensity map can be split into multiple sub-maps (two-dimensional intensity sub-maps). Each sub-map covers a part of the scene, so that more focus can be placed on the annotation of local features, which is also conducive to parallel processing and annotation using deep learning models. In an example, a 1k (1000 pixel) sub-map can be cut out along the trajectory from the two-dimensional intensity map.

[0096] It should be noted that the annotation model is used for annotation. For each split two-dimensional intensity sub-image, the annotation model is applied for identification and annotation. The annotation model can be a trained deep learning model, such as a model based on a convolutional neural network (CNN) or a Transformer, which is specifically used to detect and identify static road elements, such as lane lines, sidewalks, traffic signs, arrows, etc. The model outputs the annotation result of at least one static road element, including the location, shape and category information of the element.

[0097] In an optional implementation, after all 2D intensity sub-maps are processed by the annotation model, the annotation information on these sub-maps needs to be merged to generate the annotation result of the entire scene. The merging process needs to consider the overlapping areas between the sub-maps to ensure the consistency and integrity of the annotation information. The final result is the annotation of all static elements of the road on the entire 2D intensity map. Finally, the annotation information on the 2D intensity map is back-projected into the 3D point cloud map to generate 3D true value data.

[0098] Through the above-mentioned implementation of the present application, static elements of the road can be accurately extracted and annotated from the two-dimensional intensity map, and these annotation results can be converted back into three-dimensional coordinates, providing the autonomous driving system with more refined and detailed environmental modeling and evaluation data.

[0099] In an optional embodiment, the above-mentioned use of the annotation model to annotate the two-dimensional intensity sub-map to obtain the annotation of at least one road static element includes: extracting features from the two-dimensional intensity sub-map to obtain a feature map; inputting the feature map into a linear element annotation model to obtain the annotation of at least one first road static element, wherein the first road static element is used to indicate the linear static element of the road; inputting the feature map into an icon element annotation model to obtain the annotation of at least one second road static element, wherein the second road static element is used to indicate the icon static element of the road; merging the annotation of the first road static element and the annotation of the second road static element to obtain the annotation result corresponding to the two-dimensional intensity sub-map.

[0100] Figure 6 is a flow chart of an optional method for labeling a two-dimensional intensity map according to an embodiment of the present application; in the process of labeling the two-dimensional intensity map, the following can be used: Figure 6 The steps shown are executed, step S602, splitting the two-dimensional intensity map into two-dimensional intensity sub-maps; then executing step S604, extracting features from the two-dimensional intensity sub-maps to obtain feature maps; then executing step S606-1, inputting the feature map into the linear element annotation model to obtain the annotation of the first road static element; synchronously or asynchronously executing step S606-2, inputting the feature map into the icon element annotation model to obtain the annotation of the second road static element. Finally, executing step S608, merging the annotation of the first road static element and the annotation of the second road static element, and merging the annotation results of multiple two-dimensional intensity sub-maps to obtain the annotation result of the two-dimensional intensity map.

[0101] In an optional implementation, ResNet50 can be used as a feature extractor and Vision Transformer (ViT) can be used as an element annotation model to process the two-dimensional intensity sub-map to identify and annotate road static elements. The specific steps can be as follows:

[0102] Feature extraction of 2D intensity sub-maps to obtain feature maps: At this stage, ResNet50 can be used as a feature extractor for 2D intensity sub-maps. ResNet50 is a model based on a deep residual network, which can effectively extract multi-level features of images, including low-level texture and edge information, as well as high-level shape and structure information. When processing 2D intensity sub-maps, ResNet50 divides the image into multiple regions, and gradually builds a higher-level representation of the image through multiple convolutional layers and residual blocks. Finally, the feature map output by ResNet50 is a compressed and abstract image feature representation, which contains the potential features of the static elements of the road in the sub-map.

[0103] The feature map is input into the linear element annotation model to obtain the annotation of at least one first road static element: The linear element annotation model is a model based on ViT, which is used to identify and annotate road linear static elements (such as lane lines, curbs, stop lines). As a model with a Transformer architecture, ViT is good at processing sequence data. It can regard each position in the feature map as part of the sequence. Through the multi-head self-attention mechanism (Multi-Head Self-Attention), the model can learn the dependencies between different positions in the feature map, and can capture global context information even when processing local feature maps. The linear element annotation model outputs the annotation information of at least one first road static element, which usually includes the attributes of the linear element such as category, position, direction, line width and line type, and is described in the form of a set of ordered coordinate points.

[0104] The feature map is input into the icon element annotation model to obtain the annotation of at least one second road static element: The icon element annotation model is also a ViT-based model that focuses on identifying road icon static elements (such as sidewalks, arrows, ground text, traffic signs, etc.). Similarly, ViT's multi-head self-attention mechanism can capture the relationship between different positions in the feature map, which is crucial for identifying complex icon elements. The icon element annotation model outputs the annotation information of at least one second road static element, usually expressed as a set of polygon vertex coordinates and category information, which is used to describe the shape and position of the icon element.

[0105] The annotations of the first road static element and the second road static element are merged to obtain the annotation results corresponding to the two-dimensional intensity sub-map: Finally, the annotation results of the line elements and the icon elements are merged to generate the complete annotation information of all road static elements on the two-dimensional intensity sub-map. This merging process ensures the comprehensiveness and consistency of the annotations, so that all road static elements on each sub-map can be accurately identified and annotated. Usually, this process involves sorting and organizing the coordinate points, and possible post-processing steps, such as removing overlapping annotations or correcting annotation boundaries.

[0106] Figure 7 is a schematic diagram of an optional annotation result of a two-dimensional intensity map according to an embodiment of the present application; Figure 7 As shown, after the labeling is completed, there may be indicated pixel points on the road static elements of the two-dimensional intensity map, which may indicate the labeling result.

[0107] Through the above implementation of the present application, the powerful feature extraction capability of ResNet50 and the flexibility and global understanding capability of ViT in processing sequence data are utilized. By extracting features from the two-dimensional intensity sub-graphs and then inputting them into the linear element annotation model and the icon element annotation model respectively, efficient and accurate annotation of static road elements is achieved. The final annotation result contains detailed information of all static elements in the image, providing important data support for subsequent road scene analysis and autonomous driving system performance evaluation.

[0108] In an optional embodiment, the above-mentioned back-projection of the annotations of all static road elements into the three-dimensional point cloud map to obtain the three-dimensional true value includes: performing the following operations on the pixel points in the point set corresponding to the static road elements: determining the abscissa of the corresponding three-dimensional point cloud by the product of the horizontal coordinate corresponding to the pixel point and the zoom factor and the sum of the horizontal coordinate of the minimum coordinate point cloud; determining the vertical coordinate of the corresponding three-dimensional point cloud by the product of the vertical coordinate corresponding to the pixel point and the zoom factor and the sum of the vertical coordinate of the minimum coordinate point cloud; determining the height information corresponding to the pixel point according to the two-dimensional elevation map, and determining the height information as the z-axis coordinate of the corresponding three-dimensional point cloud; determining the annotation of the corresponding three-dimensional point cloud according to the annotation of the static road element corresponding to the pixel point.

[0109] It should be noted that the road static element information on the annotated two-dimensional image can be accurately converted back into a three-dimensional point cloud map to generate three-dimensional true value data.

[0110] As mentioned earlier, each pixel on the two-dimensional image represents an area in the three-dimensional point cloud map. In order to convert these pixels back to three-dimensional coordinates, we first need to determine their horizontal coordinates (X) and vertical coordinates (Y) in three-dimensional space.

[0111] It should be noted that, as mentioned above, in a 3D point cloud map, there is usually a minimum coordinate point cloud (min_coord) as a reference. The coordinates of this point cloud can be used as the origin for subsequent coordinate conversion. In data processing, the scaling factor (resolution) is used to scale the 3D point cloud coordinates to the 2D image coordinates. For example, the actual length of 1 meter corresponds to 20 pixels on the image. To perform the inverse conversion, the same scaling factor needs to be used.

[0112] Pixel to 3D coordinate conversion: For each pixel (x, y) on the image, the horizontal coordinate (X) and vertical coordinate (Y) of the corresponding 3D point cloud are determined by the following formula:

[0113] point[0]=x*resolution+min_coord[0]

[0114] point[1]=y*resolution+min_coord[1]

[0115] In an optional implementation, only the x and y coordinate information is retained on the two-dimensional image, while the height information (z coordinate) is missing. In order to supplement this information, a two-dimensional elevation map can be used. The specific steps are as follows: Determine the height information corresponding to the pixel point: For each annotated pixel point, query the height value (Z) of the point from the two-dimensional elevation map. Use the height information as the z-axis coordinate of the three-dimensional point cloud: Assign the queried height value (Z) to the corresponding three-dimensional point cloud as the z-axis coordinate information.

[0116] In an optional embodiment, after determining the three-dimensional point cloud data corresponding to the points on the two-dimensional intensity map, the annotation information of the three-dimensional point cloud can be determined. In the two-dimensional image, the static elements of the road have been annotated, including the category, position, direction and other information of the elements. In order to completely convert this information into a three-dimensional point cloud map, the annotation information of the static elements of the road corresponding to each pixel point can be used as an additional attribute of the three-dimensional point cloud. The specific steps are as follows: Determine the annotation information corresponding to the pixel point: For each pixel point, query the annotation information on the two-dimensional image to determine which category of road static elements the point belongs to, as well as other attributes of the element. Assign the annotation information to the three-dimensional point cloud: The queried annotation information is used as an additional attribute of the three-dimensional point cloud and saved together with the three-dimensional coordinates.

[0117] It should be noted that in order to increase the efficiency of conversion and reduce processing time, in the process of converting the points on the two-dimensional image into a three-dimensional point cloud, only the point set corresponding to the above-mentioned identified static elements of the road can be three-dimensionally converted, that is, only the points with annotation information can be back-projected onto the three-dimensional point cloud map.

[0118] Figure 8 is a schematic diagram of an optional three-dimensional true value according to an embodiment of the present application; Figure 8 As shown, after the annotated two-dimensional intensity map is back-projected onto the three-dimensional point cloud map, the three-dimensional lane lines corresponding to the annotated results can be displayed in the three-dimensional point cloud map.

[0119] Fig. 9 is a schematic diagram of an optional two-dimensional truth value according to an embodiment of the present application; Fig. 9As shown in , after obtaining the 3D true value, the 3D point cloud map can be converted to the camera coordinate system using the camera parameters, and the 2D true value can be obtained, as shown in Fig. 9 The dotted line in the figure is the marking of the lane line.

[0120] Through the above implementation of the present application, the road static element annotation information on the two-dimensional image can be accurately converted back to the three-dimensional point cloud map to generate three-dimensional true value data. This conversion process ensures the consistency and integrity of the information from two-dimensional to three-dimensional, and provides an important basis for the subsequent model performance evaluation. The three-dimensional true value data not only contains the precise position of the static elements in the three-dimensional space, but also retains the annotation information, such as the category and attributes of the elements, which is crucial for evaluating the three-dimensional space understanding ability of the perception model.

[0121] In an optional embodiment, before projecting the three-dimensional point cloud map obtained by the true value vehicle onto a two-dimensional plane, it also includes: extracting features from the point cloud data obtained by the true value vehicle and removing the point cloud data corresponding to dynamic obstacles; performing pre-integration processing and forward optimization on the true value vehicle to obtain the posture change of the true value vehicle; and generating a three-dimensional point cloud map based on the posture change of the true value vehicle and the point cloud data.

[0122] It should be noted that the real-value car can collect various sensor data during the data collection process, involving vision, lidar and vehicle dynamic information, such as: 2D image data: including images taken by cameras in front, behind and on the sides of the vehicle, these images can be collected at a frequency of 20Hz (20 frames per second), providing rich visual information. 3D point cloud data: can be generated by a rotating lidar (such as Velodyne HDL-64E), can be collected at a frequency of 10Hz, and provide 3D structural information of the vehicle's surroundings. Self-vehicle position information: obtained through the global navigation satellite system (GNSS), such as GPS or Beidou system, providing the precise position of the vehicle on the earth. Self-vehicle attitude information: provided by the inertial measurement unit (IMU), including the vehicle's tilt angle, rotation speed and acceleration, etc., used to understand the dynamic state of the vehicle. These data are usually combined and stored in data packets, such as RosBag files in ROS (Robot Operating System), for subsequent processing and analysis.

[0123] Fig.10 is a schematic diagram of an optional three-dimensional point cloud data according to an embodiment of the present application; through a laser radar and other equipment, point cloud data of the surrounding environment of the real value vehicle can be collected at a certain frequency to obtain Fig.10 The three-dimensional point cloud data shown.

[0124] After acquiring these data, the point cloud data can be processed to generate a continuous frame point cloud map of the entire scene. In an optional implementation, the key step of processing the collected multimodal sensor data and constructing a three-dimensional map of the environment can be as follows:

[0125] Preprocessing: The data undergoes a preprocessing phase, which can align the time series data of different sensors to ensure that the data from different sensors are synchronized in time. In addition, some far and near points are removed to reduce noise and unnecessary data volume, and point cloud dedistortion is performed to improve the quality of point cloud data.

[0126] Feature Extraction: In the feature extraction stage, the data can be further processed to identify ground points and other key features. The segmentation of ground points helps to separate road surface information, while dynamic verification is used to identify and remove moving objects (such as vehicles and pedestrians). Dynamic objects such as vehicles and pedestrians can be identified and removed to avoid including these moving elements in the map. Parallel points can be removed to reduce redundant data generated by parallel surfaces. Planar areas and edges of the ground can be identified to extract plane and edge features.

[0127] IMU pre-integration: IMU data is processed through pre-integration and combined with wheel speed observation (i.e., vehicle speed information) for forward prediction to construct calibrated point cloud data. Pre-integration helps estimate the dynamic displacement of the vehicle and provides temporal continuity for the point cloud data.

[0128] Factor graph backend optimization: Factor graph is an optimization framework used to process multi-sensor fusion data and find the best dataset match through optimization algorithms. At this stage, GNSS factors (providing positioning information), loop factors (helping to identify known environments for path optimization) and ground constraint factors are used to optimize the pose and build a local map. Through matching and optimization, the accuracy and consistency of map construction can be improved, and a local continuous frame point cloud map can be built.

[0129] Fig.11 is a schematic diagram of an optional point cloud map according to an embodiment of the present application; after the real value vehicle collects various types of data through sensors, the collected multimodal sensor data can be processed in the above manner to obtain Fig.11 The point cloud map shown in Fig.11 As shown, dynamic objects are removed from the point cloud map.

[0130] Through the above implementation of the present application, including the preprocessing of collected data, feature extraction, forward prediction of dynamic data and optimization of posture, a continuous frame point cloud map is finally generated. This map not only contains the three-dimensional structure of the static environment, but also takes into account the dynamic information of the vehicle. It can effectively process and fuse multi-source sensor data to construct a continuous frame point cloud map that integrates visual information and three-dimensional structural information, providing an accurate and rich environment model for autonomous driving vehicles, thereby providing effective information for subsequent true value annotation.

[0131] In an optional embodiment, after obtaining the three-dimensional true value and the two-dimensional true value as mentioned above, it also includes: obtaining the two-dimensional predicted value, and drawing the two-dimensional predicted lane lines and the two-dimensional real lane lines according to the two-dimensional predicted value and the two-dimensional true value respectively; calculating the intersection and union ratio between each pair of two-dimensional predicted lane lines and the two-dimensional real lane lines, and removing the two-dimensional predicted lane lines and the two-dimensional real lane lines corresponding to the intersection and union ratio less than the two-dimensional intersection and union ratio threshold; re-matching the remaining two-dimensional predicted lane lines and the two-dimensional real lane lines to obtain a new matching relationship; and obtaining the two-dimensional evaluation result of the prediction model according to the new matching relationship between the two-dimensional predicted lane lines and the two-dimensional real lane lines.

[0132] It should be noted that during the model evaluation phase, two-dimensional prediction values ​​can be obtained from the models participating in the evaluation. These prediction values ​​include the model's estimation of the position and attributes of the lane lines in the scene. At the same time, using the previously generated two-dimensional true values, based on the predicted values ​​and true values, the predicted lane lines and the real lane lines are drawn on the two-dimensional image respectively to form a visual comparison.

[0133] In an optional embodiment, for the true value and the predicted value, the lane line can be drawn with a 30 pixel value. Specifically, the steps can be as follows: Determine the position of the lane line: For the true value, the real position data of the lane line can be directly used, and for the predicted value, the output of the lane line detection algorithm can be used. Then, an image processing library (such as OpenCV) can be used to draw the lane line on the image. The cv2.line function in the OpenCV library can be used to draw the lane line, and the starting and ending coordinates of the line, the color, and the thickness of the line can be specified.

[0134] After drawing the lane lines, the intersection over union (IoU) between each pair of two-dimensional predicted lane lines and two-dimensional real lane lines can be calculated, and the lane lines corresponding to the IoU less than the two-dimensional IoU threshold can be removed.

[0135] It should be noted that IoU is an indicator to measure the degree of overlap between the predicted lane line and the real lane line, which can include the following steps: Determine the intersection area: find the intersection area between the predicted lane line and the real lane line, usually calculating the overlapping part between the two line segments. Calculate the intersection area: calculate the area of ​​the intersection area. For line segments, it can be the length of the intersection line segment. Determine the union area: find the union area between the predicted lane line and the real lane line. Usually involves calculating the total area covered by the two line segments. Calculate the union area: calculate the area of ​​the union area. For line segments, this can be the sum of the lengths of the two line segments minus the length of the intersection line segment. Calculate IOU: IOU is the ratio of the intersection area to the union area.

[0136] In one example, the IoU between lane lines can be calculated as follows: Calculate the intersection line segment: find the starting point and the end point of the intersection line segment, and calculate the length of the intersection line segment. Calculate the union line segment: calculate the length of the union line segment. Calculate the IoU: use the intersection length to union length.

[0137] In an optional implementation, for each pair of predicted lane lines and real lane lines, the IoU between them is calculated. If the IoU of a pair of lane lines is less than a preset threshold (two-dimensional intersection-over-union threshold), the detection result of the predicted lane line is considered inaccurate and is removed from the prediction result to improve the accuracy of subsequent matching and evaluation.

[0138] After each pair of lane line combinations has gone through the above process, the remaining two-dimensional predicted lane lines and the two-dimensional real lane lines can be re-matched to obtain a new matching relationship. For the remaining predicted lane lines and real lane lines after the IoU threshold screening, the Hungarian algorithm can be used for matching to find the optimal corresponding relationship between the predicted lane lines and the real lane lines.

[0139] The steps to calculate the best match are as follows:

[0140] Initialization: All vertices (lane lines) are marked as unmatched and ready to start the matching process.

[0141] Finding augmenting paths: Starting from unmatched lane lines, a depth-first search (DFS) is used to find an augmenting path. In the path, matched lane lines and unmatched lane lines appear alternately to form a matching chain.

[0142] Update matching: Once an augmenting path is found, increase the number of matched lane lines by reversing the edges on the path (i.e., unmatched lane lines become matched, and matched lane lines become unmatched).

[0143] Repeat the process: Repeat the process of finding augmenting paths and updating until no new augmenting paths can be found, that is, all possible matches have been completed.

[0144] Output result: The final output is the maximum IoU after matching, which reflects the closest matching result between the predicted lane line and the real lane line.

[0145] It should be noted that after determining the new matching relationship between the two-dimensional predicted lane line and the two-dimensional real lane line, the two-dimensional evaluation result of the prediction model can be calculated. The corresponding confusion matrix can be obtained through the matching situation of each frame, and then key indicators such as accuracy and recall can be calculated.

[0146] In an optional implementation, a confusion matrix can be obtained through matching, and the confusion matrix can be used to evaluate the performance of the classification model. The relationship between the true value and the predicted value is recorded, including four situations: true positive (TP), false positive (FP), true negative (TN) and false negative (FN). Specifically: TP: The number of samples predicted to be positive and actually positive. FP: The number of samples predicted to be positive and actually negative. TN: The number of samples predicted to be negative and actually negative. FN: The number of samples predicted to be negative and actually positive.

[0147] The accuracy can be calculated using the following formula:

[0148]

[0149] The recall rate can be calculated using the following formula:

[0150]

[0151] At the same time, the cubic parameter equation of the corresponding lane line can be fitted through the lane line points, and the absolute deviation between the actual lane line parameters and the predicted lane line parameters can be calculated. The general form of the equation can be:

[0152] y=c0+c1*x+c2*x 2 +c3*x 3

[0153] The reference can be fitted using statistical methods such as the least square method based on the points on the two-dimensional predicted lane line and the two-dimensional real lane line, and the absolute deviation between the corresponding parameters can be calculated, such as the absolute value of the difference between c0 corresponding to the two-dimensional predicted lane line and the two-dimensional real lane line.

[0154] Through the above implementation of the present application, IoU is calculated and screened, the Hungarian algorithm is used to optimize the lane line matching, and finally the performance of the prediction model in lane line detection is evaluated. The advantages and disadvantages of the model can be identified and the accuracy of lane line detection can be quantified.

[0155] In an optional embodiment, after obtaining the three-dimensional true value and the two-dimensional true value as mentioned above, it also includes: obtaining the three-dimensional predicted value, and performing linear sampling according to the preset segmented distance based on the three-dimensional predicted value and the three-dimensional true value to obtain a point cloud pair; calculating the distance between the three-dimensional predicted value and the three-dimensional true value between the point cloud pairs, and removing the three-dimensional predicted value and the three-dimensional true value corresponding to the distance greater than the distance threshold; re-matching the remaining three-dimensional predicted value and the three-dimensional true value to obtain a new matching relationship; and obtaining the three-dimensional evaluation result of the prediction model based on the new matching relationship between the three-dimensional predicted value and the three-dimensional true value.

[0156] It should be noted that after evaluating the model in two-dimensional space, the lane line detection performance of the autonomous driving perception model in three-dimensional space can be evaluated based on the three-dimensional true value and the three-dimensional predicted value.

[0157] In an optional implementation, three-dimensional prediction values ​​may be obtained from the models participating in the evaluation during the model evaluation phase, and these prediction values ​​include the model's estimate of the position, height, and attributes of the lane lines in the scene in three-dimensional space.

[0158] Then, linear sampling can be performed according to the preset segment distance based on the 3D predicted value and the 3D true value to obtain a point cloud pair: For the true value and the predicted value, the z-axis (height direction) and x-axis (lateral) coordinates can be linearly sampled along the direction of the vehicle at a preset segment distance (such as 0.05 meters). This process ensures fine sampling of the lane line in three-dimensional space and can capture the three-dimensional details of the lane line. The point cloud pair is formed by pairing the sampled points with the corresponding predicted points, so that the matching degree of the lane line can be evaluated based on the distance between the point pairs.

[0159] Calculate the distance between the 3D prediction value and the 3D true value between the point cloud pairs, and remove the 3D prediction value and 3D true value corresponding to the distance greater than the distance threshold: For each pair of point clouds (i.e., 3D true value point and 3D prediction point), calculate the 3D distance between them. If the distance between the point pairs is greater than the preset distance threshold, it indicates that there is a significant deviation between the model's prediction result and the true value, and the point pair will be removed to optimize the subsequent matching process. The distance threshold can be the mean of the distance of each group of data points at all sampling distances.

[0160] The remaining 3D predicted values ​​and 3D true values ​​are re-matched to obtain a new matching relationship: For the remaining point cloud pairs after the distance threshold screening, the Hungarian algorithm is used for matching optimization. The specific matching optimization can be as described above.

[0161] The three-dimensional evaluation results of the prediction model are obtained based on the new matching relationship between the three-dimensional predicted value and the three-dimensional true value: for each set of matching results, the distance between the true value and the predicted value at each sampling distance is calculated, and then whether these distances are less than the threshold is determined to be a positive example or a negative example. If the proportion of positive sampling points is greater than the preset percentage threshold, the lane line combination will be recorded as a successful matching combination. The accuracy and recall rate can be calculated based on the positive and negative examples. For the accuracy rate, the ratio of the number of positive sampling points to the total number of sampling points can be determined as the accuracy rate. For the recall rate, the ratio of the number of positive sampling points to the actual number of positive sampling points can be determined as the recall rate.

[0162] In an optional implementation, similar to the two-dimensional space, the cubic parameter equation of the corresponding lane line can be fitted through the points of the lane line to calculate the absolute deviation between the actual lane line parameters and the predicted lane line parameters.

[0163] In an optional implementation, after calculating the lane line equation, the lane line equation can be used to calculate the y value obtained at 5-120 meters at 5-meter intervals, and the error between the actual value and the predicted value can be calculated to obtain the lateral deviation of the lane line at various distances.

[0164] Through the above implementation of the present application, the entire process is to optimize the lane line matching through linear sampling and distance calculation in three-dimensional space, combined with the Hungarian algorithm, and finally evaluate the lane line detection performance of the prediction model in three-dimensional space. The accuracy of the model in three-dimensional positioning can be identified, and the model's ability to restore the lane line shape can be evaluated through parameter equation deviation, providing a comprehensive and in-depth evaluation of the three-dimensional perception ability of the autonomous driving system.

[0165] Fig.12 is a flowchart of another optional method for generating a static true value of a road according to an embodiment of the present application; Fig.12 As shown, when it is desired to evaluate the model, steps S1202-S1208 may be performed in sequence.

[0166] Step S1202, data collection. The real value car can be used for data collection, and the collected data includes two-dimensional image data at each moment (including front and rear view side view with a frequency of 20HZ), three-dimensional point cloud data (from rotating laser radar with a frequency of 10HZ), vehicle position information and vehicle attitude information (from GNSS and IMU), which are saved in the form of data packets (such as RosBag).

[0167] Step S1204, generate a point cloud map. The collected data can be parsed and mapped, and a scene map of the continuous frame scene can be constructed based on the continuous frame images, point clouds and the vehicle's posture information. The process of building a continuous frame scene map mainly includes preprocessing, feature extraction, IMU pre-integration (integrated wheel speed observation) and factor graph back-end optimization. Among them, the preprocessing stage needs to align the time series data, propose some near and far points and then perform point cloud dedistortion. The feature extraction stage will segment and dynamically verify the ground points, remove parallel points and extract plane and edge features. The IMU pre-integration stage needs to input odometer information for forward optimization prediction, and construct an estimated score for calibration distortion. The factor graph optimization uses GNSS factors, loop factors, and ground constraint factors to optimize the posture and construct a local map. In this way, a continuous frame point cloud map (three-dimensional point cloud map) of the entire scene can be generated.

[0168] Step S1206, point cloud map projection. The three-dimensional point cloud map can be projected onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional elevation map. The two-dimensional intensity map is annotated to obtain the location information, attribute information, etc. of the static elements. The annotated information is then back-projected into the three-dimensional point cloud map to obtain a three-dimensional true value, and the three-dimensional true value is converted to the camera coordinate system using camera parameters to obtain a two-dimensional true value.

[0169] Step S1208, model evaluation. Using the two-dimensional prediction value and three-dimensional prediction value of the model to be evaluated, the model is evaluated in three-dimensional space and two-dimensional space according to the two-dimensional prediction value and three-dimensional prediction value, two-dimensional true value and three-dimensional true value to obtain two-dimensional and three-dimensional evaluation results.

[0170] In an optional embodiment, the truth collection vehicle includes a laser radar and a camera. The camera collects images at 20Hz, and the laser radar collects laser radar point cloud data at 10Hz. The road collection data is divided into 20-second intervals to obtain a single segment. Each segment is analyzed and mapped to obtain a three-dimensional point cloud map of the driving range. At the same time, the two-dimensional image after the alignment time stamp and the vehicle posture information are analyzed. Taking a 20-second 20Hz video segment as an example, the annotation of a single two-dimensional intensity image can generate the annotation results of continuous frames, reaching 400 frames. The automatic annotation results can be uploaded to the manual review platform for final fine-tuning and output.

[0171] Through the above-mentioned implementation mode of the present application, the map with data annotation generated by laser radar mapping will contain a complete road information structure of continuous frame fragments, and the two-dimensional annotation results will be naturally obtained through coordinate system transformation. It is simple and efficient, and this method can adapt to the data collected by the camera at any frame rate, and annotate continuous frames, which greatly reduces the need for manual review, and the data of continuous frames can be applied to indicators such as ID jumps. By pre-annotating ground elements on the two-dimensional intensity map with a large model and then restoring them through the elevation map, automatic annotation can be completed more efficiently, without manpower, and with high parallel efficiency. Through additional performance evaluation indicators, the evaluation of lane lines at different distances is refined, and by comparing the differences in parameter equations, the difference between the true value and the predicted value is more comprehensively explained, thereby reflecting the advantages and disadvantages of the model.

[0172] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0173] According to another aspect of the embodiment of the present application, a road static truth value generation device for implementing the road static truth value generation method is also provided. Fig.13 As shown, the device comprises:

[0174] A two-dimensional plane generating unit 1302 is used to project the three-dimensional point cloud map obtained by the truth vehicle onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional elevation map, wherein the value of each pixel in the two-dimensional intensity map is used to indicate the intensity information of the corresponding position, and the value of each pixel in the two-dimensional elevation map is used to indicate the height information of the corresponding position;

[0175] A three-dimensional truth value generating unit 1304 is used to annotate the two-dimensional intensity map to obtain an annotation of at least one road static element, and back-project the annotation of the road static element into the three-dimensional point cloud map to obtain a three-dimensional truth value, wherein the annotation of the road static element is used to indicate the point set and category corresponding to the road static element;

[0176] The two-dimensional truth value generating unit 1306 is used to convert the three-dimensional truth value into the camera image according to the camera parameters to obtain the two-dimensional truth value, wherein the camera image and the three-dimensional point cloud map are data collected at the same time.

[0177] Optionally, the above-mentioned two-dimensional plane generation unit 1302 includes: performing the following operations on any point cloud on the three-dimensional point cloud map: a two-dimensional horizontal coordinate determination module, used to determine the difference between the horizontal coordinate of the point cloud and the horizontal coordinate of the minimum coordinate point cloud, and determine the ratio between the difference between the horizontal coordinates and the zoom factor as the horizontal coordinate of the corresponding two-dimensional pixel point; a two-dimensional vertical coordinate determination module, used to determine the difference between the vertical coordinate of the point cloud and the vertical coordinate of the minimum coordinate point cloud, and determine the ratio between the difference between the vertical coordinates and the zoom factor as the vertical coordinate of the corresponding two-dimensional pixel point; an information determination module, used to determine the intensity information and height information corresponding to the two-dimensional pixel point; a two-dimensional plane generation module, used to generate a two-dimensional intensity map according to the horizontal coordinates, vertical coordinates and intensity information of all two-dimensional pixel points, and generate a two-dimensional elevation map according to the horizontal coordinates, vertical coordinates and height information of all two-dimensional pixel points.

[0178] Optionally, the above-mentioned information determination module is also used to: determine the three-dimensional point cloud set corresponding to the two-dimensional pixel point according to the zoom ratio, and determine the reference intensity information set and the reference height information set corresponding to the three-dimensional point cloud set; calculate the intensity standard deviation and the intensity mean corresponding to the reference intensity information set, and the height standard deviation and the height mean corresponding to the reference height information set; filter the reference intensity information set according to the intensity standard deviation and the intensity mean to obtain the target intensity information set, and filter the reference height information set according to the height standard deviation and the height mean to obtain the target height information set; determine the average value of the target intensity information set as the intensity information corresponding to the two-dimensional pixel point, and determine the average value of the target height information set as the height information corresponding to the two-dimensional pixel point.

[0179] Optionally, the above-mentioned three-dimensional truth value generation unit 1304 includes: a splitting module, which is used to split the two-dimensional intensity map to obtain multiple two-dimensional intensity sub-maps; a labeling module, which is used to label the two-dimensional intensity sub-maps using a labeling model to obtain a labeling of at least one road static element; a merging module, which is used to merge the labeling of the road static elements corresponding to all two-dimensional intensity sub-maps to obtain the labeling of all road static elements; a three-dimensional truth value generation module, which is used to back-project the labeling of all road static elements into a three-dimensional point cloud map to obtain a three-dimensional truth value.

[0180] Optionally, the above-mentioned annotation module is also used to: perform feature extraction on the two-dimensional intensity sub-map to obtain a feature map; input the feature map into a linear element annotation model to obtain an annotation of at least one first road static element, wherein the first road static element is used to indicate the linear static element of the road; input the feature map into an icon element annotation model to obtain an annotation of at least one second road static element, wherein the second road static element is used to indicate the icon static element of the road; merge the annotation of the first road static element and the annotation of the second road static element to obtain the annotation result corresponding to the two-dimensional intensity sub-map.

[0181] Optionally, the above-mentioned three-dimensional truth value generation module is also used to: perform the following operations on the pixel points in the point set corresponding to the static elements of the road: determine the abscissa of the corresponding three-dimensional point cloud by the product of the horizontal coordinate corresponding to the pixel point and the zoom factor and the sum of the horizontal coordinate of the minimum coordinate point cloud; determine the vertical coordinate of the corresponding three-dimensional point cloud by the product of the vertical coordinate corresponding to the pixel point and the zoom factor and the sum of the vertical coordinate of the minimum coordinate point cloud; determine the height information corresponding to the pixel point according to the two-dimensional elevation map, and determine the height information as the z-axis coordinate of the corresponding three-dimensional point cloud; determine the annotation of the corresponding three-dimensional point cloud according to the annotation of the static elements of the road corresponding to the pixel point.

[0182] Optionally, the above-mentioned road static truth generation device also includes: a point cloud map generation unit, which is used to extract features from the point cloud data obtained by the true value vehicle and remove the point cloud data corresponding to dynamic obstacles; perform pre-integration processing and forward optimization on the true value vehicle to obtain the posture change of the true value vehicle; and generate a three-dimensional point cloud map based on the posture change of the true value vehicle and the point cloud data.

[0183] Optionally, the above-mentioned road static true value generation device also includes: a two-dimensional evaluation unit, used to obtain the two-dimensional prediction value, and draw the two-dimensional predicted lane lines and the two-dimensional real lane lines according to the two-dimensional prediction value and the two-dimensional true value respectively; calculate the intersection and union ratio between each pair of two-dimensional predicted lane lines and the two-dimensional real lane lines, and remove the two-dimensional predicted lane lines and two-dimensional real lane lines corresponding to the intersection and union ratio less than the two-dimensional intersection and union ratio threshold; re-match the remaining two-dimensional predicted lane lines and the two-dimensional real lane lines to obtain a new matching relationship; and obtain the two-dimensional evaluation result of the prediction model according to the new matching relationship between the two-dimensional predicted lane lines and the two-dimensional real lane lines.

[0184] Optionally, the above-mentioned road static true value generation device also includes: a three-dimensional evaluation unit, used to obtain three-dimensional prediction values, and perform linear sampling according to the three-dimensional prediction values ​​and the three-dimensional true values ​​according to a preset segmented distance to obtain point cloud pairs; calculate the distance between the three-dimensional prediction values ​​and the three-dimensional true values ​​between the point cloud pairs, and remove the three-dimensional prediction values ​​and three-dimensional true values ​​corresponding to the distance greater than the distance threshold; re-match the remaining three-dimensional prediction values ​​and three-dimensional true values ​​to obtain a new matching relationship; and obtain the three-dimensional evaluation result of the prediction model based on the new matching relationship between the three-dimensional prediction values ​​and the three-dimensional true values.

[0185] Optionally, in this embodiment, the embodiments to be implemented by the above-mentioned various unit modules can refer to the above-mentioned various method embodiments, which will not be repeated here.

[0186] According to another aspect of the embodiment of the present application, an electronic device for implementing the above-mentioned method for generating a static true value of a road is also provided. The electronic device may be Fig.14The terminal device or server shown in the figure. This embodiment is described by taking the electronic device as a terminal device as an example. Fig.14 As shown, the electronic device includes a memory 1402 and a processor 1404. The memory 1402 stores a computer program, and the processor 1404 is configured to execute the steps in any of the above method embodiments through the computer program.

[0187] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0188] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0189] S1, projecting the three-dimensional point cloud map obtained by the truth vehicle onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional elevation map, wherein the value of each pixel in the two-dimensional intensity map is used to indicate the intensity information of the corresponding position, and the value of each pixel in the two-dimensional elevation map is used to indicate the height information of the corresponding position;

[0190] S2, annotating the two-dimensional intensity map to obtain an annotation of at least one road static element, and back-projecting the annotation of the road static element into the three-dimensional point cloud map to obtain a three-dimensional true value, wherein the annotation of the road static element is used to indicate the point set and category corresponding to the road static element;

[0191] S3, converting the three-dimensional true value into the camera image according to the camera parameters to obtain the two-dimensional true value, wherein the camera image and the three-dimensional point cloud map are data collected at the same time.

[0192] Alternatively, a person skilled in the art may understand that: Fig.14 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. Fig.14 The electronic device and the electronic equipment described above are not limited in structure. Fig.14 More or fewer components (such as network interfaces, etc.) as shown in, or with Fig.14 Different configurations are shown.

[0193] Among them, the memory 1402 can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for generating the static truth value of the road in the embodiment of the present application. The processor 1404 executes various functional applications and data processing by running the software programs and modules stored in the memory 1402, that is, realizing the above-mentioned method for generating the static truth value of the road. The memory 1402 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1402 may further include a memory remotely arranged relative to the processor 1404, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 1402 can be specifically, but not limited to, used to store information such as road images. As an example, such as Fig.14 As shown, the memory 1402 may include, but is not limited to, the two-dimensional plane generating unit 1302, the three-dimensional truth value generating unit 1304, and the two-dimensional truth value generating unit 1306 in the device for generating the static truth value of the road. In addition, it may also include, but is not limited to, other module units in the device for generating the static truth value of the road, which will not be described in detail in this example.

[0194] Optionally, the transmission device 1406 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wired network and a wireless network. In one example, the transmission device 1406 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 1406 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0195] In addition, the electronic device mentioned above further includes: a display 1408 for displaying a simulated road scene; and a connection bus 1410 for connecting various module components in the electronic device mentioned above.

[0196] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes through network communication. Among them, the nodes may form a peer-to-peer (P2P, Peer To Peer) network, and any form of computing device, such as a server, terminal or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0197] According to one aspect of the present application, a computer program product is provided, the computer program product comprising a computer program / instruction, the computer program / instruction comprising a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit, various functions provided by the embodiments of the present application are executed.

[0198] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0199] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned method for generating a static true value of a road.

[0200] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0201] S1, projecting the three-dimensional point cloud map obtained by the truth vehicle onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional elevation map, wherein the value of each pixel in the two-dimensional intensity map is used to indicate the intensity information of the corresponding position, and the value of each pixel in the two-dimensional elevation map is used to indicate the height information of the corresponding position;

[0202] S2, annotating the two-dimensional intensity map to obtain an annotation of at least one road static element, and back-projecting the annotation of the road static element into the three-dimensional point cloud map to obtain a three-dimensional true value, wherein the annotation of the road static element is used to indicate the point set and category corresponding to the road static element;

[0203] S3, converting the three-dimensional true value into the camera image according to the camera parameters to obtain the two-dimensional true value, wherein the camera image and the three-dimensional point cloud map are data collected at the same time.

[0204] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0205] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to execute all or part of the steps of the above methods in each embodiment of the present application.

[0206] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0207] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0208] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0209] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0210] The above are only preferred implementations of the present application. It should be pointed out that ordinary technicians in this technical field can make several improvements and modifications without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating a static true value of a road, characterized in that: include: Projecting the three-dimensional point cloud map obtained by the truth car onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional elevation map, wherein the value of each pixel in the two-dimensional intensity map is used to indicate the intensity information of the corresponding position, and the value of each pixel in the two-dimensional elevation map is used to indicate the height information of the corresponding position; Annotating the two-dimensional intensity map to obtain an annotation of at least one road static element, and back-projecting the annotation of the road static element into the three-dimensional point cloud map to obtain a three-dimensional true value, wherein the annotation of the road static element is used to indicate the point set and category corresponding to the road static element; The three-dimensional true value is converted into a camera image according to camera parameters to obtain a two-dimensional true value, wherein the camera image and the three-dimensional point cloud map are data collected at the same time.

2. The method according to claim 1, characterized in that The three-dimensional point cloud map obtained by the truth vehicle is projected onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional elevation map, including: The following operations are performed on any point cloud on the three-dimensional point cloud map: Determine the difference between the abscissa of the point cloud and the abscissa of the minimum coordinate point cloud, and determine the ratio between the difference between the abscissas and the zoom factor as the abscissa of the corresponding two-dimensional pixel point; Determine the difference between the ordinate of the point cloud and the ordinate of the minimum coordinate point cloud, and determine the ratio between the difference between the ordinates and the zoom factor as the ordinate of the corresponding two-dimensional pixel point; Determine intensity information and height information corresponding to the two-dimensional pixel point; The two-dimensional intensity map is generated according to the horizontal coordinates, vertical coordinates and the intensity information of all the two-dimensional pixel points, and the two-dimensional elevation map is generated according to the horizontal coordinates, vertical coordinates and the height information of all the two-dimensional pixel points.

3. The method according to claim 2, characterized in that The determining the intensity information and the height information corresponding to the two-dimensional pixel point includes: Determine a three-dimensional point cloud set corresponding to the two-dimensional pixel point according to the zoom ratio, and determine a reference intensity information set and a reference height information set corresponding to the three-dimensional point cloud set; Calculate the intensity standard deviation and intensity mean corresponding to the reference intensity information set, and the height standard deviation and height mean corresponding to the reference height information set; The reference intensity information set is screened according to the intensity standard deviation and the intensity mean to obtain a target intensity information set, and the reference height information set is screened according to the height standard deviation and the height mean to obtain a target height information set; The average value of the target intensity information set is determined as the intensity information corresponding to the two-dimensional pixel point, and the average value of the target height information set is determined as the height information corresponding to the two-dimensional pixel point.

4. The method according to claim 2, characterized in that: The step of labeling the two-dimensional intensity map to obtain a label of at least one road static element, and back-projecting the label of the road static element into the three-dimensional point cloud map to obtain a three-dimensional true value includes: Splitting the two-dimensional intensity map to obtain a plurality of two-dimensional intensity sub-maps; Annotating the two-dimensional intensity sub-map using an annotation model to obtain an annotation of at least one of the road static elements; Merging the annotations of the road static elements corresponding to all the two-dimensional intensity sub-images to obtain annotations of all the road static elements; The annotations of all the static elements of the road are back-projected into the three-dimensional point cloud map to obtain three-dimensional true values.

5. The method according to claim 4, characterized in that The step of labeling the two-dimensional intensity sub-map using the labeling model to obtain a labeling of at least one of the road static elements includes: Performing feature extraction on the two-dimensional intensity sub-image to obtain a feature image; Inputting the characteristic graph into a linear element annotation model to obtain an annotation of at least one first road static element, wherein the first road static element is used to indicate a linear static element of a road; Inputting the characteristic graph into an icon element annotation model to obtain an annotation of at least one second road static element, wherein the second road static element is used to indicate an icon static element of a road; The annotation of the first road static element and the annotation of the second road static element are combined to obtain an annotation result corresponding to the two-dimensional intensity sub-image.

6. The method according to claim 4, characterized in that The step of back-projecting the annotations of all the static elements of the road into the three-dimensional point cloud map to obtain a three-dimensional true value includes: The following operations are performed on the pixel points in the point set corresponding to the static elements of the road: Determine the sum of the product of the abscissa corresponding to the pixel point and the zoom factor and the abscissa of the minimum coordinate point cloud as the abscissa of the corresponding three-dimensional point cloud; Determine the sum of the product of the ordinate corresponding to the pixel point and the zoom factor and the ordinate of the minimum coordinate point cloud as the ordinate of the corresponding three-dimensional point cloud; Determine the height information corresponding to the pixel point according to the two-dimensional elevation map, and determine the height information as the z-axis coordinate of the corresponding three-dimensional point cloud; The annotation of the corresponding three-dimensional point cloud is determined according to the annotation of the static element of the road corresponding to the pixel point.

7. The method according to any one of claims 1 to 6, characterized in that Before projecting the three-dimensional point cloud map obtained by the true value vehicle onto the two-dimensional plane, the method further includes: Perform feature extraction on the point cloud data acquired by the true value vehicle, and remove the point cloud data corresponding to the dynamic obstacles; Performing pre-integration processing and forward optimization on the true value vehicle to obtain a posture change of the true value vehicle; The three-dimensional point cloud map is generated based on the position change of the true value vehicle and the point cloud data.

8. The method according to any one of claims 1 to 6, characterized in that After obtaining the three-dimensional true value and the two-dimensional true value, the method further includes: Obtaining a two-dimensional prediction value, and drawing a two-dimensional prediction lane line and a two-dimensional real lane line respectively according to the two-dimensional prediction value and the two-dimensional true value; Calculating the intersection-and-joint ratio between each pair of the two-dimensional predicted lane lines and the two-dimensional real lane lines, and removing the two-dimensional predicted lane lines and the two-dimensional real lane lines corresponding to the intersection-and-joint ratios that are less than a two-dimensional intersection-and-joint ratio threshold; Re-matching the remaining two-dimensional predicted lane lines with the two-dimensional real lane lines to obtain a new matching relationship; A two-dimensional evaluation result of the prediction model is obtained according to a new matching relationship between the two-dimensional predicted lane line and the two-dimensional real lane line.

9. The method according to any one of claims 1 to 6, characterized in that: After obtaining the three-dimensional true value and the two-dimensional true value, the method further includes: Acquire a three-dimensional prediction value, and perform linear sampling according to a preset segment distance based on the three-dimensional prediction value and the three-dimensional true value to obtain a point cloud pair; Calculating the distance between the three-dimensional predicted value and the three-dimensional true value between the point cloud pairs, and removing the three-dimensional predicted value and the three-dimensional true value corresponding to the distance greater than a distance threshold; Rematching the remaining three-dimensional predicted values ​​and the three-dimensional true values ​​to obtain a new matching relationship; A three-dimensional evaluation result of the prediction model is obtained according to a new matching relationship between the three-dimensional predicted value and the three-dimensional true value.

10. A device for generating a static true value of a road, characterized in that: include: A two-dimensional plane generating unit is used to project the three-dimensional point cloud map obtained by the truth vehicle onto a two-dimensional plane to obtain a two-dimensional intensity map and a two-dimensional elevation map, wherein the value of each pixel in the two-dimensional intensity map is used to indicate the intensity information of the corresponding position, and the value of each pixel in the two-dimensional elevation map is used to indicate the height information of the corresponding position; a three-dimensional truth value generating unit, configured to annotate the two-dimensional intensity map to obtain an annotation of at least one road static element, and back-project the annotation of the road static element into the three-dimensional point cloud map to obtain a three-dimensional truth value, wherein the annotation of the road static element is used to indicate a point set and a category corresponding to the road static element; The two-dimensional truth value generation unit is used to convert the three-dimensional truth value into a camera image according to camera parameters to obtain a two-dimensional truth value, wherein the camera image and the three-dimensional point cloud map are data collected at the same time.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 9 when executed.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 9 through the computer program.

Images