A road surface reconstruction method and apparatus, device, medium, program product

Abstract

This application provides a road reconstruction method, apparatus, device, medium, and program product. The method includes: acquiring multimodal sensing data of the road surface where a vehicle is located; wherein the multimodal sensing data includes: a laser point cloud and a camera image corresponding to the laser point cloud; determining a first initial surface element based on the multimodal sensing data; initializing the height information of the first initial surface element based on the height information of the laser point cloud in a first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grids of the first laser grid to obtain a second initial surface element; and reconstructing the second initial surface element based on the road surface visual features obtained by feature extraction from the camera image to determine the reconstructed road surface grid.

Description

Technical Field

[0001] This application relates to the field of autonomous driving perception technology, specifically to a road reconstruction method, device, equipment, medium, and program product. Background Technology

[0002] In multimodal road reconstruction techniques based on vision and laser, a global k-nearest neighbor (kNN) search algorithm is typically used to find the laser points corresponding to Gaussian surface elements, thereby achieving the matching and fusion of laser point clouds and visual features. However, this search algorithm requires traversing the laser point clouds of the entire ground to calculate the Euclidean distance from the laser point cloud to the surface element, resulting in excessively long reconstruction time. Therefore, a road reconstruction method that can balance accuracy and efficiency is urgently needed. Summary of the Invention

[0003] This application provides a road reconstruction method and apparatus, cloud device, computer storage medium, and computer program product, which are beneficial to reducing the computation time during road reconstruction without sacrificing neighborhood matching accuracy.

[0004] In a first aspect, embodiments of this application provide a road surface reconstruction method, the method comprising: acquiring multimodal sensing data of the road surface where a vehicle is located; wherein the multimodal sensing data includes: a laser point cloud and a camera image corresponding to the laser point cloud; determining a first initial surface element based on the multimodal sensing data; initializing the height information of the first initial surface element based on the height information of the laser point cloud in a first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in a neighboring laser grid of the first laser grid to obtain a second initial surface element; and reconstructing the second initial surface element based on the road surface visual features obtained by feature extraction from the camera image to determine the reconstructed road surface grid.

[0005] It is understood that in the road reconstruction method provided in this application embodiment, the following steps are taken: First, the laser point cloud of the road surface where the vehicle is located and the camera image corresponding to the laser point cloud are acquired; based on multimodal sensing data, a first initial surface element is determined; and based on the height information of the laser point cloud in the first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grid of the first laser grid, the height information of the first initial surface element is initialized to obtain a second initial surface element; then, based on the road surface visual features obtained by feature extraction from the camera image, the second initial surface element is reconstructed to determine the reconstructed road surface grid. Thus, when initializing the height information of the first initial surface element, the reference information is the height information of the laser point cloud in the first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grid of the first laser grid, rather than traversing all laser point clouds globally. This helps reduce the computation time during road reconstruction without sacrificing neighborhood matching accuracy.

[0006] In some embodiments, initializing the height information of the first initial surface element based on the height information of the laser point cloud in the first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grid of the first laser grid to obtain the second initial surface element includes: determining a first distance between the laser point cloud in the first laser grid and the laser point cloud in the neighboring laser grid and the first initial surface element; determining the weight of the height of the laser point cloud in the first laser grid and the weight of the height of the laser point cloud in the neighboring laser grid based on the corresponding first distance; and initializing the height information of the first initial surface element based on the height of the laser point cloud in the first laser grid, the height of the laser point cloud in the neighboring laser grid, and the weight of the height of each laser point cloud to obtain the second initial surface element.

[0007] It is understood that in the road reconstruction method provided in this application embodiment, the first distances between the laser point clouds in the first laser grid and the laser point clouds in the neighboring laser grid and the first initial surface element are first determined; then, the weights of the heights of the laser point clouds in the first laser grid and the laser point clouds in the neighboring laser grid are determined according to the corresponding first distances; and the second initial surface element is obtained according to the heights of the laser point clouds in the first laser grid, the heights of the laser point clouds in the neighboring laser grid, and the weights of the heights of each laser point cloud. This eliminates the need to traverse the distances between all laser point clouds globally and the first initial surface element through nearest neighbor search; only the local laser point clouds need to be matched and calculated with the first initial surface element. This helps reduce the computation time during road reconstruction without sacrificing the accuracy of neighbor matching.

[0008] In some embodiments, initializing the height information of the first initial surface element based on the height information of the laser point cloud in the first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grid of the first laser grid to obtain the second initial surface element includes: determining the first distance between the laser point cloud in the first laser grid and the laser point cloud in the neighboring laser grid and the first initial surface element; determining the laser grid corresponding to the smallest distance among a plurality of first distances as the target laser grid; and determining the height of the first initial surface element based on the height information of the laser point cloud in the target laser grid to obtain the second initial surface element.

[0009] It is understood that in the road reconstruction method provided in this application embodiment, the first distances between the laser point clouds in the first laser grid and the laser point clouds in the neighboring laser grids and the first initial surface element are firstly determined; the laser grid corresponding to the smallest distance among multiple first distances is determined as the target laser grid; based on the height information of the laser point clouds in the target laser grid, the height of the first initial surface element is determined, thus obtaining the second initial surface element. This eliminates the need to traverse the distances between all global laser point clouds and the first initial surface element through nearest neighbor search; only the local laser point clouds need to be matched and calculated with the first initial surface element. This helps reduce the computation time during road reconstruction without sacrificing the accuracy of neighbor matching.

[0010] In some embodiments, initializing the height information of the first initial surface element based on the height information of the laser point cloud in the first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grids of the first laser grid to obtain the second initial surface element includes: determining the average height of the laser point cloud in the first laser grid and the laser point cloud in each of the neighboring laser grids; determining the average value as the height of the first initial surface element to obtain the second initial surface element.

[0011] It is understood that in the road reconstruction method provided in this application embodiment, the average height of the laser point cloud in the first laser grid and the laser point cloud in the neighboring laser grid is determined as the height of the first initial surface element, thus obtaining the second initial surface element. This eliminates the need to traverse the distances between all global laser point clouds and the first initial surface element through nearest neighbor search. Therefore, it helps reduce the computation time during road reconstruction without sacrificing neighborhood matching accuracy.

[0012] In some embodiments, determining the first initial surface element based on the multimodal sensing data further includes: dividing the road surface where the vehicle is located into multiple adjacent grids; determining the first surface element corresponding to the grid based on the radius of the grid; for the first frame of the camera image, initializing the visual parameters of the first surface element corresponding to the first frame of the camera image based on the first frame of the camera image to obtain the first initial surface element corresponding to the first frame of the camera image; for the i-th frame of the camera image, determining the first initial surface element corresponding to the i-th frame of the camera image based on the first initial surface element corresponding to the i-th frame of the camera image and the first frame of the camera image; where i is greater than or equal to 2.

[0013] It is understood that in the road reconstruction method provided in this application embodiment, considering that there is a lot of repetitive content in consecutive camera images, when initializing the first surface element corresponding to the i-th frame camera image based on the i-th frame camera image, the first initial surface element corresponding to the first frame camera image is referenced to obtain the first initial surface element of the i-th frame camera image. This helps to reduce the amount of initialization calculation, thereby improving the efficiency of road reconstruction.

[0014] In some embodiments, determining the first initial surface element corresponding to the camera image in the i-th frame based on the first initial surface element corresponding to the camera image in the i-th frame and the first initial surface element corresponding to the camera image in the first frame includes: determining the motion pose of the vehicle based on the multimodal sensing data; determining a second surface element that overlaps with the first initial surface element of the camera image in the i-th frame; adjusting the spatial position of the second surface element based on the first motion pose of the camera image in the i-th frame and the second motion pose of the camera image in the first frame to obtain a third initial surface element; and determining the first initial surface element corresponding to the camera image in the i-th frame based on the third initial surface element.

[0015] It is understood that in the road reconstruction method provided in this application embodiment, considering that there is a lot of repetitive content in the camera images of consecutive frames, for the second surface element that overlaps with the first initial surface element of the first frame camera image, the spatial position of the second surface element is adjusted according to the first motion pose corresponding to the i-th frame camera image and the second motion pose corresponding to the first frame camera image, thereby obtaining the first initial surface element of the i-th frame camera image. This helps to reduce the amount of initialization calculation and ensure the accuracy of the reconstructed road surface mesh.

[0016] In some embodiments, determining the first initial surface element corresponding to the camera image in the i-th frame based on the first initial surface element corresponding to the camera image in the i-th frame and the first initial surface element corresponding to the camera image in the first frame further includes: determining the motion pose of the vehicle based on the multimodal sensing data; determining a third surface element that overlaps with the first initial surface element of the camera image in the i-th frame and the first initial surface element of the camera image in the (i-1)-th frame; adjusting the spatial position of the third surface element based on the first motion pose of the camera image in the i-th frame and the third motion pose of the camera image in the (i-1)-th frame to obtain a fourth initial surface element; and determining the first initial surface element corresponding to the camera image in the i-th frame based on the fourth initial surface element.

[0017] It is understood that in the road reconstruction method provided in this application embodiment, considering that there is a lot of repetitive content in the camera images of consecutive frames, for the third surface element corresponding to the surface element of the i-th frame camera image and the first initial surface element of the (i-1)-th frame camera image, the spatial position of the third surface element is adjusted according to the first motion pose corresponding to the i-th frame camera image and the third motion pose corresponding to the (i-1)-th frame camera image, thereby obtaining the first initial surface element of the i-th frame camera image. This is beneficial to reduce the amount of initialization calculation and ensure the accuracy of the reconstructed road surface mesh.

[0018] In some embodiments, the neighborhood laser grid includes a laser grid centered on the first laser grid and extending within a first spatial range or a first distance range around the first laser grid.

[0019] It is understood that in the road reconstruction method provided in this application embodiment, considering the deviation of the laser point cloud, the laser grids within a first spatial range or a first distance range around the first laser grid are determined as neighboring laser grids; and based on the height information of the laser point clouds in the determined first laser grid and neighboring laser grids, the height information of the corresponding first initial surface element is determined. This helps reduce the computation time during road reconstruction without sacrificing the accuracy of neighbor matching.

[0020] Secondly, embodiments of this application provide a road reconstruction device, which includes: an acquisition module configured to acquire multimodal sensing data of the road surface where a vehicle is located; wherein the multimodal sensing data includes: a laser point cloud and a camera image corresponding to the laser point cloud; a determination module configured to determine a first initial surface element based on the multimodal sensing data; an initialization module configured to initialize the height information of the first initial surface element based on the height information of the laser point cloud in a first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in a neighboring laser grid of the first laser grid, to obtain a second initial surface element; and a reconstruction module configured to reconstruct the second initial surface element based on the road surface visual features obtained by feature extraction from the camera image, to determine the reconstructed road surface grid.

[0021] Thirdly, embodiments of this application provide a cloud device including a memory and a processor. The memory stores a computer program that can run on the processor, and when the processor executes the program, it implements the road reconstruction method described in the first aspect.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the road reconstruction method described in the first aspect.

[0023] Fifthly, embodiments of this application provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the road reconstruction method described in the first aspect. Attached Figure Description

[0024] Figure 1 A schematic diagram illustrating the implementation process of a road reconstruction method provided in this application embodiment; Figure 2 This application provides a schematic diagram of an implementation process for determining a first initial surface element. Figure 1 ; Figure 3This application provides a schematic diagram of an implementation process for determining a first initial surface element. Figure 2 ; Figure 4 This application provides a schematic diagram of an implementation process for determining a first initial surface element. Figure 3 ; Figure 5 This application provides a schematic diagram of an implementation process for obtaining a second initial surface element. Figure 1 ; Figure 6 This application provides a schematic diagram of an implementation process for obtaining a second initial surface element. Figure 2 ; Figure 7 A schematic diagram of a road reconstruction device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a cloud device provided in an embodiment of this application. Detailed Implementation

[0025] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described below in conjunction with the accompanying drawings. The embodiments described below are only some embodiments of this application, not all embodiments. Therefore, the described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] In the following description, references to “some embodiments” or “other embodiments” describe a subset of all possible embodiments. However, it is understood that “some embodiments” or “other embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0028] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0030] In one related technology, a multimodal road surface reconstruction technique based on vision and laser is provided. This technique employs a global K-Nearest Neighbor (KNN) search algorithm to find the laser point cloud corresponding to Gaussian surface elements, thereby achieving matching and fusion of laser point clouds and visual features. Furthermore, in this technique, each training sample / frame of data requires independent initialization of surface elements / regeneration of the basic 3D mesh unit.

[0031] In their research and analysis of the aforementioned related technologies, the inventors of this application discovered the following shortcomings in the related technologies: (1) The global KNN search algorithm needs to traverse all ground laser point clouds globally to calculate the Euclidean distance from the laser point cloud to the surface element. When the number of surface elements and the number of laser points reach 10, 5 At levels above 1, the computational complexity increases exponentially, resulting in excessively long reconstruction times and making it difficult to meet the efficiency requirements for project implementation. (2) Repeated initialization operations not only increase redundant computation, but also easily lead to poor consistency of cross-frame elements due to differences in initialization parameters, affecting the stability of reconstruction accuracy.

[0032] Therefore, the related technologies suffer from low efficiency in global KNN search and repetitive calculation of initialization elements.

[0033] In view of this, this application provides a road reconstruction method, which is applied to a cloud device. Figure 1 This is a schematic diagram illustrating the implementation process of a road reconstruction method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes steps 101 to 104: Step 101: Acquire multimodal sensing data of the road surface where the vehicle is located; wherein, the multimodal sensing data includes: laser point cloud and camera image corresponding to the laser point cloud; Step 102: Determine the first initial surface element based on the multimodal sensing data; Step 103: Based on the height information of the laser point cloud in the first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grid of the first laser grid, initialize the height information of the first initial surface element to obtain the second initial surface element. Step 104: Based on the road surface visual features obtained by feature extraction from the camera image, the second initial surface element is reconstructed to determine the reconstructed road surface grid.

[0034] It is understood that in the road reconstruction method provided in this application embodiment, the following steps are taken: First, the laser point cloud of the road surface where the vehicle is located and the camera image corresponding to the laser point cloud are acquired; based on multimodal sensing data, a first initial surface element is determined; and based on the height information of the laser point cloud in the first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grid of the first laser grid, the height information of the first initial surface element is initialized to obtain a second initial surface element; then, based on the road surface visual features obtained by feature extraction from the camera image, the second initial surface element is reconstructed to determine the reconstructed road surface grid. Thus, when initializing the height information of the first initial surface element, the reference information is the height information of the laser point cloud in the first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grid of the first laser grid, rather than traversing all laser point clouds globally. This helps reduce the computation time during road reconstruction without sacrificing neighborhood matching accuracy.

[0035] The following sections will describe further optional implementation methods for each of the above steps, as well as related terms.

[0036] In step 101, multimodal sensing data of the road surface where the vehicle is located is acquired; wherein, the multimodal sensing data includes: laser point cloud and camera image corresponding to the laser point cloud.

[0037] It should be understood that in the embodiments of this application, the vehicle is driving on its own road surface, and during the driving process, it acquires road surface images (i.e., camera images) at a certain sampling frequency through an on-board camera, and acquires laser point clouds at a certain sampling frequency through an on-board radar (LiDAR).

[0038] In some embodiments, the camera image corresponding to the laser point cloud refers to the camera image being synchronized in time and aligned in space with the laser point cloud. Specifically, time synchronization means that the camera image and the laser point cloud capture the physical world at the same instant; spatial alignment means that the camera image and the laser power source capture the physical world at the same location.

[0039] In some embodiments, a Precision Time Protocol (PTP) can be used to ensure that different devices (e.g., cameras, radars) operate on the same high-precision clock, guaranteeing that the data they record are time-aligned. In other embodiments, each sensor assigns a "time tag" to the moment it acquires a frame of data (such as an image or a laser point cloud), and subsequent processing uses this tag to find and associate data belonging to the same time.

[0040] In some embodiments, after acquiring multimodal sensing data of the road surface where the vehicle is located, the method further includes: filtering the laser point cloud to denoise the laser point cloud and obtaining a noise-removed laser point cloud; and correcting the camera image to obtain a corrected camera image.

[0041] In step 102, a first initial surface element is determined based on the multimodal sensing data.

[0042] In some embodiments, Figure 2 This application provides a schematic diagram of an implementation process for determining a first initial surface element. Figure 1 ,like Figure 2 As shown, determining the first initial surface element based on the multimodal sensing data further includes the following steps 201 to 204: Step 201: Divide the road surface where the vehicle is located into multiple adjacent grids; Step 202: Determine the first face element corresponding to the grid based on the radius of the grid; Step 203: For the first frame of the camera image, initialize the visual parameters of the first facet corresponding to the first frame of the camera image to obtain the first initial facet corresponding to the first frame of the camera image. Step 204: For the camera image in the i-th frame, determine the first initial surface element corresponding to the camera image in the i-th frame based on the first initial surface element corresponding to the camera image in the i-th frame and the first frame of the camera image; where i is greater than or equal to 2.

[0043] It is understood that in the road reconstruction method provided in this application embodiment, considering that there is a lot of repetitive content in consecutive camera images, when initializing the first surface element corresponding to the i-th frame camera image based on the i-th frame camera image, the first initial surface element corresponding to the first frame camera image is referenced to obtain the first initial surface element of the i-th frame camera image. This helps to reduce the amount of initialization calculation, thereby improving the efficiency of road reconstruction.

[0044] It should be understood that the radius of the first facet is not limited in this embodiment. In some embodiments, determining the first facet corresponding to the grid based on the radius of the grid includes: determining the radius of the first facet based on the radius of the grid; and determining the center point of the grid as the dot of the first facet. Further, in some embodiments, the first facet is the circumcircle of the grid. Further, in other embodiments, the radius of the first facet is greater than or equal to the side length of the grid. For example, in one possible implementation, the size of the grid is 1 cm × 1 cm; the radius of the face element is 0.8 cm.

[0045] In some embodiments, initializing the visual parameters of the first facet corresponding to the first frame of the camera image based on the first frame of the camera image includes: determining the first pixel position of the first facet on the corresponding camera image based on the calibrated camera parameters; and initializing the visual parameters of the first facet corresponding to the first frame of the camera image based on the visual parameters of the pixel corresponding to the first pixel position.

[0046] It should be understood that the visual parameters are not limited in the embodiments of this application. In some embodiments, the visual parameters include: color (RGB), texture features, semantic tags (such as asphalt, lane lines, cracks), reflectance, or brightness. In some embodiments, the visual parameters of the first facet corresponding to the first frame of the camera image are initialized, including: assigning initial values ​​for visual attributes such as color, texture, and semantic category to the first facet corresponding to the first frame of the camera image. In some embodiments, the semantic tags are obtained by semantic segmentation of pixel regions in the camera image.

[0047] It should be understood that, in the embodiments of this application, for the camera image of the i-th frame, determining the first initial surface element corresponding to the camera image of the i-th frame based on the first initial surface element corresponding to the camera image of the i-th frame and the first frame of the camera image includes: for the camera image of the i-th frame, determining the first initial surface element corresponding to the camera image of the i-th frame based on the first initial surface element corresponding to the camera image of the i-th frame and the first frame of the camera image; or, If i equals 2, the first initial surface element corresponding to the second frame of the camera image is determined based on the first initial surface element corresponding to the second frame of the camera image and the first frame of the camera image; if i is greater than or equal to 3, for the (i-1)th frame of the camera image, the first initial surface element corresponding to the (i-1)th frame of the camera image is determined based on the first initial surface element corresponding to the (i-2)th frame of the camera image; for the i-th frame of the camera image, the first initial surface element corresponding to the i-th frame of the camera image is determined based on the first initial surface element corresponding to the i-th frame of the camera image and the (i-1)th frame of the camera image.

[0048] In some embodiments, Figure 3 This application provides a schematic diagram of an implementation process for determining a first initial surface element. Figure 2 ,like Figure 3 As shown, determining the first initial surface element corresponding to the camera image in the i-th frame based on the first initial surface element corresponding to the camera image in the first frame includes the following steps 301 to 304: Step 301: Determine the motion pose of the vehicle based on the multimodal sensing data; Step 302: Determine the second face element that overlaps with the first initial face element of the camera image in the first frame; Step 303: Based on the first motion pose corresponding to the camera image in the i-th frame and the second motion pose corresponding to the camera image in the first frame, adjust the spatial position of the second face element to obtain the third initial face element; Step 304: Determine the first initial surface element corresponding to the camera image in the i-th frame based on the third initial surface element.

[0049] It is understood that in the road reconstruction method provided in this application embodiment, considering that there is a lot of repetitive content in the camera images of consecutive frames, for the second surface element that overlaps with the first initial surface element of the first frame camera image, the spatial position of the second surface element is adjusted according to the first motion pose corresponding to the i-th frame camera image and the second motion pose corresponding to the first frame camera image, thereby obtaining the first initial surface element of the i-th frame camera image. This helps to reduce the amount of initialization calculation and ensure the accuracy of the reconstructed road surface mesh.

[0050] In some embodiments, the multimodal sensing data further includes: camera calibration data (e.g., focal length, principal point, distortion coefficient, etc.) and IMU data acquired by the Inertial Measurement Unit (IMU). In some embodiments, determining the vehicle's motion pose based on the multimodal sensing data includes: determining the vehicle's motion pose based on one or more of the following: camera images, laser point clouds corresponding to camera images, camera calibration data corresponding to camera images, and IMU data corresponding to camera images.

[0051] Accordingly, in this embodiment, the first motion pose is determined based on one or more of the following: the camera image of the i-th frame, the laser point cloud corresponding to the camera image of the i-th frame, the camera calibration data corresponding to the camera image of the i-th frame, and the IMU data corresponding to the camera image of the i-th frame. The second motion pose is determined based on one or more of the following: the camera image of the first frame, the laser point cloud corresponding to the camera image of the first frame, the camera calibration data corresponding to the camera image of the first frame, and the IMU data corresponding to the camera image of the first frame.

[0052] It should be understood that, in the embodiments of this application, the spatial position of the second surface element is adjusted according to the first motion pose corresponding to the camera image of the i-th frame and the second motion pose corresponding to the camera image of the first frame, so that the third initial surface element obtained is an initial surface element adapted to the camera image of the i-th frame.

[0053] In some embodiments, determining the first initial surface element corresponding to the camera image in the i-th frame based on the third initial surface element further includes: determining a fourth surface element that does not overlap with the first initial surface element of the camera image in the i-th frame; initializing the visual parameters of the fourth surface element based on the camera image in the i-th frame to obtain a fifth initial surface element corresponding to the camera image in the i-th frame; and determining the first initial surface element based on the fifth initial surface element and the third initial surface element.

[0054] Furthermore, in some embodiments, the step of initializing the visual parameters of the fourth facet based on the camera image of the i-th frame to obtain the fifth initial facet corresponding to the camera image of the i-th frame includes: determining the second pixel position of the fourth facet on the corresponding camera image based on the calibrated camera parameters; and initializing the visual parameters of the fourth facet based on the visual parameters of the pixel corresponding to the second pixel position.

[0055] In other embodiments, Figure 4 This application provides a schematic diagram of an implementation process for determining a first initial surface element. Figure 3 ,like Figure 4 As shown, determining the first initial surface element corresponding to the camera image in the i-th frame based on the first initial surface element corresponding to the camera image in the first frame includes the following steps 401 to 404: Step 401: Determine the motion pose of the vehicle based on the multimodal sensing data; Step 402: Determine the third face element that overlaps with the first initial face element of the camera image in the i-th frame; Step 403: Based on the first motion pose corresponding to the camera image in the i-th frame and the third motion pose corresponding to the camera image in the (i-1)-th frame, adjust the spatial position of the third facet to obtain the fourth initial facet. Step 404: Determine the first initial surface element corresponding to the camera image in the i-th frame based on the fourth initial surface element.

[0056] It is understood that in the road reconstruction method provided in this application embodiment, considering that there is a lot of repetitive content in the camera images of consecutive frames, for the third surface element corresponding to the surface element of the i-th frame camera image and the first initial surface element of the (i-1)-th frame camera image, the spatial position of the third surface element is adjusted according to the first motion pose corresponding to the i-th frame camera image and the third motion pose corresponding to the (i-1)-th frame camera image, thereby obtaining the first initial surface element of the i-th frame camera image. This is beneficial to reduce the amount of initialization calculation and ensure the accuracy of the reconstructed road surface mesh.

[0057] In this embodiment of the application, the third motion pose is determined based on one or more of the following: the camera image of the (i-1)th frame, the laser point cloud corresponding to the camera image of the (i-1)th frame, the camera calibration data corresponding to the camera image of the (i-1)th frame, and the IMU data corresponding to the camera image of the (i-1)th frame.

[0058] It should be understood that, in the embodiments of this application, the spatial position of the third surface element is adjusted according to the first motion pose corresponding to the camera image of the i-th frame and the third motion pose corresponding to the camera image of the (i-1)-th frame, so that the fourth initial surface element is an initial surface element adapted to the camera image of the i-th frame.

[0059] In some embodiments, determining the first initial surface element corresponding to the camera image in the i-th frame based on the fourth initial surface element further includes: determining a fifth surface element that does not overlap with the first initial surface element of the camera image in the (i-1)-th frame; initializing the visual parameters of the fifth surface element based on the camera image in the i-th frame to obtain a sixth initial surface element corresponding to the camera image in the (i-1)-th frame; and determining the first initial surface element corresponding to the camera image in the i-th frame based on the sixth initial surface element and the fourth initial surface element.

[0060] Furthermore, in some embodiments, the step of initializing the visual parameters of the fifth facet based on the camera image of the i-th frame to obtain the sixth initial facet corresponding to the camera image of the i-th frame includes: determining the third pixel position of the fifth facet on the corresponding camera image based on the calibrated camera parameters; and initializing the visual parameters of the fifth facet based on the visual parameters of the pixel corresponding to the third pixel position.

[0061] For example, in one possible implementation, the shared initialization element can be achieved through steps 11 to 14 as follows: Step 11: Divide the road surface where the vehicle is located into multiple adjacent grids; Step 12: When the first frame data (i.e., camera image) is input, the data parameters of the face elements are initialized through the first frame data, and an initial face element set F0 is generated, storing the face element coordinates, normal vectors, dimensions and other core parameters of F0. Step 13: When inputting the data of the i-th frame (i≥2), directly call F0 as the basic surface element, and calculate the pose offset between the i-th frame and the first frame through IMU and camera calibration data; Step 14: Adjust the spatial position of the surface elements in F0 according to the pose offset to obtain the basic surface element F adapted to the i-th frame. i0 .

[0062] In step 103, the height information of the first initial surface element is initialized based on the height information of the laser point cloud in the first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grid of the first laser grid, so as to obtain the second initial surface element.

[0063] It should be understood that the neighborhood laser grid is not limited in the embodiments of this application. In some embodiments, the neighborhood laser grid includes a laser grid centered on the first laser grid and extending within a first spatial range or a first distance range around the first laser grid.

[0064] It is understood that in the road reconstruction method provided in this application embodiment, considering the deviation of the laser point cloud, the laser grids within a first spatial range or a first distance range around the first laser grid are determined as neighboring laser grids; and based on the height information of the laser point clouds in the determined first laser grid and neighboring laser grids, the height information of the corresponding first initial surface element is determined. This helps reduce the computation time during road reconstruction without sacrificing the accuracy of neighbor matching.

[0065] In this embodiment, the first spatial range is not limited. In some embodiments, laser grids belonging to the first spatial range are defined as neighboring laser grids of the first laser grid, centered on the first laser grid. In this embodiment, the first distance range is not limited. In some embodiments, laser grids within a first distance range from the first laser grid are defined as neighboring laser grids of the first laser grid.

[0066] In some embodiments, the laser point cloud includes the laser point cloud corresponding to multiple frames of camera images. That is, the laser point cloud includes all laser point clouds acquired throughout the entire acquisition process.

[0067] In some embodiments, the method further includes: extracting road surface point clouds from laser point clouds using a point cloud segmentation algorithm to obtain road surface laser point clouds; and assigning the road surface laser point clouds to corresponding grids according to the three-dimensional coordinates of the road surface laser point clouds to obtain laser grids.

[0068] In some embodiments, Figure 5 This application provides a schematic diagram of an implementation process for obtaining a second initial surface element. Figure 1 ,like Figure 5 As shown, the step of initializing the height information of the first initial surface element based on the height information of the laser point cloud in the first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grid of the first laser grid to obtain the second initial surface element includes the following steps 501 to 503: Step 501: Determine the first distance between the laser point cloud in the first laser grid and the laser point cloud in the neighboring laser grid and the first initial surface element, respectively. Step 502: Determine the weight of the height of the laser point cloud in the first laser grid and the weight of the height of the laser point cloud in the neighboring laser grid according to the corresponding first distance. Step 503: Initialize the height information of the first initial surface element according to the height of the laser point cloud in the first laser grid, the height of the laser point cloud in the neighboring laser grid, and the weight of the height of each laser point cloud to obtain the second initial surface element.

[0069] It is understood that in the road reconstruction method provided in this application embodiment, the first distances between the laser point clouds in the first laser grid and the laser point clouds in the neighboring laser grid and the first initial surface element are first determined; then, the weights of the heights of the laser point clouds in the first laser grid and the laser point clouds in the neighboring laser grid are determined according to the corresponding first distances; and the second initial surface element is obtained according to the heights of the laser point clouds in the first laser grid, the heights of the laser point clouds in the neighboring laser grid, and the weights of the heights of each laser point cloud. This eliminates the need to traverse the distances between all laser point clouds globally and the first initial surface element through nearest neighbor search; only the local laser point clouds need to be matched and calculated with the first initial surface element. This helps reduce the computation time during road reconstruction without sacrificing the accuracy of neighbor matching.

[0070] It should be understood that, in the embodiments of this application, determining the first distance between the laser point cloud in the first laser grid and the laser point cloud in the neighboring laser grid and the first initial surface element includes: determining the first distance between the laser point cloud in the first laser grid and the first initial surface element; and determining the first distance between the laser point cloud in each of the neighboring laser grids and the first initial surface element.

[0071] In this embodiment of the application, determining the weight of the height of the laser point cloud in the first laser grid and the weight of the height of the laser point cloud in the neighboring laser grid according to the corresponding first distance includes: determining the weight of the height of the laser point cloud in the first laser grid according to the first distance between the laser point cloud in the first laser grid and the first initial surface element; and determining the weight of the height of the laser point cloud in the corresponding neighboring laser grid according to the first distance between the laser point cloud in each of the neighboring laser grids and the first initial surface element.

[0072] It should be understood that, in this embodiment of the application, the step of initializing the height information of the first initial surface element according to the height of the laser point cloud in the first laser grid, the height of the laser point cloud in the neighboring laser grid, and the weight of the height of each laser point cloud to obtain the second initial surface element includes: normalizing the weight of the height of each laser point cloud to obtain a normalized weight; and initializing the height information of the first initial surface element according to the height of the laser point cloud in the first laser grid, the height of the laser point cloud in the neighboring laser grid, and the normalized weight of each laser point cloud to obtain the second initial surface element.

[0073] In some embodiments, the method further includes: averaging the heights of the road surface laser point clouds belonging to the first laser grid to obtain the height of the laser point cloud in the first laser grid; and averaging the heights of the road surface laser point clouds belonging to the neighboring laser grid to obtain the height of the laser point cloud in the neighboring laser grid.

[0074] In some embodiments, Figure 6 This application provides a schematic diagram of an implementation process for obtaining a second initial surface element. Figure 2 ,like Figure 6 As shown, the step of initializing the height information of the first initial surface element based on the height information of the laser point cloud in the first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grid of the first laser grid to obtain the second initial surface element includes the following steps 601 to 603: Step 601: Determine the first distance between the laser point cloud in the first laser grid and the laser point cloud in the neighboring laser grid and the first initial surface element, respectively. Step 602: Determine the laser grid corresponding to the minimum distance among the multiple first distances as the target laser grid; Step 603: Determine the height of the first initial surface element based on the height information of the laser point cloud in the target laser grid, and obtain the second initial surface element.

[0075] It is understood that in the road reconstruction method provided in this application embodiment, the first distances between the laser point clouds in the first laser grid and the laser point clouds in the neighboring laser grids and the first initial surface element are firstly determined; the laser grid corresponding to the smallest distance among multiple first distances is determined as the target laser grid; based on the height information of the laser point clouds in the target laser grid, the height of the first initial surface element is determined, thus obtaining the second initial surface element. This eliminates the need to traverse the distances between all global laser point clouds and the first initial surface element through nearest neighbor search; only the local laser point clouds need to be matched and calculated with the first initial surface element. This helps reduce the computation time during road reconstruction without sacrificing the accuracy of neighbor matching.

[0076] It should be understood that, in the embodiments of this application, determining the first distance between the laser point cloud in the first laser grid and the laser point cloud in the neighboring laser grid and the first initial surface element includes: determining the first distance between the laser point cloud in the first laser grid and the first initial surface element; and determining the first distance between the laser point cloud in each of the neighboring laser grids and the first initial surface element.

[0077] In some embodiments, the target laser grid includes multiple laser point clouds. The step of determining the height of the first initial surface element and obtaining the second initial surface element based on the height information of the laser point clouds in the target laser grid includes: determining the average height of the laser point clouds in the target laser grid as the height of the first initial surface element and obtaining the second initial surface element.

[0078] In other embodiments, the target laser grid includes multiple laser point clouds. The step of determining the height of the first initial surface element and obtaining the second initial surface element based on the height information of the laser point clouds in the target laser grid includes: determining the weights of the heights of the laser point clouds in the target laser grid according to the corresponding first distances; and initializing the height information of the first initial surface element according to the heights of the laser point clouds in the target laser grid and the weights of the heights of each laser point cloud in the target laser grid to obtain the second initial surface element.

[0079] In some other embodiments, the target laser grid includes a laser point cloud, and the step of determining the height of the first initial surface element and obtaining the second initial surface element based on the height information of the laser point cloud in the target laser grid includes: determining the height of the laser point cloud in the target laser grid as the height of the first initial surface element and obtaining the second initial surface element.

[0080] In some embodiments, the method further includes: averaging the heights of the road surface laser point clouds belonging to the first laser grid to obtain the height of the laser point clouds in the first laser grid; and averaging the heights of the road surface laser point clouds belonging to the neighboring laser grid to obtain the height of the laser point clouds in the neighboring laser grid. Further, in some embodiments, determining the height of the first initial surface element and obtaining the second initial surface element based on the height information of the laser point clouds in the target laser grid includes: determining the height of the laser point clouds in the target laser grid as the height of the first initial surface element to obtain the second initial surface element.

[0081] In some embodiments, initializing the height information of the first initial surface element based on the height information of the laser point cloud in the first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grid of the first laser grid to obtain the second initial surface element includes: determining the average height of the laser point cloud in the first laser grid and the laser point cloud in the neighboring laser grid; determining the average value as the height of the first initial surface element to obtain the second initial surface element.

[0082] It is understood that in the road reconstruction method provided in this application embodiment, the average height of the laser point cloud in the first laser grid and the laser point cloud in the neighboring laser grid is determined as the height of the first initial surface element, thus obtaining the second initial surface element. This eliminates the need to traverse the distances between all global laser point clouds and the first initial surface element through nearest neighbor search. Therefore, it helps reduce the computation time during road reconstruction without sacrificing neighborhood matching accuracy.

[0083] Furthermore, in some embodiments, the method further includes: averaging the heights of the road surface laser point clouds belonging to the first laser grid to obtain the height of the laser point cloud in the first laser grid; and averaging the heights of the road surface laser point clouds belonging to the neighboring laser grid to obtain the height of the laser point cloud in the neighboring laser grid.

[0084] For example, in one possible implementation, the height information of the first initial surface element can be initialized through steps 21 to 24 as follows: Step 21: Pre-build the road surface grid coordinate system of the reconstruction area and assign a unique identifier (ID) to each grid; wherein, the road surface grid is a bird's-eye-view (BEV) / top view grid; Step 22: Segment the ground laser point cloud from the laser point cloud, and calculate the BEV mesh ID of the ground laser point cloud using its three-dimensional coordinates to obtain the laser mesh; Step 23: For the first initial surface element, search from the laser mesh. The search range is the 5×5 laser mesh cluster formed by the current mesh and the adjacent meshes above, below, left, and right. Step 24: Calculate the distance between the point to be matched (i.e., the laser point cloud) and the first initial surface element only within the grid cluster, and select the laser point cloud with the smallest distance. Use the height value of the laser point cloud with the smallest distance as the height value of the first initial surface element. Alternatively, the height values ​​of the laser point clouds in the 25 laser grid clusters can be averaged and used as the height value of the first initial surface element. Alternatively, the height of the laser point cloud can be determined according to the distance, and the result of the weighted average of the height and weight can be used as the height value of the first initial surface element.

[0085] In step 104, the second initial surface element is reconstructed based on the road surface visual features obtained by feature extraction from the camera image, and the reconstructed road surface grid is determined.

[0086] In some embodiments, the step of reconstructing the second initial surface element based on the road surface visual features obtained by feature extraction from the camera image and determining the reconstructed road surface grid includes: reconstructing the second initial surface element based on the road surface visual features obtained by feature extraction from the camera image to obtain the final surface element; and determining the reconstructed road surface grid based on the final surface element corresponding to each camera image.

[0087] The following describes an exemplary application of the embodiments of this application in a real-world application scenario.

[0088] This application provides a method for optimizing the efficiency of road reconstruction algorithms based on sliding window nearest neighbor search instead of global KNN search (i.e., an example of a road reconstruction method). Specifically, it relates to an algorithm efficiency optimization method in a multimodal (vision + laser) road reconstruction scenario (i.e., an example of a road reconstruction method), which is applicable to autonomous driving static environment construction scenarios that need to balance reconstruction accuracy and engineering implementation efficiency.

[0089] Based on the characteristic that "surface elements are arranged in a grid-like pattern in spatial distribution" in road reconstruction scenarios, the 3D reconstruction space is divided into several fixed-size grids of 5×5×H (5 can be customized as needed, and H is the step size in the road height direction. In the field of road reconstruction, H can be ignored and the focus is on the two-dimensional space). When the point to be matched (i.e., the laser point cloud) is input, the neighborhood surface element search is performed only within its own grid and the adjacent 5×5 grids, replacing the traditional global KNN search.

[0090] In some embodiments, the 5×5 grid neighborhood search replacing the global KNN search includes the following steps 21 to 24: Step 21: Pre-build the road surface grid coordinate system of the reconstruction area and assign a unique identifier (ID) to each grid; wherein, the road surface grid is a bird's-eye-view (BEV) / top view grid; Step 22: Segment the ground laser point cloud from the laser point cloud, and calculate the BEV mesh ID of the ground laser point cloud using its three-dimensional coordinates to obtain the laser mesh; Step 23: For the first initial surface element, search from the laser mesh. The search range is the 5×5 laser mesh cluster formed by the current mesh and the adjacent meshes above, below, left, and right. Step 24: Calculate the distance between the point to be matched (i.e., the laser point cloud) and the first initial surface element only within the grid cluster, and select the laser point cloud with the smallest distance. Use the height value of the laser point cloud with the smallest distance as the height value of the first initial surface element. Alternatively, the height values ​​of the laser point clouds in the 25 laser grid clusters can be averaged and used as the height value of the first initial surface element. Alternatively, the height of the laser point cloud can be determined according to the distance, and the result of the weighted average of the height and weight can be used as the height value of the first initial surface element.

[0091] It is understood that in this embodiment of the application, the search range is narrowed from global facets to local facets within 25 grids, and the computational complexity is reduced from O(N) (where N is the number of global facets) to O(1), with a facet count of 10. 5 At the intermediate level, the computation time is reduced by more than 50% without sacrificing neighborhood matching accuracy (error is controlled within ±5 mm).

[0092] In some embodiments, during the multi-frame data training phase, the initial surface element (including parameters such as surface element position, size, and initial color) generated based on the laser ground point in the first frame serves as the common basic surface element for all subsequent frames; subsequent frames only perform local updates based on the common surface element (such as adjusting the surface element precision according to visual features and adding new road surface details), without the need to regenerate the complete initial surface element.

[0093] In some embodiments, the shared initialization element includes the following steps 11 to 15: Step 11: Divide the road surface where the vehicle is located into multiple adjacent grids; Step 12: When the first frame data (i.e., camera image) is input, the data parameters of the face element are initialized through the first frame data, and the core parameters such as the face element coordinates, normal vector, and size of F0 are stored. Step 13: When inputting the data of the i-th frame (i≥2), directly call F0 as the basic surface element, and calculate the pose offset between the i-th frame and the first frame through IMU and camera calibration data; Step 14: Adjust the spatial position of the surface elements in F0 according to the pose offset to obtain the basic surface element F adapted to the i-th frame. i0 ; Step 15: Combining the visual road surface features of the i-th frame (such as road marking texture and crack details), analyze F. i0 Perform local surface element precision optimization to generate the final surface element F of the i-th frame. i No reinitialization is required.

[0094] It is understood that, in the embodiments of this application, sharing initialization elements can reduce the amount of initialization calculation by more than 55%, and the processing time for one iteration of 600 data points is shortened from 236 minutes to 83 minutes; at the same time, due to sharing initialization elements, the consistency of cross-frame elements is improved, and the accuracy of road marking reconstruction is stabilized at over 90%.

[0095] In some embodiments, the algorithm efficiency optimization method for multimodal (visual + laser) road reconstruction scenarios provided in this application includes the following steps 31 to 35: Step 31, Multi-sensor data acquisition: Acquire laser point cloud, camera images, IMU inertial data, and sensor calibration parameters; Step 32, Pose Calculation: Fusing laser point cloud, camera images and IMU data to obtain accurate vehicle motion pose; Step 33, Surface element initialization: The initialization operation of the global Gaussian surface element is reused by multiple cameras, and the initialization surface element F0 is shared. Step 34, Neighborhood search matching: Use 5×5 grid neighborhood search instead of KNN to complete the matching of points to be matched with surface elements; Step 35, Output: Generate a 3D reconstruction model of the road surface, including static information such as road surface smoothness and marking position.

[0096] For example, in one of the possible implementations, the algorithm efficiency optimization method for multimodal (visual + laser) road reconstruction scenarios provided in this application was tested on a dataset containing 100 extreme scenarios (night, rain, fog). The results are shown in Table 1. Table 1

[0097] It is understood that in this embodiment, by replacing the global KNN with a 5×5 grid neighborhood search and combining it with shared initialization elements, the overall reconstruction efficiency is improved by more than 50%, solving the problem of excessively long engineering implementation time. While optimizing efficiency, the reconstruction accuracy is maintained at the centimeter level, and the road marking accuracy reaches more than 90%. It can complement the pure laser solution and solve the problem of insufficient contrast of laser reflection values. It is suitable for multiple road (urban roads, highways) and multiple weather (rainy days, foggy days, night) scenarios, has strong robustness, and can be directly applied to autonomous driving static environment reconstruction business and data closed-loop system.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application, and all such modifications or substitutions should be covered within the protection scope of this application.

[0099] It should be noted that although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that these steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps; or steps from different embodiments may be combined into a new technical solution. Based on the foregoing embodiments, this application provides an apparatus comprising the included modules and the units included in each module, which can be implemented by a processor; of course, it can also be implemented by specific logic circuits; in implementation, the processor can be an AI acceleration engine (such as an NPU), a graphics processing unit (GPU), a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA), etc.

[0100] Figure 7 A schematic diagram of a road reconstruction device provided in an embodiment of this application is shown below. Figure 7 As shown, the road reconstruction device 70 includes: an acquisition module 701, a determination module 702, an initialization module 703, and a reconstruction module 704; wherein, The acquisition module 701 is configured to acquire multimodal sensing data of the road surface where the vehicle is located; wherein, the multimodal sensing data includes: laser point cloud and camera image corresponding to the laser point cloud; The determining module 702 is configured to determine a first initial surface element based on the multimodal sensing data; The initialization module 703 is configured to initialize the height information of the first initial surface element based on the height information of the laser point cloud in the first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grid of the first laser grid, so as to obtain the second initial surface element. The reconstruction module 704 is configured to reconstruct the second initial surface element based on the road surface visual features obtained by feature extraction from the camera image, and determine the reconstructed road surface grid.

[0101] In some embodiments, the initialization module 703 is further configured to determine a first distance between the laser point cloud in the first laser grid and the laser point cloud in the neighboring laser grid and the first initial surface element, respectively; determine the weight of the height of the laser point cloud in the first laser grid and the weight of the height of the laser point cloud in the neighboring laser grid according to the corresponding first distance; initialize the height information of the first initial surface element according to the height of the laser point cloud in the first laser grid, the height of the laser point cloud in the neighboring laser grid and the weight of the height of each laser point cloud, to obtain a second initial surface element.

[0102] In some embodiments, the initialization module 703 is further configured to determine the first distances between the laser point cloud in the first laser grid and the laser point cloud in the neighboring laser grid and the first initial surface element; determine the laser grid corresponding to the smallest distance among the plurality of first distances as the target laser grid; and determine the height of the first initial surface element based on the height information of the laser point cloud in the target laser grid to obtain the second initial surface element.

[0103] In some embodiments, the initialization module 703 is further configured to determine the average height of the laser point cloud in the first laser grid and the laser point cloud in the neighboring laser grid; and to determine the average height as the height of the first initial surface element to obtain the second initial surface element.

[0104] In some embodiments, the determining module 702 is further configured to divide the road surface where the vehicle is located into multiple adjacent grids; determine the first facet corresponding to the grid based on the radius of the grid; for the first frame of the camera image, initialize the visual parameters of the first facet corresponding to the first frame of the camera image based on the first frame of the camera image to obtain the first initial facet corresponding to the first frame of the camera image; for the i-th frame of the camera image, determine the first initial facet corresponding to the i-th frame of the camera image based on the first initial facet corresponding to the i-th frame of the camera image and the first initial facet corresponding to the first frame of the camera image; wherein i is greater than or equal to 2.

[0105] In some embodiments, the determining module 702 is further configured to: determine the motion pose of the vehicle based on the multimodal sensing data; determine a second surface element that overlaps with the first initial surface element of the first frame of the camera image corresponding to the i-th frame; adjust the spatial position of the second surface element based on the first motion pose of the i-th frame of the camera image and the second motion pose of the first frame of the camera image to obtain a third initial surface element; and determine the first initial surface element corresponding to the i-th frame of the camera image based on the third initial surface element.

[0106] In some embodiments, the determining module 702 is further configured to: determine the motion pose of the vehicle based on the multimodal sensing data; determine a third surface element that overlaps with a first initial surface element of a camera image in the i-th frame and a camera image in the (i-1)-th frame; adjust the spatial position of the third surface element based on the first motion pose of the camera image in the i-th frame and the third motion pose of the camera image in the (i-1)-th frame to obtain a fourth initial surface element; and determine the first initial surface element corresponding to the camera image in the i-th frame based on the fourth initial surface element.

[0107] In some embodiments, the neighborhood laser grid includes a laser grid centered on the first laser grid and extending within a first spatial range or a first distance range around the first laser grid.

[0108] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0109] It should be noted that the module division in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical units, or have two or more units integrated into one unit. The integrated units can be implemented in hardware, as software functional units, or a combination of software and hardware.

[0110] It should be noted that, in the embodiments of this application, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a cloud device to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0111] This application provides a cloud device. Figure 8 This is a schematic diagram of the structure of a cloud device provided in an embodiment of this application, such as... Figure 8 As shown, the cloud device 80 includes a memory 801 and a processor 802. The memory 801 stores a computer program that can run on the processor 802. When the processor 802 executes the program, it implements the steps in the method provided in the above embodiments.

[0112] It should be noted that the memory 801 is configured to store instructions and applications executable by the processor 802, and can also cache data to be processed or already processed in the various modules of the processor 802 and the cloud device 80 (e.g., image data, audio data, voice communication data and video communication data), which can be implemented by flash memory or random access memory (RAM).

[0113] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method provided in the above embodiments.

[0114] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the method provided in the above-described method embodiments.

[0115] It should be noted that the descriptions of the storage media and cloud device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage media and cloud device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0116] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0117] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0118] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed cloud devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between cloud devices or modules can be electrical, mechanical, or other forms.

[0120] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0121] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0122] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0123] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium.

[0124] Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a cloud device to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0125] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0126] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0127] The features disclosed in the several methods or cloud device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or cloud device embodiments.

[0128] The above are merely embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A road surface reconstruction method, characterized in that, The method includes: Acquire multimodal sensing data of the road surface where the vehicle is located; wherein, the multimodal sensing data includes: laser point cloud and camera image corresponding to the laser point cloud; Based on the multimodal sensing data, a first initial surface element is determined; Based on the height information of the laser point cloud in the first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grid of the first laser grid, the height information of the first initial surface element is initialized to obtain the second initial surface element. Based on the road surface visual features obtained by feature extraction from the camera image, the second initial surface element is reconstructed to determine the reconstructed road surface grid.

2. The road reconstruction method according to claim 1, characterized in that, The step of initializing the height information of the first initial surface element based on the height information of the laser point cloud in the first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grids of the first laser grid to obtain the second initial surface element includes: Determine the first distance between the laser point cloud in the first laser grid and the laser point cloud in the neighboring laser grid and the first initial surface element; The weights of the heights of the laser point clouds in the first laser grid and the weights of the heights of the laser point clouds in the neighboring laser grid are determined based on the corresponding first distances. The height information of the first initial surface element is initialized based on the height of the laser point cloud in the first laser grid, the height of the laser point cloud in the neighboring laser grid, and the weight of the height of each laser point cloud, to obtain the second initial surface element.

3. The road reconstruction method according to claim 1, characterized in that, The step of initializing the height information of the first initial surface element based on the height information of the laser point cloud in the first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grids of the first laser grid to obtain the second initial surface element includes: Determine the first distance between the laser point cloud in the first laser grid and the laser point cloud in the neighboring laser grid and the first initial surface element; The laser grid corresponding to the smallest distance among the multiple first distances is determined as the target laser grid; Based on the height information of the laser point cloud in the target laser grid, the height of the first initial surface element is determined, and the second initial surface element is obtained.

4. The road reconstruction method according to claim 1, characterized in that, The step of initializing the height information of the first initial surface element based on the height information of the laser point cloud in the first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grids of the first laser grid to obtain the second initial surface element includes: Determine the average height of the laser point cloud in the first laser grid and the laser point cloud in the neighboring laser grid; The average value is determined as the height of the first initial surface element, and the second initial surface element is obtained.

5. The road reconstruction method according to any one of claims 1 to 4, characterized in that, The step of determining the first initial surface element based on the multimodal sensing data further includes: The road surface where the vehicle is located is divided into multiple adjacent grids; Based on the radius of the grid, determine the first face element corresponding to the grid; For the first frame of the camera image, the visual parameters of the first face element corresponding to the first frame of the camera image are initialized based on the first frame of the camera image to obtain the first initial face element corresponding to the first frame of the camera image. For the camera image in the i-th frame, the first initial surface element corresponding to the camera image in the i-th frame is determined based on the first initial surface element corresponding to the camera image in the i-th frame and the camera image in the first frame; where i is greater than or equal to 2.

6. The road reconstruction method according to claim 5, characterized in that, The step of determining the first initial surface element corresponding to the camera image in the i-th frame based on the first initial surface element corresponding to the camera image in the i-th frame and the first frame includes: The motion pose of the vehicle is determined based on the multimodal sensing data; Determine the second face element that overlaps with the first initial face element of the camera image in the first frame; Based on the first motion pose corresponding to the camera image in the i-th frame and the second motion pose corresponding to the camera image in the first frame, the spatial position of the second face element is adjusted to obtain the third initial face element; Based on the third initial surface element, the first initial surface element corresponding to the camera image in the i-th frame is determined.

7. The road reconstruction method according to claim 5, characterized in that, The step of determining the first initial surface element corresponding to the camera image in the i-th frame based on the first initial surface element corresponding to the camera image in the first frame further includes: The motion pose of the vehicle is determined based on the multimodal sensing data; Determine the third face element that overlaps with the first initial face element of the camera image in the (i-1)th frame; Based on the first motion pose corresponding to the camera image in the i-th frame and the third motion pose corresponding to the camera image in the (i-1)-th frame, the spatial position of the third face element is adjusted to obtain the fourth initial face element; Based on the fourth initial surface element, the first initial surface element corresponding to the camera image in the i-th frame is determined.

8. The road reconstruction method according to any one of claims 1 to 4, characterized in that, The neighborhood laser grid includes a laser grid centered on the first laser grid, within a first spatial range or a first distance range around the first laser grid.

9. A road reconstruction device, characterized in that, The device includes: The acquisition module is configured to acquire multimodal sensing data of the road surface where the vehicle is located; wherein, the multimodal sensing data includes: laser point cloud and camera image corresponding to the laser point cloud; The determination module is configured to determine a first initial surface element based on the multimodal sensing data; The initialization module is configured to initialize the height information of the first initial surface element based on the height information of the laser point cloud in the first laser grid corresponding to the first initial surface element and the height information of the laser point cloud in the neighboring laser grid of the first laser grid, so as to obtain the second initial surface element. The reconstruction module is configured to reconstruct the second initial surface element based on the road surface visual features obtained by feature extraction from the camera image, and determine the reconstructed road surface grid.

10. A cloud device, characterized in that, The cloud device includes a memory and a processor, wherein: The memory is used to store computer programs that can run on the processor; The processor is configured to execute the computer program in the memory, performing the method according to claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor according to the method described in claims 1 to 8.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the method described in claims 1 to 8.

Images