Three-dimensional reconstruction method and device of road, storage medium and computer equipment

Abstract

This application discloses a method, apparatus, storage medium, and computer device for three-dimensional road reconstruction. It includes: acquiring a sampled image of the target road and the pose information of the capturing device that captured the sampled image; determining the three-dimensional coordinates of the sampled points in a preset coordinate system based on the pose information, wherein the sampled points are located on the line of sight from the capturing device to the target pixel on a preset mapping plane; calculating the symbolic signature distance of the sampled points relative to the preset mapping plane based on the X and Y axis coordinates in the three-dimensional coordinate system; and rendering the road on the preset mapping plane based on the symbolic signature distance and the color information of the sampled points in the sampled image to form a road model of the target road. This application's embodiment achieves dimensionality reduction operations in three-dimensional reconstruction, which helps save on resources such as video memory, improves road reconstruction efficiency, and can more accurately describe the geometric features of the road scene, enabling the reconstructed road model to better restore road details and achieve a more refined mapping effect.

Description

Technical Field

[0001] This application relates to the field of three-dimensional reconstruction technology, and in particular to a method, apparatus, storage medium, and computer device for three-dimensional reconstruction of roads. Background Technology

[0002] In related technologies, 3D reconstruction is typically based on NERF (Neural Radiance Field) and RoMe (Room Environment Modeling) methods. The NERF approach represents the scene as a 5D neural radiation field, which can synthesize scenes from new perspectives, but it is difficult to extract accurate geometric boundaries; the RoMe approach involves smoothing and interpolation during rendering, making it difficult to restore details. Summary of the Invention

[0003] In view of this, this application provides a method, apparatus, storage medium and computer equipment for three-dimensional reconstruction of roads. By reducing the dimensionality of the distance function, the computational complexity is reduced and resource consumption is decreased while ensuring the accuracy of geometric shape and distance information.

[0004] According to one aspect of this application, a method for three-dimensional reconstruction of a road is provided, comprising:

[0005] Acquire sampled images of the target road and pose information of the capturing device that captured the sampled images;

[0006] Based on the pose information, the three-dimensional coordinates of the sampling point in the preset coordinate system are determined. The sampling point is located on the line of sight from the shooting device to the target pixel on the preset mapping plane. The preset mapping plane is used to represent the road surface of the target road.

[0007] The symbol signature distance of the sampling point relative to the preset mapping plane is calculated based on the X and Y axis coordinates in the three-dimensional coordinate system.

[0008] Based on the symbol signature distance and the color information of the sampling points in the sampled image, road rendering is performed on the preset mapping plane to form a road model of the target road.

[0009] In the aforementioned technical solution, on the one hand, calculating the symbolic signature distance from two-dimensional coordinates simplifies the positional relationship in three-dimensional space into a numerical value (distance) and a symbol (representing direction), achieving dimensionality reduction in 3D reconstruction. This greatly simplifies subsequent calculations and analysis, reduces computational load and data processing complexity, helps save resources such as GPU memory, and improves road reconstruction efficiency. On the other hand, road reconstruction using symbolic signature distance enhances the elevation optimization capability of the road reconstruction model, thereby keenly capturing differences in features such as bumps, depressions, and potholes on the road. This provides more comprehensive road location information, helps to more accurately describe the geometric features of the road landscape, and enables the reconstructed road model to better restore road details, resulting in more refined mapping effects.

[0010] Furthermore, based on the pose information, the three-dimensional coordinates of the sampling points in the preset coordinate system are determined, including:

[0011] Construct a preset mapping plane in a preset coordinate system;

[0012] Based on pose information, the shooting device is mapped onto a preset coordinate system, connecting the shooting device and the target pixel to form a line of sight;

[0013] Multiple sampling points are selected evenly along the line of sight based on a preset number, and the three-dimensional coordinates of the sampling points in the preset coordinate system are determined.

[0014] In the above technical solution, the shooting device is mapped to a preset coordinate system based on pose information to accurately determine its position and orientation in three-dimensional space. Furthermore, selecting a preset number of sampling points evenly along the line of sight ensures comprehensive and balanced sampling of spatial information in that direction, avoiding information omissions or deviations that may result from uneven sampling point distribution. This allows the acquired data to more accurately reflect the actual scene.

[0015] Furthermore, the symbol signature distance of the sampling point relative to the preset mapping plane is calculated based on the X and Y axis coordinates of the sampling point, including:

[0016] The X and Y coordinates of the sampling points are input into the first multilayer perceptron to determine the predicted height value of the sampling points. The first multilayer perceptron is trained based on coordinate points that include height changes. At least part of the feature coordinate points are obtained based on the viewing angle direction of the line of sight and the device parameters of the shooting device.

[0017] The difference between the Z-axis coordinate and the predicted height value in the three-dimensional coordinate system is calculated as the symbol signature distance.

[0018] In the above technical solution, since the road itself has relatively few three-dimensional spatial structures, it is possible to use the X and Y axis coordinates as input while ensuring prediction accuracy. Compared with directly processing three-dimensional coordinates, this reduces the dimensionality of the input data, lowers the computational complexity of the model, and saves computational resources and time.

[0019] Furthermore, the three-dimensional reconstruction methods for roads also include:

[0020] The X and Y coordinates of the sampling points are input into the second multilayer perceptron to determine the predicted correction value of the sampling points. The second multilayer perceptron is trained based on the prior data of the sample road and the feature coordinate points of the sample road.

[0021] Adjust the symbol signature distance based on the predicted correction value.

[0022] In the above technical solution, the symbol signature distance is further adjusted by the predicted correction value output by the second multilayer perceptron, so that the symbol signature distance more accurately reflects the dynamic information of the road environment and accurately measures the distance difference between the sampling point and the real road surface, thereby improving the accuracy of the symbol signature distance as an error metric and providing a more accurate basis for subsequent processing and analysis.

[0023] Furthermore, the three-dimensional reconstruction methods for roads also include:

[0024] During the training of the first or second multilayer perceptron, a loss term for the elevation corresponding to the pose information is added to the loss function of the first or second multilayer perceptron.

[0025] In the above technical solution, the elevation loss term provides additional constraints, enabling the multilayer perceptron to pay more attention to the fitting degree of the elevation data, thus enabling the multilayer perceptron to better adapt to elevation changes in various scenarios, reduce the error between the predicted elevation and the actual elevation, and improve the positioning accuracy of the model in three-dimensional space.

[0026] Furthermore, the three-dimensional reconstruction methods for roads also include:

[0027] During the training of the first or second multilayer perceptron, the model parameters of the first or second multilayer perceptron are optimized based on the second derivative.

[0028] In the above technical solution, by considering the second derivative to optimize model parameters, the multilayer perceptron can automatically adjust the learning rate or step size based on the local curvature of the loss function. This allows for a better determination of whether the current local optimum is the global optimum or a better local optimum. Especially when the surface of the loss function is complex, it can more accurately find the descent direction, thereby accelerating the model's convergence speed, reducing the training time and number of iterations, and contributing to improved model performance and interpretability.

[0029] Furthermore, based on the symbol signature distance and the color information of the sampling points in the sampled image, road rendering is performed on a preset mapping plane to form a road model of the target road, including:

[0030] Obtain the color information of the sampling points based on the sampled image;

[0031] The color weight of the sampling points is determined based on the symbol signature distance;

[0032] The color information of the target pixel is determined by weighting the color information of at least one sampling point on the line of sight based on color weight.

[0033] The target pixels are rendered based on their color information to generate a road model.

[0034] In the above technical solution, during the rendering process, color weights are used to represent the visibility of objects, and the attenuation of light as it passes through these media is accurately calculated to simulate the behavior of light in real scenes, thereby achieving high-quality rendering effects and making the road details in the road model clearer.

[0035] Furthermore, based on the color information of the target pixels, the target pixels are rendered to generate a road model, including:

[0036] The grid cells with the target pixel as the vertex are determined based on the hash index. The grid cells are obtained by dividing the three-dimensional space under the preset coordinate system.

[0037] Linear interpolation is performed on the color information of the target pixel to determine the color information of other pixels in the grid cell besides the target pixel.

[0038] The color information of the target pixel and other pixels is combined to form the color information of the grid unit.

[0039] The road model is obtained by rendering the grid cells based on their color information.

[0040] In the aforementioned technical solution, on the one hand, dividing the grid cells based on the hash grid embedding algorithm not only eliminates the need to traverse the entire 3D space to locate the grid where the target pixel is located, improving positioning efficiency, but also allows different grids to accurately represent complex road surface details such as traffic signs and textures based on their position and function in the road model. On the other hand, using linear interpolation to predict the color information of other pixels can efficiently generate complete color information while ensuring accurate and natural color transitions within the grid cell, precisely restoring the color characteristics of the road. Furthermore, it avoids performing complex lighting and color calculations for each pixel individually, fully utilizing the parallel computing capabilities of modern graphics processing units (GPUs) and significantly improving rendering speed.

[0041] Further, acquiring sampled images of the target road includes:

[0042] Acquire sampled video of the target road;

[0043] The sampled video is segmented based on a preset sampling interval to obtain keyframe images from the sampled video;

[0044] The keyframe image, the vector encoding of the shooting device, and the pose information are input into the image processing model. The image processing model performs color restoration processing on the keyframe image to obtain the sampled image.

[0045] The image processing model is trained using the original image and the color-shifted image. The color-shifted image is obtained by adjusting the color of the original image based on vector encoding and pose information when the original image was captured. The vector encoding is used to represent the color changes caused by different shooting devices.

[0046] The aforementioned technical solution, on the one hand, utilizes the vector encoding and pose information of the shooting device to better adapt to environmental changes such as lighting and dynamic object occlusion, minimizing the use of transient interpretation, improving the robustness of different devices to environmental changes, and reducing the impact of device and environmental factors on image quality. Even when different shooting devices capture the same object in different environments, it can ensure as much consistency in image color as possible. On the other hand, color restoration can improve the visibility of details in the image, facilitating the identification and reconstruction of object and scene details in the image, and providing a more reliable input for subsequent automated systems.

[0047] Furthermore, the three-dimensional reconstruction methods for roads also include:

[0048] Extract the original color features of each pixel in the original image;

[0049] Based on the coordinates of the pixels in the original image and the pose information corresponding to the original image, the viewing angle information of the shooting device relative to the pixels is determined;

[0050] The color offset of a pixel is obtained by processing the viewpoint information and vector encoding based on the third multilayer perceptron. The third multilayer perceptron is trained based on the first sample image of the specified pattern captured in different poses and the vector encoding.

[0051] The original color features of corresponding pixels in the original image are adjusted according to the color offset to form a color-offset image.

[0052] In the aforementioned technical solution, the use of color-shifted images and original images allows the model to better learn color changes caused by different shooting devices and environments, thereby improving the accuracy of image color reproduction. Furthermore, in the automated 3D road reconstruction process, CMLP can reduce manual intervention, decrease the workload and cost of subsequent 3D road reconstruction, and improve the overall efficiency of the processing flow.

[0053] According to another aspect of this application, a three-dimensional reconstruction apparatus for a road is provided, comprising:

[0054] The acquisition module is used to acquire sampled images of the target road and the pose information of the capturing device that captured the sampled images;

[0055] The positioning module is used to determine the three-dimensional coordinates of the sampling point in a preset coordinate system based on the pose information. The sampling point is located on the line of sight from the shooting device to the target pixel on the preset mapping plane. The preset mapping plane is used to represent the road surface of the target road.

[0056] The first processing module is used to calculate the symbol signature distance of the sampling point relative to the preset mapping plane based on the X and Y axis coordinates in the three-dimensional coordinate system.

[0057] The reconstruction module is used to perform road rendering on the preset mapping plane based on the symbol signature distance and the color information of the sampling points in the sampled image, to form a road model of the target road.

[0058] Furthermore, the positioning module is specifically used to construct a preset mapping plane in a preset coordinate system; to map the shooting device onto the preset coordinate system based on pose information, to connect the shooting device and the target pixel point to form a line of sight; and to uniformly select multiple sampling points on the line of sight based on a preset number, and to determine the three-dimensional coordinates of the sampling points in the preset coordinate system.

[0059] Further, the first processing module is specifically used to input the X and Y axis coordinates of the sampling points into the first multilayer perceptron to determine the predicted height value of the sampling points. The first multilayer perceptron is trained based on coordinate points that include height changes, and at least part of the feature coordinate points are obtained based on the viewing angle direction of the line of sight and the device parameters of the shooting device. The difference between the Z axis coordinate and the predicted height value in the three-dimensional coordinate system is calculated as the symbol signature distance.

[0060] Furthermore, the first processing module is also used to input the X and Y axis coordinates of the sampling points into the second multilayer perceptron to determine the prediction correction value of the sampling points. The second multilayer perceptron is trained based on the prior data of the sample road and the feature coordinate points of the sample road. The symbol signature distance is adjusted based on the prediction correction value.

[0061] Furthermore, the three-dimensional reconstruction device for roads also includes:

[0062] The model training module adds a loss term for the elevation corresponding to the pose information to the loss function of the first or second multilayer perceptron during the training process, and / or optimizes the model parameters of the first or second multilayer perceptron based on the second derivative.

[0063] Furthermore, the reconstruction module is specifically used to obtain the color information of sampling points based on the sampled image; determine the color weight of the sampling points based on the symbol signature distance; perform weighted calculation on the color information of at least one sampling point on the line of sight based on the color weight to determine the color information of the target pixel; and perform rendering processing on the target pixel based on the color information of the target pixel to generate a road model.

[0064] Furthermore, the positioning module is also used to determine the grid cells with the target pixel as the vertex based on the hash index. The grid cells are obtained by dividing the three-dimensional space under the preset coordinate system.

[0065] The reconstruction module is specifically used to perform linear interpolation calculations on the color information of the target pixel to determine the color information of other pixels in the grid cell besides the target pixel; and to stitch together the color information of the target pixel and other pixels to form the color information of the grid cell; and to render the grid cell based on the color information of the grid cell to obtain the road model.

[0066] Furthermore, the acquisition module is also used to acquire sampled video of the target road;

[0067] The road 3D reconstruction device also includes:

[0068] The segmentation module is used to segment the sampled video based on a preset sampling interval and obtain keyframe images from the sampled video;

[0069] The second processing module is used to input the keyframe image, the vector encoding of the shooting device and the pose information into the image processing model, and perform color restoration processing on the keyframe image through the image processing model to obtain the sampled image.

[0070] The image processing model is trained using the original image and the color-shifted image. The color-shifted image is obtained by adjusting the color of the original image based on vector encoding and pose information when the original image was captured. The vector encoding is used to represent the color changes caused by different shooting devices.

[0071] Furthermore, the second processing module is also used to extract the original color features of each pixel in the original image; determine the viewing angle information of the shooting device relative to the pixel based on the coordinates of the pixel in the original image and the pose information corresponding to the original image; process the viewing angle information and vector encoding based on the third multilayer perceptron to obtain the color offset of the pixel, which is obtained by training the third multilayer perceptron based on the first sample image of the specified pattern captured by the third multilayer perceptron in different poses and the vector encoding; adjust the original color features of the corresponding pixel in the original image according to the color offset to form a color offset image.

[0072] The three-dimensional reconstruction device for this road possesses all the beneficial effects of the three-dimensional reconstruction method for the aforementioned roads, which will not be elaborated upon further.

[0073] According to another aspect of this application, a computer-readable storage medium is provided, having stored thereon a program or instructions that, when executed by a processor, implement the steps of the aforementioned three-dimensional road reconstruction method. This readable storage medium possesses all the beneficial effects of the aforementioned three-dimensional road reconstruction method, which will not be elaborated further.

[0074] According to another aspect of this application, a computer device is provided, including at least one processor coupled to a memory storing a computer program that runs on the processor. When the processor executes the program, it implements the steps of the aforementioned three-dimensional road reconstruction method. This computer device possesses all the beneficial effects of the aforementioned three-dimensional road reconstruction method, which will not be elaborated further.

[0075] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0076] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0077] Figure 1 A flowchart illustrating the three-dimensional reconstruction method for roads provided in an embodiment of this application is shown.

[0078] Figure 2 A schematic diagram of sampling points provided in an embodiment of this application is shown;

[0079] Figure 3 This paper shows a structural block diagram of a road three-dimensional reconstruction device provided in an embodiment of this application;

[0080] Figure 4 A structural block diagram of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0081] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0082] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0083] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “attached” to another element, it can be directly connected or attached to the other element, or there may be intermediate elements. Furthermore, “connected” or “attached” as used herein can include wireless connections or wireless interconnections. The term “and / or” as used herein includes all or any unit and all combinations of one or more associated listed items.

[0084] Exemplary embodiments according to this application will now be described in more detail with reference to the accompanying drawings. However, these exemplary embodiments may be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein. It should be understood that these embodiments are provided so that the disclosure of this application is thorough and complete, and that the concept of these exemplary embodiments is fully conveyed to those skilled in the art.

[0085] This embodiment provides a method for three-dimensional reconstruction of roads, such as... Figure 1 As shown, the method includes:

[0086] Step 101: Obtain the sampled image of the target road and the pose information of the camera that captured the sampled image.

[0087] The capturing device can be mounted on terminal devices such as mobile phones, cameras, and vehicle control systems; this application embodiment does not impose specific limitations. It is understood that the road 3D reconstruction method provided in this application embodiment is generally executed by a terminal device, and correspondingly, the road 3D reconstruction device is generally installed in the terminal device. However, those skilled in the art will readily understand that the road 3D reconstruction method provided in this embodiment can also be executed by a server, and correspondingly, the road 3D reconstruction device can also be installed in the server; this exemplary embodiment does not impose special limitations on this. For example, a user can upload a captured sample image to a server via a terminal device. The server then generates a road model of the target road using the road 3D reconstruction method provided in this embodiment and transmits the road model to the terminal device for display.

[0088] In specific application scenarios, the sampled image can be an image captured by a camera or an image segmented from a video captured by the camera. Pose information includes the camera's rotation matrix (elevation, yaw, roll angle), pitch, and translation vectors (position coordinates), etc.

[0089] In one embodiment, obtaining a sampled image of the target road in step 101 specifically includes the following steps:

[0090] Step 101-1: Obtain the sampled video of the target road.

[0091] Step 101-2: Segment the sampled video based on a preset sampling interval to obtain keyframe images from the sampled video.

[0092] Step 101-3: Input the keyframe image, the vector encoding of the shooting device, and the pose information into the image processing model. The image processing model performs color restoration processing on the keyframe image to obtain the sampled image.

[0093] The image processing model is trained using the original image and the color-shifted image. The color-shifted image is obtained by adjusting the color of the original image based on the vector encoding and the pose information when the original image was captured. The vector encoding is used to represent the color changes caused by different shooting devices.

[0094] In this embodiment, the color of the original image is pre-adjusted based on the original image, the vector encoding corresponding to the shooting device, and the pose information when the sampled image was captured, to obtain a color-shifted image for training the 3D reconstruction model of the road. When it is necessary to restore the color of the sampled image, the 3D reconstruction model of the road can be used to identify the deviations that the current shooting environment of the shooting device may cause to the image color. The color deviation is then used to perform color restoration processing on the sampled image, and a sampled image closer to the real color is output. On the one hand, the vector encoding and pose information of the shooting device better adapt to environmental changes such as lighting and dynamic object occlusion, avoiding the use of transient interpretation as much as possible, improving the robustness of different devices to environmental changes, and reducing the impact of device and environmental factors on image quality. Even if different shooting devices capture the same object in different environments, the consistency of image color can be guaranteed as much as possible. On the other hand, color restoration can improve the visibility of details in the image, making it easier to identify and reconstruct object and scene details in the image, providing a more reliable input for subsequent automated systems.

[0095] Specifically, the vector code can be obtained by the following method: obtaining a second sample image of the specified pattern captured by the shooting device and the device parameters of the shooting device; mapping the second sample image to the RGB color space to obtain the second sample color features of the second sample image; encoding the second difference color features between the second sample color features and the standard color features of the specified pattern according to the device parameters to generate the vector code.

[0096] The equipment parameters of the shooting device include focal length, aperture, lens type, sensor type, resolution, and pixel size.

[0097] In this embodiment, color features are obtained by mapping the second sample image to the RGB color space, enabling an intuitive and standardized description of image colors. Furthermore, the color features of the specified pattern are known, i.e., standard color features. By comparing the color features of the second sample image with the standard color features of the specified pattern, the potential second color difference features present when the image was captured by the capturing device can be determined. The second color difference features generated by the second sample image relative to the specified pattern are encoded using the device parameters of the capturing device to generate a vector code for that device. Thus, the vector code can be used to interpret color variations caused by different capturing devices. This reduces color deviations caused by device differences (such as sensor characteristics, lighting conditions, etc.), improving the accuracy and consistency of colors in the image.

[0098] Furthermore, the original image is color-adjusted based on the vector encoding corresponding to the shooting device and the pose information when the sampled image was captured. Specifically, this includes: extracting the original color features of each pixel in the original image; determining the viewing angle information of the shooting device relative to the pixel based on the coordinates of the pixel in the original image and the pose information corresponding to the original image; processing the viewing angle information and vector encoding based on a third multilayer perceptron to obtain the color offset of the pixel, which is trained based on the first sample image of the specified pattern captured by the third multilayer perceptron in different poses and the vector encoding; and adjusting the original color features of the corresponding pixel in the original image based on the color offset to form a color-shifted image.

[0099] The original image serves as a reference for defining color features and can be any historical image sampled by the capturing device. Viewpoint information includes the field of view, focal length, and depth of field. The third multilayer perceptron is the CMLP (Color Multilayer Perceptron), used for color-related tasks (e.g., color classification, color recognition, color prediction). The CMLP is a multilayer structure composed of multiple neurons. Its basic structure includes an input layer, one or more hidden layers, and an output layer. Each layer consists of multiple neurons, each generating an output by weighted summation of input values ​​followed by an activation function. By learning the mapping relationship between input data and output results, the CMLP can handle more complex nonlinear relationships and learn more abstract feature representations. The third multilayer perceptron can be trained using multiple first sample images of a specified pattern captured by the capturing device in different poses and vector encodings. Specifically: Multiple first sample images of a specified pattern are acquired by an imaging device in different poses; the viewing angle information of the imaging device relative to the same pixel in the multiple first sample images is determined; the first sample images are mapped to the RGB color space to obtain the first sample color features; the first sample color features are adjusted according to vector encoding, and the first difference color feature between the adjusted first sample color feature and the standard color feature of the specified pattern is determined; the color multilayer perceptron is trained using gradient descent based on the first difference color feature corresponding to the same pixel in the multiple first sample images, the viewing angle information, and the vector encoding. This effectively improves the color multilayer perceptron's learning ability and prediction accuracy for color shifts, allowing the trained color multilayer perceptron to better adapt to different shooting scenarios and devices, exhibiting high versatility and easy application in various 3D reconstruction software or systems requiring color correction, eliminating color deviations caused by different lighting or imaging device differences, and ensuring that the acquired color features are closer to the standard color features.

[0100] Specifically, color features can use the RGB color space as the color standard. The RGB color space obtains various colors by varying the red (R), green (G), and blue (B) color channels and superimposing them. RGB represents the colors of the red, green, and blue channels. In this space, each pixel's color is represented by the values ​​of these three channels, typically ranging from 0 to 255. For example, an RGB value of (255, 0, 0) indicates that this pixel is pure red. The values ​​of each pixel in these three channels can be directly read as part of the color feature.

[0101] In this embodiment, a unified three-dimensional spatial coordinate system is established, and the imaging device and the original image are mapped to this three-dimensional spatial coordinate system according to the pose information. The three-dimensional spatial coordinates of each pixel are calculated based on its two-dimensional pixel coordinates in the original image. Finally, the viewing angle information of the imaging device relative to the pixel can be solved using the pixel and the imaging device's coordinates. After the viewing angle information and vector encoding are input into the CMLP, the CMLP uses the viewing angle information and vector encoding to determine the color shift caused by the device itself and the color shift caused by the viewing angle position, and outputs the total color shift of the pixels. Then, pixel by pixel, the original color features of each pixel in the original image are adjusted one by one using the color shift. This transforms the original image into a color-shifted image that may appear due to environmental influences. This allows the model to better learn the color changes caused by different imaging devices and shooting environments by using the color-shifted image and the original image, thereby improving the accuracy of image color restoration. Furthermore, in the automated road 3D reconstruction process, CMLP can reduce manual intervention, reduce the workload and cost of subsequent road 3D reconstruction, and improve the efficiency of the overall processing flow.

[0102] In one embodiment, the training method of the image processing model is as follows: inputting the color-shifted image into a preset neural network model to obtain a predicted image; determining the predicted pixel offset of the predicted image relative to the original image in units of pixels; constructing a loss function with the goal of minimizing the predicted pixel offset; updating the model parameters of the preset neural network model according to the loss function until the preset neural network model converges, and then outputting the image processing model.

[0103] In this embodiment, the predicted pixels between the predicted image and the original image are calculated on a pixel-by-pixel basis. A loss function aimed at minimizing the predicted pixel offset is used to iteratively train a pre-defined neural network model, continuously updating the model parameters until convergence. On one hand, using pixel-level predicted pixel offsets as model training data allows the trained 3D road reconstruction model to clearly understand the prediction deviation in each tiny region, thus improving color reproduction when dealing with images containing complex details. On the other hand, the loss function aimed at minimizing predicted pixel offsets ensures the model focuses on reducing color shifts and noise, allowing the confidence of the sampled image after color reproduction processing by the 3D road reconstruction model to decrease along its RGB loss, generating an image closer to the real scene.

[0104] For example, first, a learnable vector code for the vehicle camera is obtained. In LaTeX, this vector code can be represented as `rgb = color_mlp(f, camera_enc})`, where `mlp(f)` represents the operation function of the color multilayer perceptron. When training the 3D reconstruction model of the road, the original image is used as a basis, and the color features of the original image are adjusted using the vector code corresponding to the shooting device and the shooting angle to obtain a color-shifted image. At this time, the color features of each pixel in the color-shifted image can be represented as `rgb = orig_color(f) + delta_color(f, x, y, viewdir})`.

[0105] The model is defined as follows: `\text{camera enc}`, where `orig_color(f)` represents the intrinsic color features of the original image, and `delta_color(f,x,y,\text{view dir},\text{camera enc})` represents the color offset of the color-shifted image at a given pixel coordinate. The model is then trained using both the original image and the color-shifted image as samples, while a loss function is added to make `delta_color` approach zero, allowing the model to be interpreted primarily by `orig_color`. Thus, the trained 3D road reconstruction model outputs only sampled images containing `orig_color`.

[0106] Step 102: Based on the pose information, determine the three-dimensional coordinates of the sampling points in the preset coordinate system.

[0107] The preset coordinate system can be a standard world coordinate system or a user-defined coordinate system, so that data from different sources and at different times can be processed and analyzed under the same spatial reference. The preset coordinate system includes a preset mapping plane that serves as the basis for the road model; this preset mapping plane represents the road surface of the road model. Sampling points are located on the line of sight from the imaging device to the target pixel on the preset mapping plane.

[0108] In this embodiment, the camera is transformed into a preset coordinate system using the geometric relationships of the camera and known image intrinsic parameters. A line-of-sight sampling point is then selected between the camera and the target pixel on the preset mapping plane. This allows the line-of-sight sampling point to better reflect the relative positional relationship between the sampling point and the target pixel on the preset mapping plane, ensuring that subsequent road reconstruction fully considers location elevation and that the reconstructed road model better reproduces road details. Furthermore, using the known preset mapping plane as the sampling basis simplifies the geometric structure, streamlines the line-of-sight sampling process, and improves training speed.

[0109] In one embodiment, step 102, namely determining the three-dimensional coordinates of the sampling point in the preset coordinate system based on the pose information, specifically includes the following steps:

[0110] Step 102-1: Construct a preset mapping plane in the preset coordinate system.

[0111] Step 102-2: Based on the pose information, map the shooting device onto a preset coordinate system, connect the shooting device and the target pixel to form a line of sight.

[0112] Understandably, the target pixel can be the vertex of a 3D spatial grid cell obtained based on a hash grid embedding algorithm. This facilitates the rapid location of a certain position in 3D space through hash index during subsequent rendering, and makes it easier to optimize and calculate the geometric shape and appearance features of the scene, thus achieving efficient and accurate 3D scene processing.

[0113] Step 102-3: Select multiple sampling points evenly along the line of sight based on a preset number, and determine the three-dimensional coordinates of the sampling points in a preset coordinate system.

[0114] The preset number can be set reasonably according to the detection accuracy. It is worth mentioning that since the preset mapping plane is known, more spatial references can be obtained. While ensuring the accuracy of road reconstruction, the number of sampling points can be appropriately reduced when calculating spatially related information.

[0115] In this embodiment, a preset mapping plane is constructed under a preset coordinate system to serve as the basis for road reconstruction. The imaging device is mapped onto the preset coordinate system based on pose information to accurately determine its position and orientation in three-dimensional space. After determining the target pixels on the preset mapping plane, a line of sight is formed by connecting the optical center of the imaging device and the target pixels. Multiple sampling points are then uniformly selected along this line of sight to obtain the spatial distribution of the road scene in a specific direction. This ensures comprehensive and balanced sampling of spatial information along this line of sight, avoiding information omissions or deviations that may result from uneven sampling point distribution, and making the acquired data more accurately reflect the actual scene.

[0116] For example, such as Figure 2 As shown, after determining the preset mapping plane, the shooting device is mapped onto the preset coordinate system. The shooting device is represented by a "triangle" in the figure. A ray is emitted from the shooting device to the preset mapping plane to represent the line of sight of the shooting device to the target pixel P. Five sampling points (O) are selected along the line of sight. 1～5 ).

[0117] Step 103: Calculate the symbol signature distance of the sampling point relative to the preset mapping plane based on the X and Y axis coordinates in the three-dimensional coordinate system.

[0118] The symbolic signature distance is defined as the nearest distance from a sampling point to a preset mapping plane, and is assigned a positive or negative sign depending on whether the sampling point is inside or outside the preset mapping plane (generally, a distance value of a point outside the preset mapping plane is positive, a distance value of a point inside the preset mapping plane is negative, and a distance value of a point on the surface of the preset mapping plane is 0). The symbolic signature distance can be calculated using the symbolic distance function (SDF).

[0119] In this embodiment, the symbolic signature distance of the sampling point relative to the preset mapping plane is calculated based on the two-dimensional coordinates (X and Y axis coordinates) of the sampling point. The road surface is expressed through the symbolic signature distance, which facilitates the regression of geometric position using volume rendering. This improves the elevation optimization capability of the road reconstruction model, and the system can better capture the differences in features such as bumps, depressions, and potholes on the road. This provides more comprehensive road position information, helps to more accurately describe the geometric features of the road scene, and enables the reconstructed road model to better restore road details, resulting in a more refined mapping effect.

[0120] In one embodiment, step 103, namely calculating the symbol signature distance of the sampling point relative to the preset mapping plane based on the X and Y axis coordinates in three-dimensional coordinates, specifically includes the following steps:

[0121] Step 103-1: Input the X and Y coordinates of the sampling points into the first multilayer perceptron to determine the predicted height value of the sampling points.

[0122] The first multilayer perceptron is trained based on coordinate points that include height variations. This first multilayer perceptron can be an SDF MLP (Signed Distance Function Multilayer Perceptron). Specifically, some coordinate points with height variations are selected. These coordinate points can be pre-defined spatial coordinate points, point cloud data detected by radar sensors, or feature coordinate points of previously detected sample roads obtained using the method in step 102. The first multilayer perceptron is iteratively trained using the X and Y coordinates of these coordinate points as samples and the Z-axis coordinate as labels, enabling it to learn the potential relationship between two-dimensional coordinates and height values.

[0123] Furthermore, during the training of the first multilayer perceptron, the first multilayer perceptron can be optimized in the following ways:

[0124] Method 1: Add a loss term for the elevation corresponding to the pose information to the loss function of the first multilayer perceptron.

[0125] In this embodiment, an additional constraint is provided by the elevation loss term, which makes the first multilayer perceptron pay more attention to the fitting degree of the elevation data, enabling the first multilayer perceptron to better adapt to elevation changes in various scenarios, reduce the error between the predicted elevation and the actual elevation, and improve the positioning accuracy of the model in three-dimensional space.

[0126] Method 2: During the training of the first multilayer perceptron, optimize the model parameters of the first or second multilayer perceptron based on the second derivative.

[0127] In this embodiment, the second derivative contains curvature information of the function. By considering the second derivative to optimize model parameters, the first multilayer perceptron can automatically adjust the learning rate or step size based on the local curvature of the loss function. For example, in regions where the loss function changes gently, the step size is appropriately increased to accelerate convergence; in regions where the loss function changes drastically, the step size is decreased to avoid missing the optimal solution. This allows for a better determination of whether the current local optimum is the global optimum or a better local optimum. Especially when the surface of the loss function is complex, it can more accurately find the descent direction, thereby accelerating the model's convergence speed, reducing the training time and number of iterations, and contributing to improved model performance and interpretability.

[0128] It should be noted that at least part of the training samples (coordinate points) for the first multilayer perceptron are obtained based on the viewing angle and camera parameters. For example, during model training, there is a 1 / 5 probability that data is obtained using the same viewing angle and camera parameters as during output. In this way, the first multilayer perceptron can access conditions that are closer to the real output scene, which helps it learn general features and patterns under different viewing angles and camera settings. This helps reduce fluctuations and uncertainties during training, making it easier for the model to converge to a better state, improving training efficiency and success rate, and thus better adapting to various practical application scenarios and improving generalization ability.

[0129] Step 103-2: Calculate the difference between the Z-axis coordinate and the predicted height value in the three-dimensional coordinate system as the symbol signature distance.

[0130] Specifically, the symbolic distance function of the first multilayer perceptron is expressed as: SDF(x,y,z) = z - mlp(x,y), where SDF(x,y,z) is the symbolic signature distance, and mlp(x,y) is the predicted height value calculated by the first multilayer perceptron. Figure 2 As shown, a ray is emitted from the imaging device to the preset mapping plane to represent the line of sight of the imaging device to the target pixel P. Five sampling points (O) are selected on the line of sight. 1～5 The distance between the symbol signatures of sampling point O3 and the symbol signature is a.

[0131] In this embodiment, the three-dimensional spatial construction problem is transformed into prediction on a two-dimensional plane. The predicted position is then compared with the actual Z-axis value in the three-dimensional coordinate system to calculate the symbolic signature distance, reflecting the difference between the actual and predicted positions of the sampling points in the height direction. Since the road itself has relatively little three-dimensional spatial structure, X and Y axis coordinates can be used as input while maintaining prediction accuracy. Compared to directly processing three-dimensional coordinates, this reduces the dimensionality of the input data, lowers the computational complexity of the model, and saves computational resources and time.

[0132] Furthermore, in real-world road scenarios, the terrain can be quite complex, and relying solely on the height values ​​predicted by the first multilayer perceptron to calculate the symbol signature distance may not accurately reflect the true situation of the sampling points. Following step 103, the 3D road reconstruction method further includes: inputting the X and Y coordinates of the sampling points into the second multilayer perceptron to determine the prediction correction values ​​for the sampling points; and adjusting the symbol signature distance based on the prediction correction values.

[0133] The second multilayer perceptron is trained based on prior data of the sample roads and feature coordinate points of the sample roads. The prior data includes actual parameters of the sample roads, such as road curvature, slope, self-pose data, radar point cloud data, etc., which will not be listed one by one in this embodiment.

[0134] Specifically, the symbolic distance function of the second multilayer perceptron is expressed as: SDF(x,y,z)=z-mlp(x,y)-mlp'(x,y). Where SDF(x,y,z) is the symbolic signature distance, mlp(x,y) is the predicted height value calculated by the first multilayer perceptron, and mlp'(x,y) is the predicted correction value calculated by the second multilayer perceptron.

[0135] In this embodiment, the prior data and feature coordinates of the sample road reflect the actual scene and geometric features of the sample road. The second multilayer perceptron, trained using the prior data and feature coordinates of the sample road, learns some general characteristics and patterns about the road surface, making its output closer to the real road conditions. Thus, the predicted correction value output by the second multilayer perceptron further adjusts the symbolic signature distance, making it more accurately reflect the dynamic information of the road environment and precisely measure the distance difference between the sampling point and the real road surface. This improves the accuracy of the symbolic signature distance as an error metric, providing a more accurate basis for subsequent processing and analysis.

[0136] It is understandable that, based on the same principle as training the first multilayer perceptron, the training of the second multilayer perceptron can also be optimized by adding an elevation loss term and / or the second derivative of the loss function, as will not be elaborated in the embodiments of this application.

[0137] Step 104: Based on the symbol signature distance and the color information of the sampling points in the sampled image, road rendering is performed on the preset mapping plane to form a road model of the target road.

[0138] The 3D road reconstruction method provided in this application utilizes the geometric relationship of the imaging device and known image intrinsic parameters to transform the imaging device into a preset coordinate system, and selects a sampling point on the line of sight of the target pixel on the preset mapping plane. Based on the two-dimensional coordinates (X and Y axes) of this sampling point, the symbolic signature distance relative to the preset mapping plane is calculated, reflecting the relative positional relationship between the sampling point and the target pixel. Finally, using the symbolic signature distance and the color information of the sampling point in the sampled image, road rendering is performed on the preset mapping plane to construct a 3D road model of the target road. On the one hand, calculating the symbolic signature distance from two-dimensional coordinates simplifies the positional relationship in 3D space to a numerical value (distance) and a symbol (representing direction), achieving dimensionality reduction in 3D reconstruction, greatly simplifying subsequent calculations and analysis, reducing computational load and data processing complexity, helping to save resources such as video memory, and improving road reconstruction efficiency. On the other hand, road reconstruction using symbolic signature distance can improve the elevation optimization capability of the road reconstruction model, thereby keenly capturing the differences in features such as bumps, depressions, and potholes on the road, providing more comprehensive road location information, helping to more accurately describe the geometric features of the road landscape, and enabling the reconstructed road model to better restore road details and achieve more refined mapping results.

[0139] In one embodiment, step 104, namely, based on the symbol signature distance and the color information of the sampling points in the sampled image, performs road rendering on a preset mapping plane to form a road model of the target road, specifically includes the following steps:

[0140] Step 104-1: Obtain the color information of the sampling points based on the sampled image.

[0141] Specifically, the sampling points are mapped onto the camera coordinate system using the pose information of the shooting device, so that they can be projected onto the sampled image. The color information of the pixels projected onto the sampled image is used as the color information of the sampling point, thus ensuring that the road model constructed subsequently is highly consistent with the actual situation in terms of color.

[0142] Step 104-2: Determine the color weight of the sampling points based on the symbol signature distance.

[0143] Specifically, the color weights are calculated using the following formula:

[0144]

[0145] In the formula, α i It refers to the opacity of the sampling point, i.e., the color weight; f(p(t) i )) is the distance from the i-th point on the line of sight to the nearest surface, which is also the symbol signature distance, Φ sIt is the default Sigmoid function.

[0146] Step 104-3: Perform weighted calculation on the color information of at least one sampling point on the line of sight based on color weight to determine the color information of the target pixel.

[0147] Step 104-4: Render the target pixels based on their color information to generate a road model.

[0148] In this embodiment, color weights are used to represent the color changes of a road when viewed from different distances. When rendering a target pixel, the color information of the target pixel is calculated using the color information and color weights of multiple sampling points along the viewpoint corresponding to that target pixel. This target pixel color information is then used for rendering. After rendering all target pixels, a complete road visual model is formed. Thus, during the rendering process, color weights are used to represent the visibility of objects and accurately calculate the attenuation of light as it passes through these media, simulating the behavior of light in real-world scenes and achieving high-quality rendering effects, making road details in the road model clearer.

[0149] In practical application scenarios, step 104-4 specifically includes: determining the grid cell with the target pixel as the vertex based on the hash index; performing linear interpolation calculation on the color information of the target pixel to determine the color information of other pixels in the grid cell besides the target pixel; splicing the color information of the target pixel and other pixels to form the color information of the grid cell; rendering the grid cell based on the color information of the grid cell to obtain the road model.

[0150] The grid cells are obtained by dividing a three-dimensional space under a preset coordinate system.

[0151] In this embodiment, when the 3D space is divided into grids based on a hash grid embedding algorithm, the grid cell containing the target pixel is first quickly located using a hash index. Linear interpolation is then performed on the color information of the target pixel to utilize the spatial linear relationship between pixels and deduce the color information of other pixels within the grid cell based on the known color of the target pixel. After obtaining the color information of the target pixel and other pixels, the color information of the corresponding grid cell is fused. Finally, color rendering is performed using the grid cell as the basic unit to obtain the final road model. On one hand, dividing the grid cells based on the hash grid embedding algorithm not only eliminates the need to traverse the entire 3D space to locate the grid containing the target pixel, improving positioning efficiency, but also allows different grids to accurately represent complex traffic signs, textures, and other road surface details based on their position and function in the road model. On the other hand, using linear interpolation to predict the color information of other pixels ensures accurate and natural color transitions within the grid cell while efficiently generating complete color information, accurately restoring the color characteristics of the road. Furthermore, it avoids complex lighting and color calculations for each pixel individually, fully utilizing the parallel computing capabilities of modern graphics processing units (GPUs) and significantly improving rendering speed.

[0152] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0153] Furthermore, such as Figure 3 As shown, as a specific implementation of the above-mentioned three-dimensional reconstruction method for roads, this application provides a three-dimensional reconstruction device 300 for roads, which includes: an acquisition module 301, a positioning module 302, a first processing module 303, and a reconstruction module 304.

[0154] The acquisition module 301 is used to acquire the sampled image of the target road and the pose information of the capturing device that captured the sampled image;

[0155] The positioning module 302 is used to determine the three-dimensional coordinates of the sampling point in the preset coordinate system based on the pose information. The sampling point is located on the line of sight from the shooting device to the target pixel on the preset mapping plane. The preset mapping plane is used to represent the road surface of the target road.

[0156] The first processing module 303 is used to calculate the symbol signature distance of the sampling point relative to the preset mapping plane based on the X and Y axis coordinates in the three-dimensional coordinate system.

[0157] The reconstruction module 304 is used to render roads on a preset mapping plane based on the symbol signature distance and the color information of the sampling points in the sampled image, forming a road model of the target road.

[0158] Furthermore, the positioning module 302 is specifically used to construct a preset mapping plane in a preset coordinate system; map the shooting device onto the preset coordinate system based on the pose information, connect the shooting device and the target pixel to form a line of sight; uniformly select multiple sampling points on the line of sight based on a preset number, and determine the three-dimensional coordinates of the sampling points in the preset coordinate system.

[0159] Further, the first processing module 303 is specifically used to input the X and Y axis coordinates of the sampling point into the first multilayer perceptron to determine the predicted height value of the sampling point. The first multilayer perceptron is trained based on coordinate points that include height changes. At least part of the feature coordinate points are obtained based on the viewing angle direction of the line of sight and the device parameters of the shooting device. The difference between the Z axis coordinate and the predicted height value in the three-dimensional coordinate system is calculated as the symbol signature distance.

[0160] Furthermore, the first processing module 303 is also used to input the X and Y axis coordinates of the sampling points into the second multilayer perceptron to determine the prediction correction value of the sampling points. The second multilayer perceptron is trained based on the prior data of the sample road and the feature coordinate points of the sample road. The symbol signature distance is adjusted based on the prediction correction value.

[0161] Furthermore, the road 3D reconstruction device 300 also includes: a model training module (not shown in the figure);

[0162] The model training module adds a loss term for the elevation corresponding to the pose information to the loss function of the first or second multilayer perceptron during the training process, and / or optimizes the model parameters of the first or second multilayer perceptron based on the second derivative.

[0163] Furthermore, the reconstruction module 304 is specifically used to obtain the color information of the sampling points based on the sampled image; determine the color weight of the sampling points based on the symbol signature distance; perform weighted calculation on the color information of at least one sampling point on the line of sight based on the color weight to determine the color information of the target pixel; and perform rendering processing on the target pixel based on the color information of the target pixel to generate a road model.

[0164] Furthermore, the positioning module 302 is also used to determine the grid unit with the target pixel as the vertex based on the hash index. The grid unit is obtained by dividing the three-dimensional space under the preset coordinate system.

[0165] The reconstruction module 304 is specifically used to perform linear interpolation calculation on the color information of the target pixel to determine the color information of other pixels in the grid cell besides the target pixel; and to splice the color information of the target pixel and other pixels to form the color information of the grid cell; and to render the grid cell based on the color information of the grid cell to obtain the road model.

[0166] Furthermore, the three-dimensional reconstruction device 300 for roads also includes: a segmentation module (not shown in the figure) and a second processing module (not shown in the figure);

[0167] The acquisition module is also used to acquire sampled video of the target road;

[0168] The segmentation module is used to segment the sampled video based on a preset sampling interval and obtain keyframe images from the sampled video;

[0169] The second processing module is used to input the keyframe image, the vector encoding of the shooting device and the pose information into the image processing model, and perform color restoration processing on the keyframe image through the image processing model to obtain the sampled image.

[0170] The image processing model is trained using the original image and the color-shifted image. The color-shifted image is obtained by adjusting the color of the original image based on vector encoding and pose information when the original image was captured. The vector encoding is used to represent the color changes caused by different shooting devices.

[0171] Furthermore, the second processing module is also used to extract the original color features of each pixel in the original image; determine the viewing angle information of the shooting device relative to the pixel based on the coordinates of the pixel in the original image and the pose information corresponding to the original image; process the viewing angle information and vector encoding based on the third multilayer perceptron to obtain the color offset of the pixel, which is obtained by training the third multilayer perceptron based on the first sample image of the specified pattern captured by the third multilayer perceptron in different poses and the vector encoding; adjust the original color features of the corresponding pixel in the original image according to the color offset to form a color offset image.

[0172] Specific limitations regarding the 3D road reconstruction device can be found in the limitations of the 3D road reconstruction method described above, and will not be repeated here. Each module in the aforementioned 3D road reconstruction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0173] Based on the above, Figure 1Accordingly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method. Figure 1 The method for three-dimensional reconstruction of the road shown.

[0174] Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive), and includes several instructions to cause a computer device (such as a personal computer, server, or network device) to execute the methods described in the various implementation scenarios of this application.

[0175] Based on the above, Figure 1 The method shown, and Figure 3 The virtual device embodiment shown is designed to achieve the above objectives, such as... Figure 4 As shown, this application embodiment also provides a computer device 400, which includes at least one processor 401. Optionally, the computer device 400 further includes a memory 402, which stores a program or instructions executable on the processor 401. When the program or instructions are executed by the processor 401, they implement the above-described... Figure 1 The method for three-dimensional reconstruction of the road shown.

[0176] The memory 402 can be used to store software programs and various data. The memory 402 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 402 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 402 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0177] Processor 401 may include one or more processing units; optionally, processor 401 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 401.

[0178] Computer equipment can specifically include personal computers, servers, network devices, etc.

[0179] Optionally, the computer device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Bluetooth interfaces, Wi-Fi interfaces), etc.

[0180] Those skilled in the art will understand that the computer device structure provided in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0181] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages and stores the hardware and software resources of a computer device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software within the physical device.

[0182] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platform, or by hardware implementation to acquire the sampling image of the target road and the pose information of the shooting device that captures the sampling image; based on the pose information, the three-dimensional coordinates of the sampling point in the preset coordinate system are determined, the sampling point is located on the line of sight from the shooting device to the target pixel on the preset mapping plane, the preset mapping plane is used to represent the road surface of the target road; based on the X and Y axis coordinates in the three-dimensional coordinate system, the symbolic signature distance of the sampling point relative to the preset mapping plane is calculated; based on the symbolic signature distance and the color information of the sampling point in the sampling image, road rendering is performed on the preset mapping plane to form the road model of the target road. In the embodiments of this application, the shooting device is transformed to the preset coordinate system using the geometric relationship of the shooting device and the known image intrinsic parameters, and the sampling point on the line of sight from the shooting device to the target pixel on the preset mapping plane is selected. The symbolic signature distance of the sampling point relative to the preset mapping plane is calculated according to the two-dimensional coordinates (X and Y axis coordinates) of the sampling point, so as to reflect the relative positional relationship between the sampling point and the target pixel through the symbolic signature distance. Finally, by using the symbolic signature distance and the color information of the sampling points in the sampled image, road rendering is performed on the preset mapping plane to construct a 3D road model of the target road. On one hand, calculating the symbolic signature distance from two-dimensional coordinates simplifies the positional relationships in 3D space to a single numerical value (distance) and a symbol (representing direction), achieving dimensionality reduction in 3D reconstruction. This greatly simplifies subsequent calculations and analysis, reduces computational load and data processing complexity, helps save on memory usage, and improves road reconstruction efficiency. On the other hand, road reconstruction using the symbolic signature distance enhances the elevation optimization capability of the road reconstruction model, allowing it to more accurately capture differences in features such as bumps, depressions, and potholes on the road. This provides more comprehensive road location information, helps to more precisely describe the geometric features of the road landscape, and enables the reconstructed road model to better restore road details, resulting in a more refined mapping effect.

[0183] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or can be modified to be located in one or more apparatuses different from this embodiment. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.

[0184] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any variations conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A method of three-dimensional reconstruction of a road, characterized in that, The method includes: Acquire sampled images of the target road and pose information of the capturing device that captured the sampled images; Based on the pose information, the three-dimensional coordinates of the sampling point in the preset coordinate system are determined. The sampling point is located on the line of sight from the shooting device to the target pixel on the preset mapping plane. The preset mapping plane is used to represent the road surface of the target road. The symbol signature distance of the sampling point relative to the preset mapping plane is calculated based on the X and Y axis coordinates in the three-dimensional coordinate system. Based on the symbol signature distance and the color information of the sampling points in the sampled image, road rendering is performed on the preset mapping plane to form a road model of the target road.

2. The method of three-dimensional reconstruction of a road according to claim 1, characterized in that, The step of determining the three-dimensional coordinates of the sampling point in the preset coordinate system based on the pose information includes: Construct the preset mapping plane in the preset coordinate system; Based on the pose information, the shooting device is mapped onto the preset coordinate system, and the shooting device and the target pixel are connected to form the line of sight; Multiple sampling points are uniformly selected along the line of sight based on a preset number, and the three-dimensional coordinates of the sampling points in the preset coordinate system are determined.

3. The method of three-dimensional reconstruction of a road according to claim 1, characterized in that, The calculation of the symbol signature distance of the sampling point relative to the preset mapping plane based on the X and Y axis coordinates of the sampling point includes: The X and Y coordinates of the sampling points are input into a first multilayer perceptron to determine the predicted height value of the sampling points. The first multilayer perceptron is trained based on coordinate points that include height changes. At least some of the feature coordinate points are obtained based on the viewing angle direction of the line of sight and the device parameters of the shooting device. The difference between the Z-axis coordinate in the three-dimensional coordinate system and the predicted height value is calculated as the symbol signature distance.

4. The method of three-dimensional reconstruction of a road according to claim 3, characterized in that, The method further includes: The X and Y coordinates of the sampling points are input into the second multilayer perceptron to determine the prediction correction value of the sampling points. The second multilayer perceptron is trained based on the prior data of the sample road and the feature coordinate points of the sample road. The symbol signature distance is adjusted based on the predicted correction value.

5. The method of three-dimensional reconstruction of a road according to claim 1, characterized in that, The process of rendering a road model of the target road on a preset mapping plane based on the symbol signature distance and the color information of the sampling points in the sampled image includes: The color information of the sampling points is obtained based on the sampled image; The color weight of the sampling point is determined based on the symbol signature distance; The color information of the target pixel is determined by weighting the color information of at least one of the sampling points on the line of sight based on the color weight. The target pixel is rendered based on its color information to generate the road model.

6. The method of three-dimensional reconstruction of a road according to claim 5, characterized in that, The step of rendering the target pixel based on its color information to generate the road model includes: The grid cells with the target pixel as the vertex are determined based on the hash index, and the grid cells are obtained by dividing the three-dimensional space under the preset coordinate system; Linear interpolation is performed on the color information of the target pixel to determine the color information of other pixels in the grid cell besides the target pixel. The color information of the target pixel and the other pixels is combined to form the color information of the grid unit; The road model is obtained by rendering the grid cells based on their color information.

7. The method of three-dimensional reconstruction of a road according to any one of claims 1 to 6, characterized in that, The acquisition of the sampled image of the target road includes: Acquire sampled video of the target road; The sampled video is segmented based on a preset sampling interval to obtain keyframe images from the sampled video. The keyframe image, the vector encoding of the shooting device, and the pose information are input into the image processing model. The image processing model is then used to perform color restoration processing on the keyframe image to obtain the sampled image. The image processing model is trained using the original image and the color-shifted image. The color-shifted image is obtained by adjusting the color of the original image based on the vector encoding and the pose information when the original image was captured. The vector encoding is used to represent the color changes caused by different shooting devices.

8. A device for three-dimensional reconstruction of a road, characterized in that The device includes: The acquisition module is used to acquire sampled images of the target road and pose information of the capturing device that captured the sampled images; The positioning module is used to determine the three-dimensional coordinates of the sampling point in a preset coordinate system based on the pose information. The sampling point is located on the line of sight from the shooting device to the target pixel on the preset mapping plane. The preset mapping plane is used to represent the road surface of the target road. The first processing module is used to calculate the symbol signature distance of the sampling point relative to the preset mapping plane based on the X and Y axis coordinates in the three-dimensional coordinate system. The reconstruction module is used to perform road rendering on the preset mapping plane based on the symbol signature distance and the color information of the sampling points in the sampled image, to form a road model of the target road.

9. A computer-readable storage medium having stored thereon a program or instructions, characterized in that, When the program or instructions are executed by the processor, they implement the steps of the three-dimensional reconstruction method for roads as described in any one of claims 1 to 7.

10. A computer device comprising at least one processor coupled with a memory, the memory storing a computer program to run on the processor, characterized in that, When the processor executes the program, it implements the three-dimensional reconstruction method of the road as described in any one of claims 1 to 7.

Images