Method, device and equipment for training monocular image three-dimensional scene reconstruction model

Abstract

The present disclosure provides a monocular image three-dimensional scene reconstruction model training method, device and equipment, which uses a pre-trained monocular depth estimation model to manufacture a relative disparity map as a depth prior, introduces the installation height of a vehicle-mounted monocular camera as a physical prior to anchor the relative disparity map into an absolute scale pseudo-depth map, uses the absolute scale pseudo-depth map as a label to determine a cross-modal depth distillation loss, and trains a monocular image three-dimensional scene reconstruction model through the cross-modal depth distillation loss, a temporal self-supervised reprojection loss and an Eikonal loss. The present disclosure uses a cross-modal pseudo-supervised distillation mechanism based on a generative model to train a monocular image three-dimensional scene reconstruction model, without the need for sensors such as a laser radar and a multi-view image array to provide a supervision signal, or the need for manual labeling. The monocular image three-dimensional reconstruction model obtained through training can reconstruct a 3D scene using only a single image, and can be deployed in real time on a vehicle-mounted embedded device.

Description

Technical Field

[0001] This disclosure relates to the field of autonomous driving technology, and in particular to a training method, apparatus and device for a monocular image 3D scene reconstruction model. Background Technology

[0002] Monocular image 3D scene reconstruction technology is one of the key means to achieve low-cost and high-efficiency environmental spatial perception in the field of autonomous driving. It can recover the 3D geometric structure of a scene from a single 2D RGB image.

[0003] Currently, deep network models used for monocular image 3D scene reconstruction generally rely on high-precision 3D geometric ground truth as supervision signals for training. Related technologies use 3D point clouds or voxel meshes obtained from LiDAR scanning, or multi-view images of the same scene acquired by multi-view camera arrays, as supervision signals to train end-to-end networks such as Posed Signed Distance Function (PosedSDF) and Occupancy Networks, thereby achieving monocular image 3D scene reconstruction. However, this requires manual annotation of point clouds and multi-view images, including occlusion removal, noise reduction, and semantic completion. The construction of large-scale, high-quality 3D geometric ground truth datasets is extremely costly, resulting in huge equipment costs and annotation overhead for monocular image 3D scene reconstruction technology, making it difficult to apply to hardware-restricted autonomous driving scenarios.

[0004] Therefore, there is an urgent need for a training scheme for monocular image 3D scene reconstruction models that does not require expensive sensors such as LiDAR and multi-view camera arrays for real-time annotation. Summary of the Invention

[0005] In view of this, this disclosure provides a training method, apparatus and device for a monocular image 3D scene reconstruction model.

[0006] According to a first aspect of this disclosure, a method for training a monocular image 3D scene reconstruction model is provided, comprising: the monocular image 3D scene reconstruction model being applied to a vehicle, the vehicle being equipped with an onboard monocular camera, and the training method for the monocular image 3D scene reconstruction model comprising: Depth estimation of the current frame training image is performed by a pre-trained monocular depth estimation model to obtain the relative disparity map of the current frame training image. Based on the intrinsic parameters and installation height of the vehicle-mounted monocular camera and the relative time difference map of the current frame training image, the absolute scale pseudo depth map of the current frame training image is obtained. The SDF field of the current frame training image is obtained using a monocular image 3D scene reconstruction model; A rendering depth map of the original pose is rendered from the SDF field of the current frame training image using volume rendering. The cross-modal depth distillation loss is determined based on the rendering depth map of the original pose and the absolute scale pseudo-depth map of the current frame training image. The relative pose of the vehicle-mounted monocular camera between frames is estimated based on the IMU information corresponding to the current frame training image. The next frame rendering image of the current frame training image is rendered from the SDF field of the current frame training image through volume rendering. The temporal self-supervised reprojection loss is determined based on the relative pose of the vehicle-mounted monocular camera between frames, the next frame rendering image of the current frame training image, and the next frame real image of the current frame training image. The Eikonal loss is determined based on the SDF field of the current frame training image; The total loss of the current frame training image is determined based on the cross-modal depth distillation loss, temporal self-supervised reprojection loss, and Eikonal loss. The parameters of the monocular image 3D scene reconstruction model are updated based on the total loss of the training images in the current frame.

[0007] In some embodiments of the first aspect of this disclosure, the method further includes: generating new view pseudo-labels for the current frame training image in N predetermined virtual poses using a pre-trained new view synthesis model, where N is an integer greater than 1; rendering N new view images from the SDF field of the current frame training image in the N predetermined virtual poses using volume rendering; determining a multi-view appearance distillation loss based on the N new view images and the new view pseudo-labels for the current frame training image in the N predetermined virtual poses; and determining the total loss of the current frame training image based on the cross-modal depth distillation loss, the temporal self-supervised reprojection loss, and the Eikonal loss, including: determining the total loss of the current frame training image based on the cross-modal depth distillation loss, the temporal self-supervised reprojection loss, the Eikonal loss, and the multi-view appearance distillation loss.

[0008] In some embodiments of the first aspect of this disclosure, the method further includes: performing semantic segmentation on the current frame training image using a pre-trained semantic segmentation model to obtain a dynamic object mask of the current frame training image; determining a dynamic object depth supervision loss based on the dynamic object mask, the rendered depth map under the original pose, and the absolute scale pseudo-depth map of the current frame training image; and determining the total loss of the current frame training image based on the cross-modal depth distillation loss, the temporal self-supervised reprojection loss, and the Eikonal loss, including: determining the total loss of the current frame training image based on the cross-modal depth distillation loss, the temporal self-supervised reprojection loss, the Eikonal loss, and the dynamic object depth supervision loss.

[0009] In some embodiments of the first aspect of this disclosure, a static background mask of the current frame training image is obtained simultaneously with the dynamic object mask; the cross-modal depth distillation loss is determined based on the static background mask, the rendered depth map under the original pose, and the absolute scale pseudo-depth map of the current frame training image; the temporal self-supervised reprojection loss is determined based on the static background mask, the current inter-frame relative pose of the vehicle-mounted monocular camera, the next frame rendered image of the current frame training image, and the next frame real image of the current frame training image.

[0010] In some embodiments of the first aspect of this disclosure, the method further includes: determining the ground plane geometric consistency loss based on the rendered depth map under the original pose, the intrinsic parameters of the vehicle-mounted monocular camera, and the installation height; the step of determining the total loss of the current frame training image based on the cross-modal depth distillation loss, the temporal self-supervised reprojection loss, and the Eikonal loss includes: determining the total loss of the current frame training image based on the cross-modal depth distillation loss, the temporal self-supervised reprojection loss, the Eikonal loss, and the ground plane geometric consistency loss.

[0011] In some embodiments of the first aspect of this disclosure, the step of rendering a rendering depth map in the original pose from the SDF field of the current frame training image through volume rendering includes: performing the following processing on a pixel-by-pixel basis on the current frame training image to obtain a rendering depth map in the original pose: generating a ray corresponding to the current pixel based on the pose and intrinsic parameters of the vehicle-mounted monocular camera in the current frame training image; collecting multiple discrete points along the ray corresponding to the current pixel between a predetermined near and far plane to form a sampling point sequence; querying the SDF field of the current frame training image for each point in the sampling point sequence to obtain its SDF value; converting the SDF value of each point in the sampling point sequence into the opacity of multiple sampling intervals, and accumulating the rendering depth value of the current pixel point by point based on the opacity of the multiple sampling intervals.

[0012] In some embodiments of the first aspect of this disclosure, obtaining the absolute-scale pseudo-depth map of the current frame training image based on the intrinsic parameters and installation height of the vehicle-mounted monocular camera and the relative time difference map of the current frame training image includes: The ground pixel set of the current frame training image is extracted using a pre-trained semantic segmentation model; Based on the intrinsic parameters and installation height of the vehicle-mounted monocular camera, the ground pixel set of the current frame training image and its relative time difference map, the scale parameters of the affine model of relative disparity and true depth are obtained by solving the ground plane equation by least squares closed-form. Based on the scale parameters and the relative disparity map of the current frame training image, an absolute scale pseudo-depth map of the current frame training image is generated through an affine model of the relative disparity and the true depth.

[0013] According to a second aspect of this disclosure, an apparatus is provided for training a monocular image 3D scene reconstruction model, the monocular image 3D scene reconstruction model being applied to a vehicle, the vehicle being equipped with an onboard monocular camera; the apparatus for training the monocular image 3D scene reconstruction model comprises: The pseudo-depth map generation unit is used to perform depth estimation on the current frame training image through a pre-trained monocular depth estimation model to obtain the relative disparity map of the current frame training image, and to obtain the absolute scale pseudo-depth map of the current frame training image based on the intrinsic parameters and installation height of the vehicle-mounted monocular camera and the relative time difference map of the current frame training image. The forward inference unit is used to obtain the SDF field of the current frame training image from the monocular image 3D scene reconstruction model; A cross-modal depth distillation loss determination unit is used to render a rendered depth map in the original pose from the SDF field of the current frame training image through volume rendering, and to determine the cross-modal depth distillation loss based on the rendered depth map in the original pose and the absolute scale pseudo depth map of the current frame training image. The temporal self-supervised reprojection loss determination unit is used to estimate the current inter-frame relative pose of the vehicle-mounted monocular camera based on the IMU information corresponding to the current frame training image, render the next frame rendering image of the current frame training image from the SDF field of the current frame training image through volume rendering, and determine the temporal self-supervised reprojection loss based on the current inter-frame relative pose of the vehicle-mounted monocular camera, the next frame rendering image of the current frame training image, and the next frame real image of the current frame training image. The Eikonal loss determination unit is used to determine the Eikonal loss based on the SDF field of the training image in the current frame. The total loss determination unit is used to determine the total loss of the current frame training image based on the cross-modal depth distillation loss, temporal self-supervised reprojection loss, and Eikonal loss. The parameter update unit is used to update the parameters of the monocular image 3D scene reconstruction model based on the total loss of the training image in the current frame.

[0014] According to a third aspect of this disclosure, an electronic device is provided, comprising: a processor and a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to perform the methods described above.

[0015] According to a fourth aspect of this disclosure, a computer-readable storage medium storing a program, the program including instructions that, when executed by a processor, cause the processor to perform the methods described above.

[0016] This embodiment employs a cross-modal pseudo-supervised distillation mechanism based on a generative model to train a monocular image 3D scene reconstruction model. It does not require supervision signals from sensors such as LiDAR or multi-view image arrays, nor does it require manual annotation. The trained monocular image 3D reconstruction model can reconstruct a 3D scene using only a single image, resulting in low training costs and real-time deployment on vehicle-mounted embedded devices. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the system architecture for an application scenario of a monocular image 3D scene reconstruction model trained through embodiments of this disclosure; Figure 2 A flowchart illustrating the training method for a monocular image 3D scene reconstruction model provided in this embodiment of the disclosure; Figure 3 A schematic diagram of the structure of the apparatus for training a monocular image 3D scene reconstruction model provided in an embodiment of this disclosure; Figure 4 A schematic block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0019] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0020] The terminology used in the embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure. The singular forms “a,” “the,” and “the” as used in the embodiments of this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0022] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0023] The following is a definition of the terms used in this article: Original pose: This refers to the position and orientation of the vehicle-mounted monocular camera at the instant the current frame of the training image is captured. The original pose is not obtained through calculation or sensor measurement, but is manually set as the origin of the entire scene coordinate system, that is, a manually set reference coordinate system.

[0024] Figure 1 A schematic diagram of the architecture of a system applicable to a monocular image 3D scene reconstruction model trained through embodiments of this disclosure is shown. See also Figure 1 The system to which the monocular image 3D scene reconstruction model trained by the embodiments of this disclosure is applicable may include: a domain controller, an onboard monocular camera, and an inertial measurement unit (IMU). The onboard monocular camera and the IMU are respectively connected to the domain controller, which can be used to perform 3D scene reconstruction based on monocular images by the monocular image 3D scene reconstruction model obtained by the training method described below.

[0025] In specific applications, the domain controller can also be implemented as, but is not limited to, other similar electronic devices. The specifications of the vehicle-mounted monocular camera are not limited and can be flexibly configured as needed. For example, the hardware specifications of the vehicle-mounted monocular camera may be, but are not limited to: a resolution of 1920×1080 and a frame rate of 30fps. The connection methods between the vehicle-mounted monocular camera and the electronic device, as well as the connection methods between the IMU and the electronic device, can be various forms such as wired and wireless; this disclosure does not impose any limitations on these methods.

[0026] The monocular image 3D scene reconstruction model trained using the embodiments of this disclosure is applicable to systems including, but not limited to, autonomous driving systems and intelligent driver assistance systems. See also Figure 1 The monocular image 3D scene reconstruction model obtained through the embodiments of this disclosure can be loaded into a vehicle and applied, but not limited to, intelligent driver assistance systems, autonomous driving systems, etc.

[0027] The monocular image 3D scene reconstruction model trained through the embodiments of this disclosure can be applied to, but is not limited to, intelligent control of various vehicles such as multiple wheeled mobile robots, mobile robots, vehicles, aircraft, ships, autonomous rail rapid transit systems (ART), and industrial automation equipment. Vehicles can be, but are not limited to, passenger cars, commercial vehicles (e.g., trucks, buses, vans), special-purpose vehicles (e.g., ambulances, fire trucks, engineering vehicles, rescue vehicles), agricultural and industrial vehicles (e.g., harvesters, forklifts), transportation and logistics vehicles (e.g., container trucks, refrigerated trucks), new energy vehicles (e.g., electric vehicles, hybrid vehicles), and special vehicles (e.g., garbage trucks, water trucks). In other words, "vehicle" in the embodiments of this disclosure is equivalent to the aforementioned various devices.

[0028] The monocular image 3D scene reconstruction model trained through the embodiments of this disclosure can be applied to various scenarios such as urban transportation, highways, ports, mines, farms, closed parks, and industrial production. It is applicable to many aspects such as passenger travel, public transportation, logistics distribution, unmanned transportation, last-mile delivery, automated agricultural operations, and automated sanitation. This disclosure does not limit the application scenarios and applicable fields of the embodiments of this disclosure.

[0029] Those skilled in the art will understand that the monocular image 3D scene reconstruction model trained through the embodiments of this disclosure is applicable to systems not limited to... Figure 1 The architecture shown is not limited to the above-mentioned application scenarios.

[0030] Figure 2 The diagram illustrates a flowchart of a training method for a monocular image 3D scene reconstruction model provided in this embodiment. The monocular image 3D scene reconstruction model trained by this method is applied to a vehicle equipped with an onboard monocular camera. See also... Figure 2 The training method for the monocular image 3D scene reconstruction model in this embodiment of the present disclosure may include: Step 201: Perform depth estimation on the current frame training image using a pre-trained monocular depth estimation model to obtain the relative disparity map of the current frame training image. Based on the intrinsic parameters and installation height of the vehicle-mounted monocular camera and the relative time difference map of the current frame training image, obtain the absolute scale pseudo-depth map of the current frame training image. Step 202: Obtain the SDF field of the current frame training image using the monocular image 3D scene reconstruction model; Step 203: Render the rendering depth map of the original pose from the SDF field of the current frame training image through volume rendering, and determine the cross-modal depth distillation loss based on the rendering depth map of the original pose and the absolute scale pseudo depth map of the current frame training image. Step 204: Estimate the current inter-frame relative pose of the vehicle-mounted monocular camera based on the IMU information corresponding to the current frame training image, render the next frame rendering image of the current frame training image from the SDF field of the current frame training image through volume rendering, and determine the temporal self-supervised reprojection loss based on the current inter-frame relative pose of the vehicle-mounted monocular camera, the next frame rendering image of the current frame training image, and the next frame real image of the current frame training image. Step 205: Determine the Eikonal loss based on the SDF field of the training image in the current frame; Step 206: Determine the total loss of the current frame training image based on the cross-modal depth distillation loss, temporal self-supervised reprojection loss, and Eikonal loss. Step 207: Update the parameters of the monocular image 3D scene reconstruction model based on the total loss of the training images in the current frame.

[0031] This embodiment uses a pre-trained monocular depth estimation model to generate a relative disparity map as a depth prior, solving the problem of no ground truth. It introduces the "installation height of the vehicle-mounted monocular camera" as a physical prior, and through the transformation of "ground plane constraints," anchors the relative disparity map to an absolute-scale pseudo-depth map, solving the absolute-scale problem. The absolute-scale pseudo-depth map is used as a label to determine the cross-modal depth distillation loss. The monocular image 3D scene reconstruction model is trained using the cross-modal depth distillation loss, temporal self-supervised reprojection loss, and Eikonal loss. Thus, the training of the monocular image 3D scene reconstruction model is achieved through a cross-modal pseudo-supervised distillation mechanism based on a generative model. This eliminates the need for supervision signals from sensors such as LiDAR or multi-view image arrays, and also eliminates the need for manual annotation. The trained monocular image 3D reconstruction model can reconstruct a 3D scene using only a single image and can be deployed in real-time on vehicle-mounted embedded devices.

[0032] In practical applications, a pre-trained model library can be pre-built. This pre-trained model library may include, but is not limited to, pre-trained monocular depth estimation models, novel perspective synthesis models (described below), and semantic segmentation models (described below). In the methods described in the embodiments of this disclosure, the required models can be directly called from this pre-trained model library. For example, in step 201, the monocular depth estimation model can be directly called from the pre-trained model library.

[0033] Monocular depth estimation models can utilize relative depth estimation networks such as Depth Anything and DPT, which are general-purpose models trained on a variety of depth datasets. The input data for a monocular depth estimation model is any natural image, and the output data is a pixel-wise relative disparity or pseudo-depth map.

[0034] The novel perspective synthesis model can be based on a diffusion model architecture (such as Zero-1-to-3, MVDream, etc.) and can be pre-trained on pre-selected large-scale multi-view or multimodal data. The novel perspective synthesis model can be used to obtain corresponding novel view pseudo-labels from a source RGB image and embedding vectors representing N virtual poses. These novel view pseudo-labels include pre-selected N novel perspective view pseudo-labels, where N is an integer greater than 1. Preferably, N=8.

[0035] Semantic segmentation models can be selected from pre-trained models with semantic segmentation capabilities, such as SegFormer-B2, Mask2Former, PIDNet, DeepLabV3+, or others. Semantic segmentation models can be used to extract ground pixel sets and generate dynamic object masks and static background masks.

[0036] The parameters of the pre-trained models, such as the novel perspective synthesis model, monocular depth estimation model, and semantic segmentation model involved in the embodiments of this disclosure, are all frozen and do not participate in the parameter updates involved in the embodiments of this disclosure.

[0037] In this embodiment of the disclosure, the installation height of the vehicle-mounted monocular camera refers to the installation height of the vehicle-mounted monocular camera relative to the ground. The installation height of the vehicle-mounted monocular camera can be measured in advance, and the intrinsic parameters of the vehicle-mounted monocular camera can be calibrated in advance.

[0038] In step 201, an exemplary implementation of obtaining the absolute scale pseudo-depth map of the current frame training image may include the following steps a1 to a3: Step a1: Extract the set of ground pixels of the current frame training image using a pre-trained semantic segmentation model; Step a2: Based on the intrinsic parameters and installation height of the vehicle-mounted monocular camera, the ground pixel set of the current frame training image and its relative time difference map, the scale parameters of the affine model of relative disparity and true depth are obtained by solving the ground plane equation by least squares closed form. Specifically, the scale parameters of the affine model of relative parallax and true depth can include a scale factor and translation corrections. In some examples, the scale factor and translation corrections of the affine model of relative parallax and true depth can be obtained by solving the following ground plane equations using a least-squares closed-form method.

[0039]

[0040] in, This represents the set of ground pixels in the current frame of the training image. This represents the homogeneous image coordinates of ground pixels in the ground pixel set of the current frame training image. Represents pixels in a relative disparity map The relative time difference at that location This represents the inverse of the intrinsic parameter matrix of the vehicle-mounted monocular camera. Represents the ground normal vector. This indicates the installation height of the vehicle-mounted monocular camera above the ground. This represents the scale factor to be solved. This represents the translation correction amount to be solved.

[0041] Step a3: Based on the scale parameter and the relative disparity map of the current frame training image, an absolute scale pseudo-depth map of the current frame training image is generated through an affine model of relative disparity and true depth.

[0042] Specifically, generating an absolute-scale pseudo-depth map using an affine model of relative parallax and true depth can be expressed as follows:

[0043] in, This represents the absolute scale pseudo-depth map of the current frame training image. This represents the relative disparity map of the current frame training image.

[0044] Therefore, by utilizing the installation height of the vehicle-mounted monocular camera and based on the road plane assumption, relative parallax can be converted into absolute metric depth. This allows the generation of an absolute-scale pseudo-depth map for each new view pseudo-label using the relative time difference map of each new view pseudo-label. By introducing the "installation height of the vehicle-mounted monocular camera" as a physical prior and transforming it using "ground plane constraints," the relative parallax map is anchored to an absolute-scale pseudo-depth map, thus resolving the absolute scale problem.

[0045] The input data of the monocular image 3D scene reconstruction model is a single-frame image, and the output data is the symbolic distance function (SDF) field of the scene, which can query the SDF value of any 3D point. The architecture of the monocular image 3D scene reconstruction model is not limited. In some examples, the monocular image 3D scene reconstruction model may include an image encoder, a scene encoder, and an SDF decoder connected in sequence. The image encoder can use a convolutional neural network (such as ResNet-50) or a Transformer backbone to extract the 2D image features of the single-frame image. The scene encoder can use a three-plane representation or a sparse voxel mesh structure to upscale the 2D image features to 3D space, allowing any 3D point to obtain its corresponding feature vector through projection and interpolation. The SDF decoder can use a lightweight multilayer perceptron to output the symbolic distance function value of each point based on the input 3D point coordinates and their feature vectors, thereby implicitly defining the SDF field of the entire scene. In step 202, the current frame training image is processed sequentially by the image encoder, scene encoder, and SDF decoder in the monocular image 3D scene reconstruction model to obtain the SDF field of the current frame training image.

[0046] In some implementations, in step 203, the following steps b1 to b4 are performed pixel-by-pixel on the current frame training image to obtain the rendered depth map under the original pose: Step b1: Generate the light corresponding to the current pixel based on the original pose and extrinsic parameters of the vehicle-mounted monocular camera; Specifically, for the current pixel p(u,v) in the current frame of the training image, the parametric equation of its ray can be expressed as follows:

[0047]

[0048] in, Represents a point on a light ray. The translation vector represents the original pose, i.e., the position of the vehicle-mounted monocular camera in the world coordinate system. This represents the depth value, which is the distance from the optical center along the direction of the light ray, and is expressed in meters. This represents the unit vector indicating the direction of the ray at the current pixel. This indicates the internal parameters of the vehicle-mounted monocular camera. This represents the rotation matrix in the original pose of the vehicle-mounted monocular camera. This represents the homogeneous pixel coordinates of the current pixel p.

[0049] The parametric equations of the aforementioned ray describe a ray that originates from the optical center of the vehicle-mounted monocular camera, passes through the current pixel's corresponding position, and extends into three-dimensional space. This ray forms the geometric basis of volumetric rendering, and all subsequent sampling and integration are performed on this ray.

[0050] Step b2: Collect multiple discrete points along the ray corresponding to the current pixel between the predefined near and far planes to form a sampling point sequence; For example, on the light path of the current pixel, a predetermined near-far plane is defined. Divide the sample into Q uniform intervals, and randomly select one point from each interval to obtain Q discrete sampling points. Arrange the Q discrete sampling points in ascending order of depth to form a sampling point sequence. Here, additional points can be sampled in the region near the surface within the Q intervals. This disclosure does not limit this aspect. This represents the predetermined depth value of the near-plane. This represents the depth value of a pre-selected far plane.

[0051] Step b3: For each point in the sampling point sequence, query the SDF field of the current frame training image to obtain its SDF value; In some examples, the SDF value of each point in the sampling point sequence can be calculated using the following formula:

[0052] in, This represents the SDF value of the i-th sampling point. This represents the monocular image unit scene reconstruction model to be trained. Represents the three-dimensional coordinates of the i-th sampling point. The depth value of the i-th sampling point represents the actual physical distance of the sampling point from the optical center. This represents the unit vector indicating the direction of the ray at the current pixel.

[0053] Step b4: Convert the SDF value of each point in the sampling point sequence into the opacity of multiple sampling intervals, and accumulate the rendering depth value of the current pixel point by point according to the opacity of multiple sampling intervals.

[0054] In some examples, the opacity can be obtained by the following conversion:

[0055]

[0056] in, This represents the opacity of the i-th sampling interval. , This represents two adjacent sampling points corresponding to the i-th sampling interval. This represents the Sigmoid function with a scale parameter, used to convert SDF values... Mapping to the (0,1) interval provides a smooth transition; express The scale parameter, which is a fixed hyperparameter, is used to control the sharpness of the surface transition. The larger the value, the clearer the surface. This indicates a truncation operation.

[0057] When light comes from the outside ( ) penetrate the surface and enter the interior ( )hour, Approaching 1 When the value is close to 0, the molecule is positive, resulting in opacity. Conversely, if the light is always outside or inside, and The values ​​are close to zero, and the opacity approaches 0. This ensures that the weight distribution of volumetric rendering is concentrated precisely near the zero isosurface defined by SDF, which is beneficial for achieving unbiased surface reconstruction.

[0058] Specifically, the rendering depth value of the current pixel is calculated by accumulating light rays sequentially from the near end to the far end. In some examples, the cumulative transmittance can be calculated first based on the opacity of the sampling interval, and then the rendering depth value of the current pixel can be calculated using the cumulative transmittance and the opacity.

[0059] In some examples, the cumulative transmittance can be calculated using the following formula:

[0060] in, Let represent the cumulative transmittance when reaching the i-th sampling interval, and let represent the proportion of light that was not blocked before reaching that position. ; This represents the opacity of the j-th sampling interval. The symbol represents a product. This formula can be used to simulate the physical process of light propagating through a medium, where the intensity of light decreases as it passes through a certain interval. times. Indicates the depth reached by light. How much energy remains unabsorbed? As the number of zones traversed increases, It gradually decreases until it approaches 0.

[0061] In some examples, the rendering depth value of the current pixel can be calculated using the following formula:

[0062] Where Q represents the total number of sampling points, which is a preset hyperparameter; This represents the rendering depth value of the current pixel. This represents the cumulative transmittance of the i-th sampling interval. This represents the opacity of the i-th sampling interval. This represents the depth of the i-th sampling point.

[0063] That is, the depth values ​​of all sampling intervals are weighted according to their visibility. Perform a weighted average. Visibility weight. This represents the probability that the light ray is terminated by the scene surface exactly in the i-th sampling interval. Therefore, This represents the expected flight distance of a photon upon its first impact on the surface, i.e., the depth value of the current pixel. In an SDF field, the visibility weight... It will be highly concentrated near the zero isosurface, thus accurately capturing the surface position.

[0064] In step 203, the cross-modal depth distillation loss can be calculated using the following formula based on the rendered depth map of the original pose and the pseudo-depth map of the absolute scale of the original pose. ;

[0065] in, This represents the loss value of cross-modal depth distillation. This represents the depth value at pixel p in the rendered depth map under the original pose. Let represent the absolute scale pseudo-depth value at pixel p in the absolute scale pseudo-depth map of the current frame training image, and M represent the total number of pixels in the image. Here, the original pose refers to the pose when the vehicle-mounted monocular camera captured the current frame training image. Therefore, using inverse depth L1 loss can naturally balance the penalty weights for near and far pixels, ensuring that the training signal in distant regions is not overwhelmed.

[0066] By using cross-modal depth distillation loss, the absolute scale geometric prior of the teacher model can be injected into the monocular image 3D scene reconstruction model, making its predicted SDF surface depth values ​​consistent with the physical world.

[0067] In some embodiments, the method of this disclosure may further include: generating new view pseudo-labels for the current frame training image in N predetermined virtual poses using a pre-trained new view synthesis model, where N is an integer greater than 1; rendering N new view images from the SDF field of the current frame training image in N predetermined virtual poses using volume rendering; and determining a multi-view appearance distillation loss based on the N new view images and the new view pseudo-labels for the current frame training image in the N predetermined virtual poses. In this embodiment, in step 206, the total loss of the current frame training image can be determined based on the cross-modal depth distillation loss, the temporal self-supervised reprojection loss, the Eikonal loss, and the multi-view appearance distillation loss.

[0068] Specifically, the process of generating new view pseudo-labels for the current frame training image under N predetermined virtual poses using a pre-trained new view synthesis model can include: pre-configuring N virtual poses, encoding the N virtual poses to obtain corresponding N pose codes, and inputting the current frame training image and the N pose codes into the new view synthesis model to generate N new view pseudo-labels, which correspond one-to-one with the N virtual poses.

[0069] N virtual poses represent N possible positions and orientations that the vehicle-mounted monocular camera may reach along the vehicle's trajectory within a very short time. In some examples, each of the N virtual poses may include a translation vector and a rotation matrix. The translation vector represents a possible position that the vehicle-mounted monocular camera may reach along the vehicle's trajectory within a very short time, and the rotation matrix represents the attitude change of the vehicle-mounted monocular camera within a very short time. The translation vector includes forward displacement, lateral displacement, and vertical displacement. The forward displacement represents the actual physical distance the vehicle-mounted monocular camera moves forward along the current vehicle travel direction (Z-axis). The lateral displacement represents a small translation of the vehicle-mounted monocular camera in the horizontal direction perpendicular to the travel direction, and the vertical displacement represents a small up-and-down movement of the vehicle-mounted monocular camera in the vertical direction. The rotation matrix in each virtual pose is approximately an identity matrix.

[0070] In some examples, multi-view appearance distillation loss It can be calculated using the following formula:

[0071] in, The value represents the multi-view appearance distillation loss, N represents the total number of virtual poses, and n represents the virtual pose identifier, n=1,2,…,N; This represents the new viewpoint image corresponding to the virtual pose n. This represents the pseudo-label of the new view corresponding to the virtual pose n.

[0072] Therefore, a pre-trained novel perspective synthesis model can be used to create multi-view pseudo-labels. The multi-view appearance distillation loss can be calculated using these pseudo-labels and the novel perspective images obtained from volume rendering. The multi-view appearance distillation loss provides rich 3D appearance consistency constraints, which helps the monocular image 3D scene reconstruction model learn complete scene geometry information, especially the scene geometry information of occluded areas and / or distant areas.

[0073] In step 204, the next frame rendering image of the current frame training image is the color map of the next frame view of the current frame training image. Its rendering method is basically the same as the rendering method of the aforementioned depth map. The difference is that the value of each pixel in the next frame rendering image is the pixel's rendering color value.

[0074] In some examples, the rendered color value for each pixel can be obtained using the following formula:

[0075]

[0076] in, This represents the rendered color value of the current pixel. This represents the RGB color value of the i-th sampling point. Similar to the rendering depth mentioned earlier, color accumulation involves weighted summation of the colors along the ray according to their visibility weights. Physically, this is equivalent to calculating the radiometric intensity reflected from the scene surface and reaching the vehicle's monocular camera. Represents the three-dimensional coordinates of the i-th sampling point. Indicates sampling point The scene feature vector at a given location is obtained by interpolation using the scene encoder in the monocular image 3D scene reconstruction model. This represents the unit vector indicating the direction of the ray at the current pixel. This is a branch of the monocular image 3D scene reconstruction model, used to learn complex mapping functions from geometry / features to RGB colors.

[0077] In some implementations, in step 204, the L1 photometric error can be calculated pixel by pixel between the next frame rendered image and the next frame real image, and the average can be taken only on the effective pixel set to obtain the temporal self-supervised reprojection loss.

[0078] Specifically, temporal self-supervised reprojection loss It can be calculated using the following formula:

[0079] in, This represents the temporal self-supervised reprojection loss value. This represents the value at pixel p in the next frame of the rendered image, which is the training image of the current frame. This represents the pixel value at pixel p in the next frame of the real image from the current training image. This represents the set of valid pixels.

[0080] Among them, the effective pixel set The relative pose between frames of the vehicle-mounted monocular camera can be calculated through back projection, coordinate transformation, projection, and depth consistency checks. This is used to eliminate out-of-bounds and occluded pixels caused by camera motion, ensuring that the loss only applies to the corresponding observable pixels, thereby accurately injecting real physical motion information into SDF geometry optimization.

[0081] In step 204, the inter-frame relative pose of the vehicle-mounted monocular camera can be estimated using IMU-assisted information. The inter-frame relative pose of the vehicle-mounted monocular camera refers to the inter-frame relative pose between the current training image and the next real image. The inter-frame relative pose describes a relative motion from the position of the vehicle-mounted monocular camera at acquisition time t of the current training image to the position of the vehicle-mounted monocular camera at acquisition time t+1 of the next real image.

[0082] In some examples, the inter-frame relative pose of the vehicle-mounted monocular camera can be calculated by integrating the IMU-assisted information at acquisition time t of the current training image and the IMU-assisted information at acquisition time t+1 of the next real image. Here, the IMU-assisted information may include, but is not limited to, acceleration and angular velocity. Specifically, relative rotation is obtained by attitude integration of the IMU angular velocity, and relative translation is obtained by double integration of the gravity-free acceleration. The relative rotation and relative translation are combined and a camera-IMU extrinsic parameter transformation is applied to obtain the inter-frame relative pose of the vehicle-mounted monocular camera. It should be noted that this method is only an example. Those skilled in the art should understand that the specific calculation method for the inter-frame relative pose of the vehicle-mounted monocular camera is not limited in the embodiments of this disclosure.

[0083] In some implementations, in step 205, the Eikonal loss can be calculated using the following formula:

[0084] in, This represents the value of the Eikonal regularization loss term, used to measure the degree to which the output of the SDF field of the current training image violates the mathematical constraint that "gradient magnitude is 1". This represents calculating the expected value of a sampling point x in space, where x represents the coordinates of the sampling point in three-dimensional space. It can be any location in 3D space to be queried, and is used to detect whether the SDF field at that location satisfies the distance field property. This represents the SDF value of the SDF field of the current frame training image at spatial point x. This represents the gradient vector of the SDF field of the current frame training image at a spatial point x with respect to the coordinates at x.

[0085] In some embodiments, this disclosure may further include: determining the ground plane geometric consistency loss based on the rendered depth map in the original pose, the intrinsic parameters of the vehicle-mounted monocular camera, and the installation height. In this embodiment, in step 206, the total loss of the current frame training image can be determined based on the cross-modal depth distillation loss, temporal self-supervised reprojection loss, Eikonal loss, and ground plane geometric consistency loss. The ground plane geometric consistency loss can provide the most direct absolute scale anchoring and ground geometric regularization for the monocular image 3D scene reconstruction model. The ground plane geometric consistency loss can further anchor the absolute scale and constrain ground flatness.

[0086] Specifically, the rendering depth map can be obtained from the original pose. The ground pixels are forced to satisfy the ground plane equation in their 3D points, thus calculating the ground plane geometric consistency loss. That is, the ground plane geometric consistency loss can be calculated using the following formula:

[0087] in, This represents the ground plane geometric consistency loss value. Represents the rendered depth map under the original pose. The set of ground pixels, Indicates the installation height of the vehicle-mounted monocular camera. This indicates the internal parameters of the vehicle-mounted monocular camera. Represents the rendered depth map under the original pose. The value at pixel p. Represents the ground normal vector , indicating perpendicular to the ground downwards.

[0088] Specifically, a pre-trained semantic segmentation model can be used to extract the set of ground pixels from the rendered depth map under the original pose. .

[0089] In this embodiment, in step 206, the total loss of the current frame training image can be determined based on the cross-modal depth distillation loss, temporal self-supervised reprojection loss, Eikonal loss, and ground plane constraint loss.

[0090] In some embodiments, the method of this disclosure may include: performing semantic segmentation on the current frame training image using a pre-trained semantic segmentation model to obtain a dynamic object mask for the current frame training image; and determining a dynamic object depth supervision loss based on the dynamic object mask, the rendered depth map in the original pose, and the absolute scale pseudo-depth map of the current frame training image. In this embodiment, in step 206, the total loss of the current frame training image may be determined based on the cross-modal depth distillation loss, the temporal self-supervised reprojection loss, the Eikonal loss, and the dynamic object depth supervision loss.

[0091] In this implementation, a static background mask for the current frame training image can be obtained simultaneously with the dynamic object mask. In some examples, the cross-modal depth distillation loss can be determined based on the static background mask, the rendered depth map under the original pose, and the absolute-scale pseudo-depth map of the current frame training image. The temporal self-supervised reprojection loss can be determined based on the static background mask, the inter-frame relative pose of the vehicle-mounted monocular camera, the next frame rendered image of the current frame training image, and the next frame real image of the current frame training image. In other examples, a static background mask can be introduced into the aforementioned cross-modal depth distillation loss, temporal self-supervised reprojection loss, and multi-view appearance distillation loss. Specifically, the specific calculation methods for the cross-modal depth distillation loss, temporal self-supervised reprojection loss, and multi-view appearance distillation loss are basically the same as the aforementioned formulas; it is only necessary to explicitly multiply the static background mask into the corresponding loss calculation formula. Thus, dynamic region interference can be eliminated.

[0092] In some examples, the depth supervision loss for dynamic objects can be calculated using the following formula:

[0093] in, This represents the depth supervision loss value for dynamic objects. This represents the value of the dynamic object mask at pixel p. Represents the rendered depth map under the original pose. The value at pixel p, This represents the value of the absolute-scale pseudo-depth map at pixel p under the original pose. It is a very small constant set by the user to prevent the denominator from being zero when there are no moving objects in the frame.

[0094] The dynamic object depth supervision loss forces the monocular image 3D scene reconstruction model to predict the depth of dynamic objects numerically close to the pseudo-labels, but does not require the multi-frame geometric consistency of these objects. By using the dynamic object depth supervision loss, we can avoid mistaking dynamic objects, such as a car, for being at infinity or attached to the camera, thereby improving the reliability of subsequent obstacle perception.

[0095] In step 206, the total loss can be obtained by weighted summation of the various losses. The weights of the various losses can be preset based on experience.

[0096] In some examples, the total loss can also be obtained by weighting and summing the cross-modal depth distillation loss, temporal self-supervised reprojection loss, Eikonal loss, multi-view appearance distillation loss, dynamic object depth supervision loss, and ground plane geometric consistency loss.

[0097] Specifically, when introducing depth supervision loss for dynamic objects, the total loss can be calculated using the following formula:

[0098] in, Indicates the total loss. , , , , , These are the weights for the corresponding loss terms, and these weight values ​​can be preset based on experience. For example, , , , , , .

[0099] In some implementations, in step 207, the gradient of the total loss with respect to the parameters of the monocular image 3D scene reconstruction model can be calculated by backpropagation, and the parameters of the monocular image 3D scene reconstruction model can be updated based on the gradient using an optimizer such as Adam.

[0100] In practical applications, steps 201 to 206 can be repeated to traverse each training image in the pre-constructed training set until convergence, thus obtaining the trained monocular image 3D scene reconstruction model.

[0101] In this embodiment, the entire training process of the monocular image 3D scene reconstruction model requires no supervision signals from sensors such as LiDAR or multi-view camera systems, nor does it require manual annotation. It relies solely on monocular images, intrinsic parameters of the vehicle-mounted monocular camera, and installation height, significantly reducing data acquisition and annotation costs, and consequently, training costs. Simultaneously, by utilizing ground plane constraints, the relative depth priors provided by pre-trained models such as monocular depth estimation networks are tightly coupled with the physical scale (camera height), achieving stable 3D scene reconstruction at the absolute metric scale within the ground truth-less SDF reconstruction framework. The trained monocular image 3D scene reconstruction model can reconstruct 3D scenes based on monocular images and is suitable for real-time vehicle deployment.

[0102] Figure 3 A schematic diagram of an apparatus for training a monocular image 3D scene reconstruction model, provided in an embodiment of this disclosure, is shown. This monocular image 3D scene reconstruction model can be applied to vehicles equipped with an onboard monocular camera. See also... Figure 3 The apparatus 300 for training a monocular image 3D scene reconstruction model according to embodiments of this disclosure may include: The pseudo-depth map generation unit 301 is used to perform depth estimation on the current frame training image through a pre-trained monocular depth estimation model to obtain the relative disparity map of the current frame training image, and to obtain the absolute scale pseudo-depth map of the current frame training image based on the intrinsic parameters and installation height of the vehicle-mounted monocular camera and the relative time difference map of the current frame training image. The forward inference unit 302 is used to obtain the SDF field of the current frame training image using a monocular image 3D scene reconstruction model; The cross-modal depth distillation loss determination unit 303 is used to render the rendered depth map under the original pose from the SDF field of the current frame training image through volume rendering, and determine the cross-modal depth distillation loss based on the rendered depth map under the original pose and the absolute scale pseudo depth map of the current frame training image. The temporal self-supervised reprojection loss determination unit 304 is used to estimate the current inter-frame relative pose of the vehicle-mounted monocular camera based on the IMU information corresponding to the current frame training image, render the next frame rendering image of the current frame training image from the SDF field of the current frame training image through volume rendering, and determine the temporal self-supervised reprojection loss based on the current inter-frame relative pose of the vehicle-mounted monocular camera, the next frame rendering image of the current frame training image, and the next frame real image of the current frame training image. Eikonal loss determination unit 305 is used to determine Eikonal loss based on the SDF field of the training image in the current frame; The total loss determination unit 306 is used to determine the total loss of the current frame training image based on the cross-modal depth distillation loss, the temporal self-supervised reprojection loss, and the Eikonal loss. The parameter update unit 307 is used to update the parameters of the monocular image 3D scene reconstruction model based on the total loss of the training images in the current frame.

[0103] Furthermore, the apparatus 300 for training a monocular image 3D scene reconstruction model according to embodiments of this disclosure may further include: a multi-view appearance distillation loss determination unit 308, configured to generate new view pseudo-labels for the current frame training image in N predetermined virtual poses using a pre-trained new view synthesis model, where N is an integer greater than 1; render N new view images from the SDF field of the current frame training image in N predetermined virtual poses through volume rendering; and determine the multi-view appearance distillation loss based on the N new view images and the new view pseudo-labels for the current frame training image in N predetermined virtual poses; and a total loss determination unit, which may also be configured to determine the total loss of the current frame training image based on cross-modal depth distillation loss, temporal self-supervised reprojection loss, Eikonal loss, and multi-view appearance distillation loss.

[0104] Furthermore, the apparatus 300 for training a monocular image 3D scene reconstruction model according to embodiments of this disclosure may further include: a dynamic object depth supervision loss determination unit 309, configured to perform semantic segmentation on the current frame training image using a pre-trained semantic segmentation model to obtain a dynamic object mask of the current frame training image; determine the dynamic object depth supervision loss based on the dynamic object mask, the rendered depth map under the original pose, and the absolute scale pseudo-depth map of the current frame training image; and a total loss determination unit, which may also be configured to determine the total loss of the current frame training image based on cross-modal depth distillation loss, temporal self-supervised reprojection loss, Eikonal loss, and dynamic object depth supervision loss.

[0105] Furthermore, the dynamic object depth supervision loss determination unit 309 can also be used to obtain the static background mask of the current frame training image while obtaining the dynamic object mask; the cross-modal depth distillation loss determination unit 303 can also be used to determine the cross-modal depth distillation loss based on the static background mask, the rendered depth map under the original pose, and the absolute scale pseudo-depth map of the current frame training image; the temporal self-supervised reprojection loss determination unit 304 can also be used to determine the temporal self-supervised reprojection loss based on the static background mask, the current inter-frame relative pose of the vehicle-mounted monocular camera, the next frame rendered image of the current frame training image, and the next frame real image of the current frame training image.

[0106] Furthermore, the apparatus 300 for training a monocular image 3D scene reconstruction model according to embodiments of this disclosure may further include: a ground plane loss determination unit 310, used to determine the ground plane geometric consistency loss based on the rendered depth map under the original pose, the intrinsic parameters of the vehicle-mounted monocular camera, and the installation height; and a total loss determination unit, which may be used to determine the total loss of the current frame training image based on the cross-modal depth distillation loss, the temporal self-supervised reprojection loss, the Eikonal loss, and the ground plane geometric consistency loss.

[0107] Furthermore, the pseudo-depth map generation unit 301 can be specifically used to obtain the absolute scale pseudo-depth map of the current frame training image in the following manner: extracting the ground pixel set of the current frame training image through a pre-trained semantic segmentation model; based on the intrinsic parameters and installation height of the vehicle-mounted monocular camera, the ground pixel set of the current frame training image and its relative time difference map, obtaining the scale parameters of the affine model of relative disparity and true depth by solving the ground plane equation by least squares closed form; and generating the absolute scale pseudo-depth map of the current frame training image based on the scale parameters and the relative disparity map of the current frame training image through the affine model of relative disparity and true depth.

[0108] Furthermore, the cross-modal depth distillation loss determination unit 303 can be specifically used to render the rendering depth map under the original pose in the following manner: performing the following processing on a pixel-by-pixel basis for the current frame training image to obtain the rendering depth map under the original pose: generating the ray corresponding to the current pixel based on the pose and intrinsic parameters of the vehicle-mounted monocular camera under the current frame training image; collecting multiple discrete points along the ray corresponding to the current pixel between a predetermined near and far plane to form a sampling point sequence; querying the SDF field of the current frame training image for each point in the sampling point sequence to obtain its SDF value; converting the SDF value of each point in the sampling point sequence into the opacity of multiple sampling intervals, and accumulating the rendering depth value of the current pixel point by point based on the opacity of multiple sampling intervals.

[0109] In specific applications, the apparatus 300 for training a monocular image 3D scene reconstruction model can be implemented by software, hardware, or a combination of both. For example, the apparatus 300 for training a monocular image 3D scene reconstruction model can be implemented as software running in the electronic device 400 described below.

[0110] Figure 4 A schematic structural diagram of an electronic device provided according to an embodiment of this disclosure is shown. See also... Figure 4 The electronic device 400 provided in this embodiment may include: one or more processors 401 and a memory 402. The memory 402 stores a computer program, which, when run by the one or more processors 401, causes the processors 401 to perform the aforementioned training method for a monocular image 3D scene reconstruction model.

[0111] Processor 401 may be, but is not limited to, a central processing unit (CPU), a graphics processing unit (GPU), or other processing units with data processing capabilities and / or instruction execution capabilities.

[0112] Memory 402 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the methods described above and / or other desired functions.

[0113] Depending on the specific application, the electronic device 400 may also include any other suitable components.

[0114] In addition to the methods and devices described above, embodiments of this disclosure may also be computer program products, including computer program instructions that, when executed by a processor, cause the processor to perform the steps in the above-described method for training a monocular image 3D scene reconstruction model.

[0115] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0116] Furthermore, embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon, which, when run by a processor, causes the processor to perform the steps in the above-described training method for a monocular image 3D scene reconstruction model.

[0117] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0118] The technical solutions provided in this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this disclosure. Furthermore, those skilled in the art will recognize that, based on the ideas of this disclosure, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

[0119] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications or equivalent substitutions made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A training method for a monocular image 3D scene reconstruction model, characterized in that, The monocular image 3D scene reconstruction model is applied to a vehicle equipped with an onboard monocular camera. The training method for the monocular image 3D scene reconstruction model includes: Depth estimation of the current frame training image is performed by a pre-trained monocular depth estimation model to obtain the relative disparity map of the current frame training image. Based on the intrinsic parameters and installation height of the vehicle-mounted monocular camera and the relative time difference map of the current frame training image, the absolute scale pseudo depth map of the current frame training image is obtained. The SDF field of the current frame training image is obtained using a monocular image 3D scene reconstruction model; A rendering depth map of the original pose is rendered from the SDF field of the current frame training image using volume rendering. The cross-modal depth distillation loss is determined based on the rendering depth map of the original pose and the absolute scale pseudo-depth map of the current frame training image. The relative pose of the vehicle-mounted monocular camera between frames is estimated based on the IMU information corresponding to the current frame training image. The next frame rendering image of the current frame training image is rendered from the SDF field of the current frame training image through volume rendering. The temporal self-supervised reprojection loss is determined based on the relative pose of the vehicle-mounted monocular camera between frames, the next frame rendering image of the current frame training image, and the next frame real image of the current frame training image. The Eikonal loss is determined based on the SDF field of the current frame training image; The total loss of the current frame training image is determined based on the cross-modal depth distillation loss, temporal self-supervised reprojection loss, and Eikonal loss. The parameters of the monocular image 3D scene reconstruction model are updated based on the total loss of the training images in the current frame.

2. The method according to claim 1, characterized in that, The method further includes: generating new view pseudo-labels for the current frame training image in N predetermined virtual poses using a pre-trained new view synthesis model, where N is an integer greater than 1; rendering N new view images from the SDF field of the current frame training image in the N predetermined virtual poses using volume rendering; and determining the multi-view appearance distillation loss based on the N new view images and the new view pseudo-labels for the current frame training image in the N predetermined virtual poses. The total loss of the current frame training image is determined based on the cross-modal depth distillation loss, temporal self-supervised reprojection loss, and Eikonal loss, including: determining the total loss of the current frame training image based on the cross-modal depth distillation loss, temporal self-supervised reprojection loss, Eikonal loss, and multi-view appearance distillation loss.

3. The method according to claim 1, characterized in that, The method further includes: performing semantic segmentation on the current frame training image using a pre-trained semantic segmentation model to obtain a dynamic object mask of the current frame training image; and determining a dynamic object depth supervision loss based on the dynamic object mask, the rendered depth map under the original pose, and the absolute scale pseudo-depth map of the current frame training image. The total loss of the current frame training image is determined based on the cross-modal depth distillation loss, temporal self-supervised reprojection loss, and Eikonal loss, including: determining the total loss of the current frame training image based on the cross-modal depth distillation loss, temporal self-supervised reprojection loss, Eikonal loss, and dynamic object depth supervision loss.

4. The method according to claim 3, characterized in that, While obtaining the dynamic object mask, a static background mask for the current frame training image is also obtained; the cross-modal depth distillation loss is determined based on the static background mask, the rendered depth map under the original pose, and the absolute scale pseudo-depth map of the current frame training image; the temporal self-supervised reprojection loss is determined based on the static background mask, the current inter-frame relative pose of the vehicle-mounted monocular camera, the next frame rendered image of the current frame training image, and the next frame real image of the current frame training image.

5. The method according to claim 1, characterized in that, The method further includes: determining the ground plane geometric consistency loss based on the rendered depth map under the original pose, the intrinsic parameters of the vehicle-mounted monocular camera, and the installation height; The step of determining the total loss of the current frame training image based on the cross-modal depth distillation loss, temporal self-supervised reprojection loss, and Eikonal loss includes: determining the total loss of the current frame training image based on the cross-modal depth distillation loss, temporal self-supervised reprojection loss, Eikonal loss, and ground plane geometric consistency loss.

6. The method according to claim 1, characterized in that, The step of rendering a rendering depth map in the original pose from the SDF field of the current frame training image through volume rendering includes: Perform the following processing pixel-by-pixel on the current frame training image to obtain the rendered depth map in the original pose: Based on the pose and intrinsic parameters of the vehicle-mounted monocular camera in the current frame training image, generate the light corresponding to the current pixel; Multiple discrete points are collected along the ray corresponding to the current pixel between a predefined near and far plane, forming a sampling point sequence; For each point in the sampling point sequence, query the SDF field of the current frame training image to obtain its SDF value; The SDF value of each point in the sampling point sequence is converted into the opacity of multiple sampling intervals, and the rendering depth value of the current pixel is obtained by accumulating the opacity of the multiple sampling intervals point by point.

7. The method according to claim 1, characterized in that, The process of obtaining the absolute-scale pseudo-depth map of the current frame training image based on the intrinsic parameters and installation height of the vehicle-mounted monocular camera and the relative time difference map of the current frame training image includes: The ground pixel set of the current frame training image is extracted using a pre-trained semantic segmentation model; Based on the intrinsic parameters and installation height of the vehicle-mounted monocular camera, the ground pixel set of the current frame training image and its relative time difference map, the scale parameters of the affine model of relative disparity and true depth are obtained by solving the ground plane equation by least squares closed-form. Based on the scale parameters and the relative disparity map of the current frame training image, an absolute scale pseudo-depth map of the current frame training image is generated through an affine model of the relative disparity and the true depth.

8. An apparatus for training a monocular image 3D scene reconstruction model, characterized in that, The monocular image 3D scene reconstruction model is applied to a vehicle, which is equipped with an onboard monocular camera. The apparatus for training a monocular image 3D scene reconstruction model includes: The pseudo-depth map generation unit is used to perform depth estimation on the current frame training image through a pre-trained monocular depth estimation model to obtain the relative disparity map of the current frame training image, and to obtain the absolute scale pseudo-depth map of the current frame training image based on the intrinsic parameters and installation height of the vehicle-mounted monocular camera and the relative time difference map of the current frame training image. The forward inference unit is used to obtain the SDF field of the current frame training image from the monocular image 3D scene reconstruction model; A cross-modal depth distillation loss determination unit is used to render a rendered depth map in the original pose from the SDF field of the current frame training image through volume rendering, and to determine the cross-modal depth distillation loss based on the rendered depth map in the original pose and the absolute scale pseudo depth map of the current frame training image. The temporal self-supervised reprojection loss determination unit is used to estimate the current inter-frame relative pose of the vehicle-mounted monocular camera based on the IMU information corresponding to the current frame training image, render the next frame rendering image of the current frame training image from the SDF field of the current frame training image through volume rendering, and determine the temporal self-supervised reprojection loss based on the current inter-frame relative pose of the vehicle-mounted monocular camera, the next frame rendering image of the current frame training image, and the next frame real image of the current frame training image. The Eikonal loss determination unit is used to determine the Eikonal loss based on the SDF field of the training image in the current frame. The total loss determination unit is used to determine the total loss of the current frame training image based on the cross-modal depth distillation loss, temporal self-supervised reprojection loss, and Eikonal loss. The parameter update unit is used to update the parameters of the monocular image 3D scene reconstruction model based on the total loss of the training image in the current frame.

9. An electronic device, characterized in that, include: A processor and a memory storing a program, the program comprising instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a program, the program comprising instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images