A three-dimensional construction network training method, a three-dimensional model generation method and a device

Abstract

The present application provides a three-dimensional construction network training method, a three-dimensional model generation method and a device in the field of computer vision, which are used to combine multi-frame images and radar point cloud data to jointly train to obtain a three-dimensional construction network. The more accurate depth value included in the radar point cloud data can be used as depth supervision, so that the three-dimensional construction network can output a more accurate three-dimensional model. It includes: obtaining multiple frames of images and shooting parameters used by the camera device when shooting multiple frames of images, and the multiple frames of images include images shot from multiple perspectives; obtaining radar point cloud data including shooting scene data; obtaining a three-dimensional construction network through multiple frames of images, shooting parameters and radar point cloud data, and the radar point cloud data is used as depth supervision when the three-dimensional construction network is performing three-dimensional construction, so that the depth value of the three-dimensional model output by the three-dimensional framework network can be constrained to be closer to the depth value corresponding to the actual application scene.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to a three-dimensional construction network training method, a three-dimensional model generation method and a device. Background Art

[0002] Three-dimensional models need to be built in many scenarios, such as intelligent driving, AR or VR, which all require rendering of three-dimensional models. In some three-dimensional scene modeling and rendering solutions, the model of the three-dimensional scene can be output through a trained model. However, the color value or depth value of each point in the three-dimensional scene needs to be fitted by the model before output, which may result in inaccurate predictions. Therefore, how to improve the accuracy of three-dimensional scene construction has become an urgent problem to be solved. Summary of the invention

[0003] The present application provides a three-dimensional construction network training method, a three-dimensional model generation method and a device, which are used to combine multiple frame images and radar point cloud data to jointly train a three-dimensional construction network. The more accurate depth values ​​included in the radar point cloud data can be used as depth supervision, so that the three-dimensional construction network can output a more accurate three-dimensional model.

[0004] In view of this, in a first aspect, the present application provides a three-dimensional construction network training method, including: obtaining multiple frames of images and shooting parameters used by a camera device when shooting the multiple frames of images, the multiple frames of images including images shot from multiple perspectives; obtaining radar point cloud data, the acquisition scene when the radar acquires the radar point cloud data has an intersection with the shooting scene when the camera device shoots the multiple frames of images; obtaining a three-dimensional construction network through the multiple frames of images, the shooting parameters and the radar point cloud data, the three-dimensional construction network can be a network constructed based on neural radiance fields (NeRF), the three-dimensional construction network can be used to perform three-dimensional construction based on input data to output a three-dimensional model, the radar point cloud data is used as depth supervision when the three-dimensional construction network performs three-dimensional construction, so as to constrain the depth value of the three-dimensional model output by the three-dimensional framework network to be closer to the depth value corresponding to the actual application scenario.

[0005] Therefore, in the implementation manner of the present application, the information collected by the radar is used as the depth supervision of the model to obtain a more accurate depth, so that the constructed three-dimensional model is more accurate; and the three-dimensional model can be constructed using a real captured scene. Compared with constructing a three-dimensional model using artificially constructed images, the three-dimensional model output by the three-dimensional construction network obtained in the present application is more realistic.

[0006] In a possible implementation, the aforementioned acquisition of a three-dimensional construction network through multiple frame images, shooting parameters, and radar point cloud data may include: first, constructing a virtual camera according to the multiple frame images and the shooting parameters to obtain virtual camera data, wherein the virtual camera data includes images from perspectives different from multiple perspectives; then, using the virtual camera data as input to the three-dimensional construction network and using the radar point cloud data as deep supervision of the three-dimensional construction network, and iteratively training the initial model to obtain the three-dimensional construction network.

[0007] Therefore, in the implementation mode of the present application, a virtual camera can be constructed to generate images at different perspectives, thereby increasing the perspective input when training the three-dimensional construction network, which is equivalent to enriching the training data, thereby making the convergence result of the three-dimensional construction network better and improving the output accuracy of the three-dimensional construction network.

[0008] In a possible implementation, the aforementioned construction of a virtual camera based on multiple frames of images and shooting parameters to obtain virtual camera data includes: obtaining the posture parameters of each frame of the multiple frames of images; projecting the multiple frames of images into the same space according to the posture of each frame of the images to obtain image point cloud data; projecting the image point cloud data based on an enhanced perspective to obtain virtual camera data, wherein the enhanced perspective is a perspective different from the multiple perspectives.

[0009] Therefore, in the embodiments of the present application, the similarities between pixels at different perspectives in space can be combined to construct a virtual perspective and generate images of different perspectives, thereby increasing the perspective input for training the three-dimensional construction network and improving the output accuracy of the three-dimensional construction network.

[0010] In a possible implementation, the aforementioned use of radar point cloud data as depth supervision for a three-dimensional construction network includes: fusing radar point cloud data and image point cloud data to obtain depth values ​​of multiple pixels in the same space; and using the depth values ​​of multiple pixels as the three-dimensional construction network for depth supervision of three-dimensional construction.

[0011] Therefore, in the implementation mode of the present application, radar point cloud data can be added as depth supervision for training the three-dimensional construction network, so as to constrain the depth value output by the three-dimensional construction network during the training process of the three-dimensional construction network, so that the depth value of the three-dimensional model output by the three-dimensional construction network is closer to the true depth value, and thus the output three-dimensional model is more realistic.

[0012] In a possible implementation, any one of the aforementioned iterative trainings may include: obtaining at least one first sampling point in a first ray in a first perspective in a three-dimensional model output by the three-dimensional construction network obtained in a previous iteration; obtaining at least one second sampling point in a second ray in a second perspective in the three-dimensional model, where the at least one second sampling point is a point projected on the second ray by at least one first sampling point, and the first perspective and the second perspective are any different perspectives in the three-dimensional model; obtaining a difference between the first sampling point and the second sampling point; and updating the three-dimensional construction network according to the difference to obtain the three-dimensional construction network updated in the current iteration.

[0013] In an implementation manner of the present application, in the process of training the three-dimensional construction network, for the three-dimensional model output by each iteration of the three-dimensional construction network, rays at different perspectives can be collected and projected to obtain points that need to overlap between rays at different perspectives based on spatial geometric consistency. The output error of the three-dimensional construction network is determined based on the projection result, and the three-dimensional construction network is updated, so that the final output result of the three-dimensional construction network is more in line with the spatial geometric consistency.

[0014] In a possible implementation, the aforementioned obtaining the difference between the first sampling point and the second sampling point includes: obtaining a first depth estimation value of the first sampling point and a second depth estimation value of the second sampling point; and obtaining the difference between the first depth estimation value and the second depth value.

[0015] In the implementation manner of the present application, the reprojection error based on geometric consistency can be calculated from the depth dimension, so as to supervise the output result of the three-dimensional construction network from the depth position and improve the output accuracy of the three-dimensional construction network.

[0016] In a possible implementation, the aforementioned three-dimensional construction network includes: a coding module, a first geometry module, a first color module, a second geometry module, and a second color module;

[0017] The encoding module is used to encode the values ​​of the input pixel points and then input them into the first geometry module, the second geometry module and the second color module respectively;

[0018] The first geometry module and the second geometry module are used to perform geometry construction based on input data and output the geometry structure of the three-dimensional scene, and the output accuracy of the second geometry module is higher than the output accuracy of the first geometry module;

[0019] The first color module and the second color module are used to construct colors based on input data and output the color value of each pixel in the three-dimensional scene. The output accuracy of the second color module is higher than the output accuracy of the first color module. The input of the first color module includes the input of the first geometry module.

[0020] Therefore, in the implementation mode of the present application, high-precision and low-precision geometry and color construction modules are respectively set up, and the high-precision color module and the high-precision geometry module are decoupled, so that the parallel construction of geometry and color can be realized, and the output accuracy and output efficiency of the model can be improved.

[0021] In a possible implementation, the inputs of the aforementioned first color module and the second color module also include depth values, and the depth values ​​include values ​​obtained according to radar point cloud data, and may also include depth values ​​predicted by other modules in the three-dimensional construction network. Therefore, during the training process, the depth value may also be used as the input of the color module, so that the color module can construct the color in combination with the depth value, thereby improving the output accuracy of the color module.

[0022] In a possible implementation, the method may further include: acquiring an input posture and camera parameters; and acquiring an output image from the three-dimensional scene based on the input posture and camera parameters.

[0023] In a possible implementation, the three-dimensional model output by the aforementioned three-dimensional construction network is applied to intelligent driving of the vehicle, so that intelligent driving can be performed based on a more realistic three-dimensional model, thereby improving driving safety.

[0024] In a second aspect, the present application provides a three-dimensional model generation method, including: first, obtaining information of an input viewing angle, such as viewing angle posture, camera parameters, etc.; then, using the information of the input viewing angle as input of a three-dimensional construction network, and outputting a three-dimensional model. The three-dimensional construction network is obtained through multiple frame images, shooting parameters, radar point cloud data, and the three-dimensional construction network. The shooting parameters are parameters used by a camera device when shooting multiple frames of images. The multiple frames of images include images shot from multiple viewing angles. The radar point cloud data is data collected by the radar. The collection scene when the radar collects the radar point cloud data has an intersection with the shooting scene when the camera device shoots multiple frames of images.

[0025] In the implementation manner of the present application, when training the three-dimensional construction network, the information collected by the radar can be used as the depth supervision of the model to obtain a more accurate depth, so that the constructed three-dimensional model is more accurate; and the three-dimensional model can be constructed using a real shot scene. Compared with constructing a three-dimensional model using artificially constructed images, the three-dimensional model output by the three-dimensional construction network obtained in the present application is more realistic.

[0026] In a possible implementation, the three-dimensional construction network includes: a coding module, a first geometry module, a first color module, a second geometry module, and a second color module;

[0027] The encoding module is used to encode the values ​​of the input pixel points and then input them into the first geometry module, the second geometry module and the second color module respectively;

[0028] The first geometry module and the second geometry module are used to perform geometry construction based on input data and output the geometry structure of the three-dimensional model, and the output accuracy of the second geometry module is higher than the output accuracy of the first geometry module;

[0029] The first color module and the second color module are used to construct colors based on the input data and output the color value of each pixel in the three-dimensional model. The output accuracy of the second color module is higher than the output accuracy of the first color module. The input of the first color module includes the input of the first geometry module.

[0030] Optionally, the training process of the three-dimensional construction network may refer to the training process mentioned in the aforementioned first aspect or any optional implementation manner of the first aspect, which will not be described in detail here.

[0031] In a third aspect, the present application provides a three-dimensional construction network training device, comprising:

[0032] An image acquisition module, used to acquire multiple frames of images and shooting parameters used by a camera device when shooting multiple frames of images, wherein the multiple frames of images include images shot from multiple viewing angles;

[0033] The radar acquisition module is used to obtain radar point cloud data. The acquisition scene when the radar acquires radar point cloud data overlaps with the shooting scene when the camera device shoots multiple frames of images.

[0034] The processing module is used to obtain a three-dimensional construction network through multiple frame images, shooting parameters and radar point cloud data. The three-dimensional construction network is used to perform three-dimensional construction and output a three-dimensional model based on the input data. The radar point cloud data is used as depth supervision when the three-dimensional construction network performs three-dimensional construction.

[0035] In one possible implementation, the processing module is specifically used to: construct a virtual camera according to multiple frame images and shooting parameters to obtain virtual camera data, wherein the virtual camera data includes images from perspectives different from multiple perspectives; use the virtual camera data as input of a three-dimensional construction network and use the radar point cloud data as deep supervision of the three-dimensional construction network, and iteratively train the initial model to obtain the three-dimensional construction network.

[0036] In one possible implementation, the processing module is specifically used to: obtain the posture parameters of each frame of multiple frames; project the multiple frames of images into the same space according to the posture of each frame of images to obtain image point cloud data; project the image point cloud data based on an enhanced perspective to obtain virtual camera data, and the enhanced perspective is a perspective different from the multiple perspectives.

[0037] In a possible implementation, the processing module is specifically used to: fuse radar point cloud data and image point cloud data to obtain depth values ​​of multiple pixel points; and use the depth values ​​of the multiple pixel points as a three-dimensional construction network to perform depth supervision of three-dimensional construction.

[0038] In a possible implementation, any iterative training in the iterative training may include: obtaining at least one first sampling point in a first ray in a first perspective in a three-dimensional model output by the three-dimensional construction network obtained in the previous iteration; obtaining at least one second sampling point in a second ray in a second perspective in the three-dimensional model, where the at least one second sampling point is a point projected on the second ray by at least one first sampling point, and the first perspective and the second perspective are any different perspectives in the three-dimensional model; obtaining a difference between the first sampling point and the second sampling point; and updating the three-dimensional construction network according to the difference to obtain the three-dimensional construction network updated in the current iteration.

[0039] In a possible implementation, the processing module is specifically configured to: obtain a first depth estimation value of a first sampling point and a second depth estimation value of a second sampling point; and obtain a difference between the first depth estimation value and the second depth value.

[0040] In a possible implementation, the three-dimensional construction network includes: a coding module, a first geometry module, a first color module, a second geometry module, and a second color module;

[0041] The encoding module is used to encode the values ​​of the input pixel points and then input them into the first geometry module, the second geometry module and the second color module respectively;

[0042] The first geometry module and the second geometry module are used to perform geometry construction based on input data and output the geometry structure of the three-dimensional scene, and the output accuracy of the second geometry module is higher than the output accuracy of the first geometry module;

[0043] The first color module and the second color module are used to construct colors based on input data and output the color value of each pixel in the three-dimensional scene. The output accuracy of the second color module is higher than the output accuracy of the first color module. The input of the first color module includes the input of the first geometry module.

[0044] In a possible implementation, inputs of the first color module and the second color module further include depth supervision obtained based on radar point cloud data.

[0045] In a possible implementation, the device further includes:

[0046] Input module, used to obtain input pose and camera parameters;

[0047] The output module is used to obtain output images from the 3D scene based on the input pose and camera parameters.

[0048] In a possible implementation, the three-dimensional model output by the three-dimensional construction network can be applied to intelligent driving of a vehicle.

[0049] In a fourth aspect, the present application provides a three-dimensional model generation device, comprising:

[0050] An input module, used to obtain information of input viewing angle;

[0051] The output module is used to use the information of the input perspective as the input of the three-dimensional construction network and output the three-dimensional model. The three-dimensional construction network is obtained through multiple frame images, shooting parameters, radar point cloud data and the three-dimensional construction network. The shooting parameters are parameters used by the camera device when shooting multiple frame images. The multiple frame images include images shot from multiple perspectives. The radar point cloud data is data collected by the radar. The collection scene when the radar collects the radar point cloud data has an intersection with the shooting scene when the camera device shoots the multiple frame images.

[0052] In a possible implementation, the three-dimensional construction network includes: a coding module, a first geometry module, a first color module, a second geometry module, and a second color module;

[0053] The encoding module is used to encode the values ​​of the input pixel points and then input them into the first geometry module, the second geometry module and the second color module respectively;

[0054] The first geometry module and the second geometry module are used to perform geometry construction based on input data and output the geometry structure of the three-dimensional model, and the output accuracy of the second geometry module is higher than the output accuracy of the first geometry module;

[0055] The first color module and the second color module are used to construct colors based on the input data and output the color value of each pixel in the three-dimensional model. The output accuracy of the second color module is higher than the output accuracy of the first color module. The input of the first color module includes the input of the first geometry module.

[0056] In a possible implementation, the apparatus may further include a training module, which is used to train the three-dimensional construction network. The steps performed by the training module may refer to the description of the first aspect or any optional implementation of the first aspect, and will not be repeated here.

[0057] In a fifth aspect, an embodiment of the present application provides a three-dimensional construction network training device, comprising: a processor and a memory, wherein the processor and the memory are interconnected via a line, and the processor calls the program code in the memory to execute the processing-related functions in the three-dimensional construction network training method shown in any one of the first aspects above. Optionally, the three-dimensional construction network training device can be a chip.

[0058] In a sixth aspect, an embodiment of the present application provides a three-dimensional model generation device, comprising: a processor and a memory, wherein the processor and the memory are interconnected via a line, and the processor calls a program code in the memory to execute the processing-related functions in the three-dimensional model generation method shown in any one of the second aspects above. Optionally, the three-dimensional model generation device can be a chip.

[0059] In the seventh aspect, an embodiment of the present application provides a three-dimensional construction network training device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform functions related to processing in the above-mentioned first aspect or any optional embodiment of the first aspect.

[0060] In the eighth aspect, an embodiment of the present application provides a three-dimensional model generation device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform functions related to processing as described in the second aspect or any optional embodiment of the second aspect.

[0061] In a ninth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enables the computer to execute a method in any optional implementation of the first aspect or the second aspect above.

[0062] In the tenth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute a method in any optional implementation of the first aspect or the second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A schematic diagram of an artificial intelligence main framework for this application;

[0064] Figure 2 A schematic diagram of a system architecture provided for this application;

[0065] Figure 3 A schematic diagram of a neural network structure provided in an embodiment of the present application;

[0066] Figure 4 A schematic diagram of a computer system of a neural network provided in an embodiment of the present application;

[0067] Figure 5 A flowchart of a three-dimensional construction network training method provided in an embodiment of the present application;

[0068] Figure 6 A flowchart of another three-dimensional construction network training method provided in an embodiment of the present application;

[0069] Figure 7 A flowchart of another three-dimensional construction network training method provided in an embodiment of the present application;

[0070] Figure 8 A flowchart of another three-dimensional construction network training method provided in an embodiment of the present application;

[0071] Fig. 9 A flowchart of another three-dimensional construction network training method provided in an embodiment of the present application;

[0072] Fig.10 A schematic diagram of a three-dimensional network structure provided in an embodiment of the present application;

[0073] Fig.11 A schematic diagram of a projection scene with different viewing angles provided in an embodiment of the present application;

[0074] Fig.12 A flowchart of another three-dimensional construction network training method provided in an embodiment of the present application;

[0075] Fig.13 A schematic diagram of a process for generating a three-dimensional model provided in an embodiment of the present application;

[0076] Fig.14 A schematic diagram of the structure of a three-dimensional network training device provided in an embodiment of the present application;

[0077] Fig.15 A schematic diagram of the structure of a three-dimensional model generating device provided in an embodiment of the present application;

[0078] Fig.16 A schematic diagram of the structure of another three-dimensional construction network training device provided in an embodiment of the present application;

[0079] Fig.17 A schematic diagram of the structure of another three-dimensional model generating device provided in an embodiment of the present application;

[0080] Fig.18 A schematic diagram of a chip structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0082] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0083] (1) Infrastructure

[0084] Infrastructure provides computing power support for artificial intelligence systems, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips, such as central processing units (CPU), neural-network processing units (NPU), graphics processing units (GPU), application specific integrated circuits (ASIC) or field programmable gate arrays (FPGA) and other hardware acceleration chips; the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0085] (2) Data

[0086] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0087] (3) Data processing

[0088] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0089] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0090] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0091] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0092] (4) General ability

[0093] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0094] (5) Smart products and industry applications

[0095] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, smart driving, smart cities, etc.

[0096] In some scenarios, such as smart terminals, smart transportation, smart medical care, smart driving, smart cities, etc., models need to be constructed. For example, in the process of building an AR map, the point cloud data of the map can be collected by laser, and then the point cloud data can be used to build the AR map. Alternatively, in a smart terminal, the point cloud data of the currently captured scene can be collected by the camera, and then a three-dimensional model of the current scene can be built based on the point cloud data, and then applied to the terminal's image processing or games, thereby improving the user experience. That is, the method provided in this application is valuable in AI systems, terminal applications, or cloud services.

[0097] In many application scenarios, 3D reconstruction is required to obtain a 3D model that matches the environment. For example, in smart driving, AR, VR or other scenarios that require rendering, rendering can be performed based on images to obtain a more three-dimensional 3D model.

[0098] With the rapid rise of neural radiance fields (NeRF) in the field of computational vision, the three-dimensional model generation method based on neural radiance fields has also demonstrated its unique advantages over other traditional methods in application scenarios such as intelligent driving, digital humans, and three-dimensional scene modeling.

[0099] Traditional 3D scene modeling and rendering methods based on computer graphics usually require explicit simulation of the model in the scene, and render the 3D scene model into image data at a specific angle through a rendering engine. The 3D model generation method based on computer graphics requires a lot of manual modeling processes, and the quality of image generation depends on the refinement of the model in the scene and the ability of the image rendering engine. If large-scale image generation is required, a large amount of art investment and accumulation of 3D assets are required. The image generation route based on neural radiance fields, on the other hand, uses scene construction data from real scene collection data, so the scene reconstruction and image synthesis have higher realism. For applications such as intelligent driving that require large-scale image generation, the scale and diversity of the scene can also grow exponentially with the amount of collected data. Neural radiance fields (NeRF) is an emerging scene expression and image rendering method that records the scene expression in a deep neural network through implicit expression, uses a deep neural network to implicitly learn a static 3D scene, and indirectly completes tasks such as 3D reconstruction of the scene and novel view synthesis.

[0100] Usually, the input of NeRF is a set of image data from multiple perspectives with known camera parameters. Based on these images, a neural network is used to implicitly learn a static three-dimensional scene. The neural network trained based on these images combined with volume rendering technology can infer and render the corresponding image data and corresponding spatial depth information from any new viewpoint. Volume rendering technology refers to calculating the occupancy and color information of the space along any ray emitted from the camera's perspective in the scene. For all points on this ray, the occupancy information and color information can be accumulated in the form of integration, and finally the color and depth information of the pixel corresponding to this ray is obtained.

[0101] For example, images can be generated using the NeRF framework. The underlying continuous volume scene function can be optimized by inputting a set of sparse views to achieve new views of complex scenes. This method represents the scene using a fully connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial position (x, y, z) and viewing direction (θ, φ)), and whose output is the volume density and view-related color information at that spatial position. The view is synthesized by querying the 5D coordinate along the camera ray, and the output color and density are projected into the image using classic volume rendering techniques. Since volume rendering is naturally differentiable, the only input required for optimization is a set of images with known camera poses. However, in this image generation method, the direct input of the network is the sampling points on the ray of the image after position encoding, and the optimization of the network is also that each sampling point is trained and estimated by the network separately. In the case of sparse data perspective (such as intelligent driving scenes, large outdoor scenes), it is often difficult to recover relatively good geometric information. In addition, the geometric consistency between the image pixel ray sampling points is not well constrained. The depth output of the network is obtained by integrating volume rendering, and the depth prediction value is relatively rough, and the effect of generating new perspective images is often poor.

[0102] For another example, in some NeRF network-based generation methods, indoor RGB images can be synthesized from relatively sparse input perspectives. The sparse input perspectives result in fewer input images for training the NeRF network. The pose data and sparse feature point clouds corresponding to the input images are obtained through preprocessing with the structure from motion (SFM) algorithm. Secondly, this method uses the deep completion network to obtain a dense depth map and its depth uncertainty from the sparse point cloud obtained by preprocessing, which is used to guide the training of the NeRF network. However, although the depth prior information is introduced, the dense depth information comes from the deep completion network, and the uncertainty of its depth information is high, and the acquisition of depth information is limited by the deep completion network (for example, a deep completion network must be trained first). In addition, multiple views are trained independently of each other, and the geometric structures of multiple views are not constrained to be consistent in the global space.

[0103] Therefore, the present application provides a three-dimensional construction network training method and a three-dimensional model generation method, which are used to combine data collected by various types of sensors to learn the NeRF-based network architecture to generate a more accurate three-dimensional model, and then generate accurate images from various perspectives based on the accurate three-dimensional model.

[0104] Specifically, the present application provides a 3D construction network training method, which can collect multiple frames of images taken from multiple perspectives in an actual scene and the corresponding shooting parameters, and collect radar point cloud data through radar, and can send electromagnetic waves in the same scene through radar, and generate radar point cloud data based on the received echo signal. Subsequently, a 3D scene is generated through multiple frames of images, corresponding shooting parameters, radar echo data and a 3D construction network, and the 3D construction network can be used for 3D construction based on input data, wherein the radar point cloud data can be used as depth supervision when performing 3D reconstruction to improve the depth accuracy of each pixel in the 3D scene.

[0105] The three-dimensional model generation method provided in the present application can output a three-dimensional model based on the three-dimensional construction network obtained by the aforementioned three-dimensional construction network training method, thereby obtaining an image of a desired perspective from the three-dimensional model.

[0106] The aforementioned three-dimensional construction network training method can be understood as a training process, and the three-dimensional model generation method can be understood as the reasoning side. The three-dimensional construction network training method and the three-dimensional model generation method provided in the present application can be applied to terminals, servers or cloud platforms, etc. For example, the three-dimensional construction network training method provided in the present application can be deployed on a cloud platform, and the three-dimensional construction network can be trained and output through the cloud platform. The three-dimensional construction network can be sent down and deployed in the terminal, and the terminal can output the three-dimensional model through the three-dimensional construction network and capture the image of the desired perspective from the three-dimensional model. Alternatively, after the cloud platform trains the three-dimensional construction network, the three-dimensional model output by the three-dimensional construction network can be sent down to the terminal, and the terminal can capture the image of the desired perspective from the three-dimensional model.

[0107] Some possible system architectures provided by this application are introduced as examples below.

[0108] For example, Figure 2 As shown, the 3D construction network training method provided in the present application can be executed by the server cluster 310, that is, the 3D construction network is trained, and the trained 3D construction network is sent to the terminal 301 through the communication network to deploy the 3D construction network in the terminal 301. The terminal can input the camera pose or camera parameters at any position or angle, and can infer the 3D model corresponding to the scene under this camera pose through the 3D construction network. When the terminal includes a vehicle, the 3D model can be used for intelligent driving, real-time navigation, etc. of the vehicle, or the 3D model can be applied to the AR game of the terminal, so that the AR game can be combined with the real scene where the user is, and the user experience is improved.

[0109] Exemplarily, the three-dimensional construction network training method provided in the present application can be executed by a server, and the trained three-dimensional construction network can be deployed in the server. The server can be used to execute the three-dimensional construction network training method provided in the present application. The server can receive radar point cloud data, multi-frame images, and corresponding shooting parameters sent by the client, or extract radar point cloud data, multi-frame images, and corresponding shooting parameters from locally stored data, and then train the three-dimensional construction network through the three-dimensional construction network training method provided in the present application. If the point cloud data is sent from the client to the server, the server can feed back the trained three-dimensional construction network to the client.

[0110] The embodiments of the present application involve related applications in the field of neural networks and images. In order to better understand the solutions of the embodiments of the present application, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0111] (1) Neural radiance fields (NeRF)

[0112] The scene expression is recorded in the deep neural network through implicit expression, and a static three-dimensional scene is implicitly learned using the deep neural network to indirectly complete tasks such as three-dimensional reconstruction of the scene and new perspective image generation (Novel View Synthesis).

[0113] like Figure 3 As shown in the figure, the input of NeRF is a set of image data (must be multi-view) with known camera parameters (camera intrinsic parameters and camera posture), and a neural network is used to implicitly learn a static three-dimensional scene based on these images. The neural network trained based on these images combined with volume rendering technology can infer and render the corresponding image data and corresponding spatial depth information from any new viewpoint. Volume rendering technology refers to calculating the occupancy degree and color information of the space along any ray emitted from the camera's perspective in the scene. For all points on this ray, the occupancy information and color information can be accumulated in the form of integrals, and finally the color and depth information of the pixel corresponding to this ray is obtained. As shown in the figure Figure 4 As shown in Figure 1, the actual implementation process is to achieve discrete sample space modeling through finite sampling points on the sampling ray and accumulation. For the pinhole camera model, each pixel in the image can be accumulated for training and reasoning in this volume rendering manner.

[0114] It should be noted that the three-dimensional construction network provided in the present application can be a network built based on NeRF, which can be used to output a rendered three-dimensional model.

[0115] (2) Loss function: also known as cost function, a measure of the difference between the predicted output of a machine learning model for a sample and the true value of the sample (also known as the supervised value), that is, it is used to measure the difference between the predicted output of a machine learning model for a sample and the true value of the sample. The loss function can usually include mean square error, cross entropy, logarithm, exponential loss functions, etc. For example, the mean square error can be used as a loss function, defined as The specific loss function can be selected according to the actual application scenario.

[0116] It is understandable that the training process of the neural network is to learn the way to control the spatial transformation, more specifically, to learn the weight matrix. The purpose of training the neural network is to make the output of the neural network as close to the expected value as possible. Therefore, the weight vector of each layer of the neural network in the neural network can be updated according to the difference between the predicted value and the expected value of the current network (of course, the weight vector can usually be initialized before the first update, that is, the parameters of each layer in the deep neural network are pre-configured). For example, if the predicted value of the network is too high, the value of the weight in the weight matrix is ​​adjusted to reduce the predicted value, and after continuous adjustment, the value output by the neural network is close to or equal to the expected value. Specifically, the difference between the predicted value and the expected value of the neural network can be measured by a loss function or an objective function. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. The training of the neural network can be understood as the process of minimizing the loss as much as possible. The process of updating the weight of the starting network and training the serial network in the following embodiments of the present application can refer to this process, which will not be repeated below.

[0117] (3) Gradient: The derivative vector of the loss function with respect to the parameters.

[0118] (4) Back propagation (BP): An algorithm that calculates the gradient of model parameters based on the loss function and updates the model parameters.

[0119] (5) Point cloud: data formed by multiple points, each point has corresponding information, such as depth, brightness or intensity.

[0120] (6) RGB (red green blue)

[0121] RGB stands for the colors of the three channels of red, green and blue. A variety of colors can be obtained by changing the three color channels of red (R), green (G) and blue (B) and superimposing them on each other.

[0122] The present application provides a three-dimensional construction network training method and a three-dimensional model generation method. It can be understood that the three-dimensional construction network training method and the three-dimensional model generation method provided by the present application are respectively a training phase and an inference phase. In the training phase, the training input perspective of the three-dimensional construction network is increased by constructing a virtual perspective, and radar point cloud data is used as deep supervision to train and output a more accurate three-dimensional construction network. In the inference phase, the prediction results of the three-dimensional construction network obtained in the training phase can be used to perform three-dimensional reconstruction to obtain a more accurate three-dimensional model.

[0123] Among them, the training phase and the inference phase can be deployed in the same device or in different devices. For example, the training phase can be performed in a cloud platform or server, the training data can be collected by the client, and the three-dimensional construction network obtained after training can be sent down and deployed on the client, so that the client can output a more accurate three-dimensional model. For another example, both the training phase and the inference phase can be deployed in a local terminal, the terminal can collect multiple frames of images and radar point cloud data, and train a three-dimensional construction network, and then the trained three-dimensional construction network can be used to output a three-dimensional model at any perspective or posture.

[0124] The method provided by the present application is divided into different stages such as a training stage and a reasoning stage for introduction below. The training stage is the neural network training method provided by the present application, and the reasoning stage is the process of the three-dimensional model generation method provided by the present application.

[0125] 1. Training Phase

[0126] See also Figure 5 , a flow chart of a three-dimensional construction network training method provided in this application is described as follows.

[0127] 501. Obtain multiple frames of images and shooting parameters.

[0128] The multiple frames of images may be obtained by taking pictures from multiple perspectives using a camera device, and each perspective may take one or more frames of images. The shooting parameters are shooting parameters used by the camera device when taking pictures of multiple frames of images, and may specifically include parameters such as camera intrinsic parameters or camera attitude. For example, camera intrinsic parameters may include preset parameters, such as the pixel, resolution, focal length and other information of the camera, and the camera attitude may be determined by the camera coordinate system and the world coordinate system.

[0129] For example, when the method provided in the present application is deployed in a vehicle, the imaging device may include a camera installed in the vehicle or a camera or other imaging terminal in the same scene as the vehicle.

[0130] The multi-frame images can specifically be images captured by RGB images, depth images, event images, or other types of cameras in real scenes, etc., and can be adjusted according to the actual application scenario, and this application does not limit this. The RGB images mentioned below in this application can also be replaced by depth images or event images, etc., which will not be repeated below.

[0131] 502. Obtain radar point cloud data.

[0132] The radar point cloud data may include data collected by the radar. For example, the radar may emit electromagnetic waves to the scene to be constructed in three dimensions, receive echo signals, and generate radar point cloud data based on the echo signals. The radar point cloud data may include depth information of multiple points in the scene.

[0133] It should be noted that the present application does not limit the execution order of step 501 and step 502. Step 501 may be executed first, or step 502 may be executed first, or step 501 and step 502 may be executed simultaneously, depending on the actual application scenario.

[0134] 503. A three-dimensional construction network is obtained by training multiple frames of images, shooting parameters and radar point cloud data.

[0135] Subsequently, the initial 3D construction network can be trained by using multiple frames of images, shooting parameters, and radar point cloud data to obtain a trained 3D construction network. The trained 3D construction network can be used to output a 3D scene from any perspective. Among them, the radar point cloud data can be used for depth supervision when training the 3D construction network, thereby improving the depth accuracy of the 3D scene output by the trained 3D construction network.

[0136] Therefore, in the implementation mode of the present application, the three-dimensional construction network can be trained in combination with images of multiple perspectives collected in the real scene and the shooting parameters, and depth supervision can be performed based on the point cloud data collected by the radar, so that the depth of the three-dimensional scene output by the three-dimensional construction network is more accurate.

[0137] In a possible implementation, a virtual camera can be constructed based on multiple frames of images and shooting parameters to obtain virtual camera data, which includes images from multiple perspectives; the virtual camera data is used as the input of the 3D construction network and the radar point cloud data is used as the deep supervision of the 3D construction network, and the initial model is iteratively trained to obtain the 3D construction network. Therefore, in the implementation of the present application, the training data for training the 3D construction network can be increased by constructing a virtual camera, thereby improving the output accuracy of the 3D construction network.

[0138] In a possible implementation, the specific process of constructing a virtual camera may include: first obtaining the pose parameters of each frame of the image, that is, the parameters representing the pose of each frame of the image in the three-dimensional scene, projecting each frame of the image into the same space according to the pose of each frame of the image, obtaining image point cloud data, and then projecting the image point cloud data based on an enhanced perspective to obtain virtual camera data, wherein the enhanced perspective includes a perspective different from the shooting perspective of the multiple frames of the image. In the implementation of the present application, multiple frames of images may first be projected into the same space to obtain image point cloud data in the same space, so that image data of more perspectives may be collected from the image point cloud data to obtain virtual camera data. It can be understood that after projecting multiple frames of images into the same space, images of perspectives different from the shooting perspective may be collected from the space, thereby increasing the training input perspective of the three-dimensional construction network and improving the output accuracy of the trained three-dimensional construction network.

[0139] Optionally, when performing depth supervision through radar point cloud data, the radar point cloud data and the image point cloud data can be fused to obtain the depth values ​​of multiple pixels in the image point cloud data, and then the depth values ​​of the multiple pixels are used as the depth supervision when the 3D construction network performs deep 3D reconstruction to train the 3D construction network. Usually, the data collected through radar echoes can include more accurate depth values, so the more accurate depth values ​​can be used as the depth supervision when training the 3D construction network to make the depth value of the 3D scene output by the 3D construction network closer to the real depth value, thereby improving the output accuracy of the 3D construction network.

[0140] Optionally, the three-dimensional construction network may specifically include a coding module, at least two geometry modules and at least two color modules, the geometry module may be used to reconstruct the geometry structure in the three-dimensional scene, and the color module may be used to reconstruct the color in the three-dimensional scene. The at least two geometry modules may be divided into a low-precision module (or referred to as a first geometry module) and a high-precision module (or referred to as a second geometry module), the output precision of the high-precision module is higher than the output precision of the low-precision module, and generally the structural complexity of the high-precision geometry module is higher than the structural complexity of the low-precision geometry module. Similarly, the aforementioned at least two color modules may also be divided into a low-precision color module (or referred to as a first color module) and a high-precision color module (or referred to as a second color module), the output precision of the high-precision color module is higher than the output precision of the low-precision color module, and generally the structural complexity of the high-precision color module is higher than the structural complexity of the low-precision color module.

[0141] Among them, the encoding module can be used to encode the input data, and the output of the encoding module is respectively connected to the input ends of the low-precision geometry module, the high-precision geometry module and the high-precision color module, the output end of the low-precision geometry module is connected to the output module of the three-dimensional construction network, the output end of the low-precision geometry module is also connected to the input end of the low-precision color module, the output end of the high-precision geometry module and the output end of the high-precision color module are respectively connected to the output module, and the input ends of the low-precision color module and the high-precision color module can also include depth information collected by radar.

[0142] Therefore, in the implementation manner of the present application, the three-dimensional construction network can add a high-precision geometry module and a high-precision color module to the NeRF network, and decouple the reconstruction of color and the reconstruction of geometry, so that geometry reconstruction and color reconstruction can be performed in parallel to improve the efficiency of three-dimensional reconstruction.

[0143] Specifically, any iterative process in iterative training of the three-dimensional construction network may include: obtaining at least one first sampling point in the first ray in the first perspective of the three-dimensional model output by the three-dimensional construction network obtained in the previous iteration; obtaining at least one second sampling point in the second ray in the second perspective of the three-dimensional model, at least one second sampling point is a point projected on the second ray by at least one first sampling point, and the first perspective and the second perspective are any different perspectives in the three-dimensional model; obtaining the difference between the first sampling point and the second sampling point; updating the three-dimensional construction network according to the difference to obtain the three-dimensional construction network updated in the current iteration. In the implementation manner of the present application, points where rays in different perspectives may intersect may be collected as sampling points, so that the three-dimensional construction network is updated based on the difference between the sampling points on different rays, so that the points where rays in different perspectives may intersect in the three-dimensional scene output by the three-dimensional construction network are closer to the intersection, thereby improving the realism of the three-dimensional scene output by the three-dimensional construction network.

[0144] Optionally, in the process of calculating the difference between sampling points on different rays, a first depth estimation value of the first sampling point and a second depth estimation value of the second sampling point may be estimated, thereby obtaining the difference between the first sampling point and the second sampling point based on the difference between the first depth estimation value and the second depth estimation value. In the implementation of the present application, points that may intersect between different perspectives in the three-dimensional scene output by the three-dimensional construction network may be made to intersect as much as possible, thereby improving the realism of the three-dimensional scene.

[0145] The above describes the process of the three-dimensional construction network training method provided by the present application. The following is a more detailed introduction to the three-dimensional construction network training method provided by the present application in combination with specific application scenarios.

[0146] For example, the process of the three-dimensional construction network training method provided in this application can be as follows: Figure 6As shown, a three-dimensional construction network is a network constructed based on the NeRF network (hereinafter referred to as the NeRF network) as an example for exemplary introduction. After collecting multiple frames of images of the real scene from multiple perspectives, the three-dimensional reconstruction algorithm is used to estimate the pose of each frame of the image. The multiple frames of images can be projected into the same space to obtain dense RGB point cloud data, or image point cloud data. Then, based on the RGB point cloud data, virtual camera data is constructed. Both the virtual camera data and the RGB point cloud data are used as training data, and the radar point cloud data is used as deep supervision to train a three-dimensional construction network with geometric and color decoupling. The stereoscopic vision reprojection error can also be estimated based on geometric consistency, and the three-dimensional construction network can be updated based on the error to obtain a three-dimensional construction network that has been updated again.

[0147] The 3D construction network provided in this application can be applied to a variety of scenarios, such as intelligent driving, AR / VR scenarios, high-precision mapping scenarios, or 3D reconstruction scenarios. Taking the intelligent driving scenario as an example, the detailed process of the 3D construction network training method provided in this application is introduced.

[0148] 1. Generate virtual camera based on tracks

[0149] After collecting multiple frames of data, the precise pose information of each image can be obtained through a 3D reconstruction algorithm, and then RGB dense point cloud data can be obtained through dense multi-view stereo matching. The 3D reconstruction algorithm can specifically include but is not limited to structure from motion (SFM) or a 3D reconstruction algorithm based on deep learning. This application exemplarily introduces the SFM algorithm as an example. The SFM algorithm mentioned below can also be replaced by other 3D reconstruction algorithms, which will not be repeated below.

[0150] For example, the generation process of the virtual camera can be as follows: Figure 7 shown.

[0151] The multi-frame images captured by the camera from multiple perspectives are used as the input of the SFM algorithm. The SFM algorithm is used to extract tracks feature points and corresponding pose data from the multi-frame images, that is, feature points of multiple observation points in the same space. Specifically, the SFM algorithm can be used to extract stable features in the image to estimate the spatial point cloud. From the multiple perspectives corresponding to the multi-frame images, the real perspective t and the real perspective t+1 are selected, and the real perspective t and the real perspective t+1 can be interpolated to obtain one or more virtual perspectives i (or enhanced perspectives).

[0152] Subsequently, based on the key feature points that appear in both the previous and next two frames of the virtual perspective i, the image within the virtual perspective range is reprojected, that is, the important feature points are projected to the virtual perspective, thereby generating a virtual perspective image.

[0153] For example, a scene that generates a virtual camera can be created as follows Figure 8 As shown in the figure, based on the common view assumption, the interpolated view of a certain track feature point in the space between the two original frames can also be considered to be continuously observed, and the neighborhood information of this feature point is also relatively stable. Therefore, a new view can be obtained by interpolation from the two views, and a virtual camera view image can be obtained by projection based on the neighborhood information.

[0154] This is equivalent to inputting the original training image data sequence in the implementation mode of the present application, and using the SFM algorithm to obtain the tracks feature points in the scene, that is, the track of a three-dimensional point connected by the matching points corresponding to different perspectives, and the pose data corresponding to each track feature point. The pose data of the virtual perspective can be obtained based on the common view assumption and pose interpolation. Then, the camera data of the virtual perspective is obtained by back-projection of the tracks feature points.

[0155] Usually, the data perspective in large outdoor scenes and intelligent driving scenes is often limited, resulting in the sparsity of the training data perspective, which ultimately affects the accurate recovery of the scene's geometric structure. The generation of virtual cameras based on tracks feature points can increase the perspective input of training data, enhance the importance of tracks feature points in the NeRF network, and improve the recovery of scene geometric information. The tracks feature points obtained by the SFM algorithm have strong features and strong geometric consistency in space. By generating virtual camera perspectives, the NeRF network's memory and learning of these strong feature points can be enhanced, improving the accuracy of recovering scene geometric information. And usually, the color of an object is related to the observation angle of the camera. By increasing the observation angle, the training data for the NeRF network can be increased, and the output accuracy of the NeRF network can be improved.

[0156] It can be understood that this application proposes a virtual camera generation method based on tracks, a display model of the scene generated based on SFM, enhances the role of strong feature points in the implicit expression of the scene, increases the training input perspective of the NeRF network, and can weaken the problem of perspective limitations in intelligent driving scene data collection.

[0157] 2. Deep Supervision

[0158] The input image sequence can be used to obtain dense RGB color point clouds and accurate camera pose data through the SFM algorithm. However, the scale obtained from the image may be different from the scale in the scene. Therefore, in order to make the scale learned by the NeRF network more accurate, a point cloud registration algorithm containing scale can be used to align the radar point cloud data and the RGB point cloud data to obtain the fused point cloud data as deep supervision for NeRF network training.

[0159] For example, Fig. 9 As shown in the figure, in addition to obtaining RGB point cloud data through the SFM algorithm, radar point cloud data can also be obtained by collecting multiple frames of input data through radar and accumulating multiple frames of data. The radar point cloud data and RGB point cloud data are aligned, and specifically, they can be registered through algorithms such as scaling iterative closest point (SICP) or scaling generalized iterative closest point (SGICP) to obtain fused point cloud data.

[0160] Generally, the radar point cloud data collected by radar can more accurately reflect the depth of each point in the scene. Aligning the depth value included in the radar point cloud data with each pixel in the RGB point cloud data is equivalent to assigning a true depth value to each pixel in the RGB point cloud data, which serves as the depth supervision for training the NeRF network, so that the estimated depth value of the three-dimensional model output by the NeRF network is closer to the true depth value and more accurate.

[0161] 3. 3D network structure and its training

[0162] The three-dimensional construction network may be a network constructed based on a NeRF network. For example, taking the three-dimensional construction network including four construction modules as an example, the structure of the three-dimensional construction network provided in the present application may be as follows: Fig.10 Show.

[0163] The three-dimensional building module may include at least one encoding module and a plurality of building modules.

[0164] The encoding module can be used to convert input data into a vector that can be recognized by the subsequent network. The encoding module can specifically adopt a model such as a hash grid encoding module (Hash-Grid Encoder), a time domain filter or a spatial filter. This application exemplarily introduces the encoding module as a hash grid encoding module as an example. The hash grid encoding module mentioned below can also be replaced by other encoding models, which will not be repeated below.

[0165] The building module may adopt a multilayer perceptron (MLP), and of course may be replaced by other neural networks, which is not limited here, and the structures or weights and other parameters between different building modules may be different.

[0166] For example, the three-dimensional building module may specifically include a hash grid coding module, a Sigma MLP, an RGB MLP, a SigmaMLP*, and an RGB MLP* module.

[0167] Sigma MLP and Sigma MLP* can be used to geometrically reconstruct a 3D scene, and the output accuracy of Sigma MLP* is higher than that of Sigma MLP. Generally, the network structure complexity of Sigma MLP* is higher than that of Sigma MLP, such as the number of network layers of Sigma MLP* is greater than the number of network layers of Sigma MLP, or the number of neurons of SigmaMLP* is greater than the number of neurons of Sigma MLP, etc.

[0168] RGB MLP and RGB MLP* can be used to reconstruct the color of a three-dimensional scene, and the output accuracy of RGB MLP* is higher than that of RGB MLP. Generally, the structural complexity of RGB MLP* is higher than that of RGB MLP, such as the number of network layers of RGB MLP* is greater than the number of network layers of RGB MLP, the weight parameters of RGB MLP* are greater than those of RGB MLP, etc., thereby achieving higher-precision output through a more complex network structure.

[0169] Among them, the input of Sigma MLP, Sigma MLP* and RGB MLP* includes the output of the hash grid coding module, and the input of RGB MLP can include the output of Sigma MLP. During the training process, the input of RGB MLP can also include depth supervision, that is, the aforementioned fused point cloud data, and the input of RGB MLP* can also include fused point cloud data. Generally, the radar point cloud data collected by radar can more accurately represent the depth value in the real scene, so the radar point cloud data can be used as depth supervision or depth guidance to make the depth value of the three-dimensional scene output by the NeRF network closer to the depth value in the actual scene.

[0170] After obtaining the RGB point cloud data and the fused point cloud data, the specific training process may include: the network can be pre-trained using the fused point cloud data (the first round of training), and then the initial weights of the NeRF network can be obtained; then the initialized scene model may be re-tuned using the RGB image sequence in the training set (the second round of training) to obtain the trained NeRF network, which can output voxels of the three-dimensional scene.

[0171] Therefore, in the implementation of the present application, the depth value in the three-dimensional scene can be supervised by fusing the real depth value included in the point cloud data, so that the depth value of the three-dimensional scene output by the NeRF network is closer to the depth value in the actual scene, thereby improving the accuracy of the network's depth output. And for high-precision modules, the geometry (sigma) and color (color) estimation networks are decoupled to improve the parallel degree of the network model and the color fitting ability, improve the rendering quality of RGB images, and accelerate the training and reasoning efficiency of the NeRF network.

[0172] 4. Stereoscopic Reprojection Error

[0173] Generally, in order to improve the NeRF network, the observation values ​​of the same sampling point from different perspectives can be used to further correct the output results of the NeRF network, thereby improving the output accuracy of the NeRF network. For example, for a neural radiation field constructed by multiple views, different adjacent perspectives often have observability for the same object. Therefore, adding geometric consistency constraints to the observation of the same object can make the network output more accurate and improve the NeRF network's ability to estimate volume density.

[0174] For example, the reprojection error can be calculated as Fig.11 and Fig.12 As shown, for each pixel at the viewing angle f, the emitted ray R f (o f ,d f ), sample along the ray to obtain a series of sample points z f , the NeRF network can be used to estimate the weights ω of these sample points f (z f ), and then synthesize the depth and color information of the corresponding pixel. For another view t, a series of rays can be emitted to hit the sample points of the rays of view f to evaluate the consistency of the light rays of the two views in the same scene. The consistency of view t can be described by the following formula:

[0175]

[0176] in, is the depth value of the sampling point in the perspective f projected on the perspective t, K ∈ is a Gaussian distribution, ∈ is half the length of the neighborhood interval. The consistency of the multi-view neural radiation field can be described as follows:

[0177]

[0178] In the formula, is the depth estimate of any pixel ray at view f.

[0179] Based on the designed consistent neural radiance field, the multi-view stereo vision reprojection error (stereo loss) is used to optimize the consistency of the model, while integrating the geometric error and photometric (color) error of the scene. In the actual optimization process, small image blocks are used as the basic unit for training to avoid ambiguity of single pixels in weak texture areas or repeated texture areas. Stereo loss is designed as follows:

[0180]

[0181] In the formula, I f is the pixel corresponding to the small block of the original training image at view angle f, The pixels corresponding to the small image block are accumulated according to the weights estimated by the sampling points at the perspective t. Stereo loss can better constrain the consistency of different perspectives of the scene and improve the geometric quality estimation of the scene without affecting the RGB image rendering.

[0182] 2. Reasoning Stage

[0183] See also Fig.13 , a flow chart of a three-dimensional model generation method provided in this application is as described below.

[0184] 1301. Obtain input perspective information.

[0185] The input posture information may specifically include information such as the camera posture or camera parameters of the input viewing angle.

[0186] For example, in an intelligent driving scenario, when it is necessary to capture images of the vehicle's driving direction, the perspective of the vehicle's current driving direction can be input to obtain a three-dimensional scene in front of the vehicle.

[0187] 1302. Use the input perspective information as input to a three-dimensional construction network, and output a three-dimensional model.

[0188] Among them, the trained three-dimensional construction network can be obtained through the three-dimensional construction network training method provided in the present application, the information of the input perspective is used as the input of the three-dimensional construction network, and the three-dimensional model is output through the trained three-dimensional construction network. The three-dimensional model can be used to represent the three-dimensional image under the input perspective.

[0189] It should be noted that the training process of the three-dimensional construction network can refer to the aforementioned Figure 5-Figure 12 The relevant description in will not be repeated here.

[0190] Therefore, in the implementation mode of the present application, the three-dimensional construction network can be trained in combination with the images of multiple perspectives collected in the real scene and the shooting parameters, and the depth supervision can be performed based on the point cloud data collected by the radar, so that the depth of the three-dimensional scene output by the three-dimensional construction network is more accurate, thereby outputting a more accurate three-dimensional model through the trained three-dimensional construction network. For example, in the field of intelligent driving, as a new generation of image data synthesizer applied to the simulation platform of intelligent driving, it has lower cost, higher image fidelity and better scene diversity than the CG image synthesis method. For example, in the field of new perspective image synthesis, key indicators of image rendering quality, such as peak signal to noise ratio (PSNR), have reached SOTA (State of the arts), achieving better three-dimensional modeling effects.

[0191] The above is an introduction to the process of the method provided in the present application. The following is an introduction to the device for executing the method provided in the present application in combination with the above method process.

[0192] See also Fig.14 , a schematic diagram of the structure of a three-dimensional construction network training device provided in the present application, the three-dimensional construction network training device comprises:

[0193] An image acquisition module 1401 is used to acquire multiple frames of images and shooting parameters used by a camera device when shooting multiple frames of images, wherein the multiple frames of images include images shot from multiple perspectives;

[0194] A radar acquisition module 1402 is used to acquire radar point cloud data. The acquisition scene when the radar acquires the radar point cloud data overlaps with the shooting scene when the camera device shoots multiple frames of images.

[0195] The processing module 1403 is used to obtain a three-dimensional construction network through multiple frame images, shooting parameters and radar point cloud data. The three-dimensional construction network is used to perform three-dimensional construction and output a three-dimensional model based on the input data. The radar point cloud data is used as depth supervision when the three-dimensional construction network performs three-dimensional construction.

[0196] In one possible implementation, the processing module 1403 is specifically used to: construct a virtual camera based on multiple frame images and shooting parameters to obtain virtual camera data, where the virtual camera data includes images from perspectives different from multiple perspectives; use the virtual camera data as input to the three-dimensional construction network and use the radar point cloud data as deep supervision of the three-dimensional construction network, and iteratively train the initial model to obtain the three-dimensional construction network.

[0197] In one possible implementation, the processing module 1403 is specifically used to: obtain the posture parameters of each frame of the multiple frames; project the multiple frames of images into the same space according to the posture of each frame of the images to obtain image point cloud data; project the image point cloud data based on an enhanced perspective to obtain virtual camera data, and the enhanced perspective is a perspective different from the multiple perspectives.

[0198] In a possible implementation, the processing module is specifically used to: fuse radar point cloud data and image point cloud data to obtain depth values ​​of multiple pixel points; and use the depth values ​​of the multiple pixel points as a three-dimensional construction network to perform depth supervision of three-dimensional construction.

[0199] In a possible implementation, any iterative training in the iterative training may include: obtaining at least one first sampling point in a first ray in a first perspective in a three-dimensional model output by the three-dimensional construction network obtained in the previous iteration; obtaining at least one second sampling point in a second ray in a second perspective in the three-dimensional model, where the at least one second sampling point is a point projected on the second ray by at least one first sampling point, and the first perspective and the second perspective are any different perspectives in the three-dimensional model; obtaining a difference between the first sampling point and the second sampling point; and updating the three-dimensional construction network according to the difference to obtain the three-dimensional construction network updated in the current iteration.

[0200] In a possible implementation, the processing module 1403 is specifically configured to: obtain a first depth estimation value of a first sampling point and a second depth estimation value of a second sampling point; and obtain a difference between the first depth estimation value and the second depth value.

[0201] In a possible implementation, the three-dimensional construction network includes: a coding module, a first geometry module, a first color module, a second geometry module, and a second color module;

[0202] The encoding module is used to encode the values ​​of the input pixel points and then input them into the first geometry module, the second geometry module and the second color module respectively;

[0203] The first geometry module and the second geometry module are used to perform geometry construction based on input data and output the geometry structure of the three-dimensional scene, and the output accuracy of the second geometry module is higher than the output accuracy of the first geometry module;

[0204] The first color module and the second color module are used to construct colors based on input data and output the color value of each pixel in the three-dimensional scene. The output accuracy of the second color module is higher than the output accuracy of the first color module. The input of the first color module includes the input of the first geometry module.

[0205] In a possible implementation, inputs of the first color module and the second color module further include depth supervision obtained based on radar point cloud data.

[0206] In a possible implementation, the device further includes:

[0207] Input module 1404, used to obtain input pose and camera parameters;

[0208] The output module 1405 is used to obtain an output image from the three-dimensional scene based on the input posture and camera parameters.

[0209] In a possible implementation, the three-dimensional model output by the three-dimensional construction network can be applied to intelligent driving of a vehicle.

[0210] See also Fig.15 , a schematic diagram of the structure of a three-dimensional model generation device provided in the present application, the three-dimensional model generation device may include:

[0211] Input module 1501, used to obtain input viewing angle information;

[0212] The output module 1502 is used to use the information of the input perspective as the input of the three-dimensional construction network and output the three-dimensional model. The three-dimensional construction network is obtained through multiple frame images, shooting parameters, radar point cloud data and the three-dimensional construction network. The shooting parameters are parameters used by the camera device when shooting multiple frame images. The multiple frame images include images shot from multiple perspectives. The radar point cloud data is data collected by the radar. The collection scene when the radar collects the radar point cloud data has an intersection with the shooting scene when the camera device shoots the multiple frame images.

[0213] In a possible implementation, the three-dimensional construction network includes: a coding module, a first geometry module, a first color module, a second geometry module, and a second color module;

[0214] The encoding module is used to encode the values ​​of the input pixel points and then input them into the first geometry module, the second geometry module and the second color module respectively;

[0215] The first geometry module and the second geometry module are used to perform geometry construction based on input data and output the geometry structure of the three-dimensional model, and the output accuracy of the second geometry module is higher than the output accuracy of the first geometry module;

[0216] The first color module and the second color module are used to construct colors based on the input data and output the color value of each pixel in the three-dimensional model. The output accuracy of the second color module is higher than the output accuracy of the first color module. The input of the first color module includes the input of the first geometry module.

[0217] In a possible implementation, the above-mentioned device may also include a training module 1503, which is used to train to obtain a three-dimensional construction network. The training module can be used to execute the process of the three-dimensional construction network training method provided in the present application. Please refer to the introduction of the above-mentioned training stage for details, which will not be repeated here.

[0218] See also Fig.16 , a structural schematic diagram of another three-dimensional construction network training device provided by the present application is described as follows.

[0219] The three-dimensional construction network training device may include a processor 1601 and a memory 1602. The processor 1601 and the memory 1602 are interconnected via a line. The memory 1602 stores program instructions and data.

[0220] The memory 1602 stores the aforementioned Figure 5-Figure 12 More specifically, the processor may also be a processor for processing images, such as a GPU or a CPU for processing images.

[0221] Processor 1601 is used to execute the above Figure 5-Figure 12 The method steps executed by the three-dimensional construction network training device shown in any embodiment.

[0222] Optionally, the three-dimensional construction network training device may further include a transceiver 1603 for receiving or sending data.

[0223] Optionally, the aforementioned Fig.16 The three-dimensional construction network training device shown in is a chip.

[0224] See also Fig.17 , a structural schematic diagram of another three-dimensional model generating device provided by the present application is described as follows.

[0225] The three-dimensional model generation device may include a processor 1701 and a memory 1702. The processor 1701 and the memory 1702 are interconnected via a line. The memory 1702 stores program instructions and data.

[0226] The memory 1702 stores the aforementioned Fig.13 More specifically, the processor may also be a processor for processing images, such as a GPU or a CPU for processing images.

[0227] Processor 1701 is used to execute the above Fig.13 The method steps are performed by the three-dimensional model generation device shown in any embodiment.

[0228] Optionally, the three-dimensional model generating device may further include a transceiver 1703 for receiving or sending data.

[0229] Optionally, the aforementioned Fig.17 The three-dimensional model generating device shown in is a chip.

[0230] The present application also provides a computer-readable storage medium in which a program is stored. When the program is run on a computer, the computer executes the above-mentioned Figure 5-Figure 13 The illustrated embodiments describe steps in a method.

[0231] The embodiment of the present application also provides a digital processing chip. The digital processing chip integrates a circuit and one or more interfaces for implementing the above-mentioned processor 1601 / 1701, or the function of the processor 1601 / 1701. When the digital processing chip integrates a memory, the digital processing chip can complete the method steps of any one or more of the above-mentioned embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory through a communication interface. The digital processing chip implements the actions in the above-mentioned embodiments according to the program code stored in the external memory.

[0232] The present application also provides a computer program product which, when executed on a computer, enables the computer to execute the above Figures 5 to 13 The illustrated embodiments describe steps in a method.

[0233] The 3D construction network training device or 3D model generation device provided in the embodiment of the present application can be a chip, and the chip can include: a processing unit and a communication unit, the processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit. The processing unit can execute the computer execution instructions stored in the storage unit to enable the chip in the server to execute the above Figures 5 to 13 Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0234] Specifically, the aforementioned processing unit or processor may be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0235] For example, see Fig.18 , Fig.18 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 180, NPU 180 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 1803, which is controlled by the controller 1804 to extract matrix data from the memory and perform multiplication operations.

[0236] In some implementations, the operation circuit 1803 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 1803 is a two-dimensional systolic array. The operation circuit 1803 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1803 is a general-purpose matrix processor.

[0237] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1802 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1801 and performs matrix operations with matrix B. The partial results or final results of the obtained matrix are stored in the accumulator 1808.

[0238] The unified memory 1806 is used to store input data and output data. The weight data is directly transferred to the weight memory 1802 through the direct memory access controller (DMAC) 1805. The input data is also transferred to the unified memory 1806 through the DMAC.

[0239] The bus interface unit (BIU) 1810 is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 1809.

[0240] The bus interface unit 1810 (BIU) is used for the instruction fetch memory 1809 to obtain instructions from the external memory, and is also used for the storage unit access controller 1805 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0241] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1806 or to transfer weight data to the weight memory 1802 or to transfer input data to the input memory 1801.

[0242] The vector calculation unit 1807 includes multiple operation processing units, which further process the output of the operation circuit when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, upsampling of feature planes, etc.

[0243] In some implementations, the vector calculation unit 1807 can store the processed output vector to the unified memory 1806. For example, the vector calculation unit 1807 can apply a linear function and / or a nonlinear function to the output of the operation circuit 1803, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1807 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1803, for example, for use in a subsequent layer in a neural network.

[0244] An instruction fetch buffer 1809 connected to the controller 1804 is used to store instructions used by the controller 1804;

[0245] Unified memory 1806, input memory 1801, weight memory 1802 and instruction fetch memory 1809 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0246] Among them, the operations of each layer in the recurrent neural network can be performed by the operation circuit 1803 or the vector calculation unit 1807.

[0247] The processor mentioned in any of the above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more processors for controlling the above Figure 5-Figure 13 The method is performed by a program of an integrated circuit.

[0248] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0249] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0250] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0251] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0252] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0253] Finally, it should be noted that the above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A three-dimensional construction network training method, characterized in that: include: Acquire multiple frames of images and shooting parameters used by a camera device when shooting the multiple frames of images, wherein the multiple frames of images include images shot from multiple perspectives; Acquire radar point cloud data, where a collection scene when the radar collects the radar point cloud data and a shooting scene when the camera device shoots the multiple frames of images have an intersection; Obtaining the pose parameters of each frame of the multiple frames of images; Projecting the multiple frames of images into the same space according to the position and posture of each frame of image to obtain image point cloud data; Projecting the image point cloud data based on an enhanced viewing angle to obtain virtual camera data, wherein the enhanced viewing angle is a viewing angle different from the multiple viewing angles, and the virtual camera data includes images of a viewing angle different from the multiple viewing angles; The virtual camera data is used as the input of the three-dimensional construction network, and the radar point cloud data is used as the depth supervision of the three-dimensional construction network. The initial model is iteratively trained to obtain the iteratively updated three-dimensional construction network, and the three-dimensional construction network is used to perform three-dimensional construction and output a three-dimensional model based on the input data.

2. The method according to claim 1, characterized in that The method of using the radar point cloud data as the depth supervision of the three-dimensional construction network includes: Fusing the radar point cloud data and the image point cloud data to obtain depth values ​​of multiple pixel points; The depth values ​​of the multiple pixel points are used as depth supervision for the three-dimensional construction of the three-dimensional construction network.

3. The method according to claim 1 or 2, characterized in that: Any one of the iterative trainings in the iterative training includes: Obtain at least one first sampling point in a first ray in a first perspective in a three-dimensional model output by a three-dimensional construction network obtained in a previous iteration; Acquire at least one second sampling point in a second ray in a second viewing angle in the three-dimensional model, where the at least one second sampling point is a point where the at least one first sampling point is projected on the second ray, and the first viewing angle and the second viewing angle are any different viewing angles in the three-dimensional model; Obtaining a difference between the first sampling point and the second sampling point; The three-dimensional construction network is updated according to the difference to obtain the iteratively updated three-dimensional construction network.

4. The method according to claim 3, characterized in that The obtaining the difference between the first sampling point and the second sampling point includes: Acquire a first depth estimation value of the first sampling point and a second depth estimation value of the second sampling point; A difference between the first depth estimation value and the second depth estimation value is obtained.

5. The method according to claim 1 or 2, characterized in that: The three-dimensional construction network includes: a coding module, a first geometry module, a first color module, a second geometry module and a second color module; The encoding module is used to encode the values ​​of the input pixel points and then input them into the first geometry module, the second geometry module and the second color module respectively; The first geometry module and the second geometry module are used to perform geometric construction based on input data and output a geometric structure of a three-dimensional scene, and the output accuracy of the second geometry module is higher than the output accuracy of the first geometry module; The first color module and the second color module are used to construct colors based on input data and output the color value of each pixel in the three-dimensional scene. The output accuracy of the second color module is higher than the output accuracy of the first color module. The input of the first color module includes the input of the first geometry module.

6. The method according to claim 5, characterized in that The inputs of the first color module and the second color module also include depth values, and the depth values ​​include values ​​obtained according to the radar point cloud data.

7. The method according to claim 1 or 2, characterized in that: The method further comprises: Get input pose and camera parameters; An output image is obtained from the three-dimensional scene based on the input pose and the camera parameters.

8. The method according to claim 1 or 2, characterized in that: The three-dimensional model output by the three-dimensional construction network is applied to the intelligent driving of the vehicle.

9. A three-dimensional model generation method, characterized in that: include: Get the input perspective information; The information of the input perspective is used as the input of the three-dimensional construction network, and a three-dimensional model is output, wherein the three-dimensional model is obtained through multiple frames of images, shooting parameters, radar point cloud data, and the three-dimensional construction network, wherein the shooting parameters are parameters used by a camera device when shooting the multiple frames of images, the multiple frames of images include images shot from multiple perspectives, the radar point cloud data are data collected by a radar, and the collection scene when the radar collects the radar point cloud data has an intersection with the collection scene when the camera device shoots the multiple frames of images; wherein the three-dimensional construction network includes: an encoding module, a first geometry module, a first color module, a second geometry module, and a second color module; The encoding module is used to encode the values ​​of the input pixel points and then input them into the first geometry module, the second geometry module and the second color module respectively; The first geometry module and the second geometry module are used to perform geometry construction based on input data and output a geometry structure of a three-dimensional model, and the output accuracy of the second geometry module is higher than the output accuracy of the first geometry module; The first color module and the second color module are used to construct colors based on input data and output the color value of each pixel in the three-dimensional model. The output accuracy of the second color module is higher than the output accuracy of the first color module. The input of the first color module includes the input of the first geometry module.

10. A three-dimensional construction network training device, characterized in that: include: An image acquisition module, used to acquire multiple frames of images and shooting parameters used by a camera device when shooting the multiple frames of images, wherein the multiple frames of images include images shot from multiple viewing angles; A radar acquisition module, used for acquiring radar point cloud data, wherein an acquisition scene when the radar acquires the radar point cloud data and a shooting scene when the camera device shoots the multiple frames of images have an intersection; Processing modules for: Obtaining the pose parameters of each frame of the multiple frames of images; Projecting the multiple frames of images into the same space according to the position and posture of each frame of image to obtain image point cloud data; Projecting the image point cloud data based on an enhanced viewing angle to obtain virtual camera data, wherein the enhanced viewing angle is a viewing angle different from the multiple viewing angles, and the virtual camera data includes images of a viewing angle different from the multiple viewing angles; The virtual camera data is used as the input of the three-dimensional construction network, and the radar point cloud data is used as the depth supervision of the three-dimensional construction network. The initial model is iteratively trained to obtain the iteratively updated three-dimensional construction network, and the three-dimensional construction network is used to perform three-dimensional construction and output a three-dimensional model based on the input data.

11. The device according to claim 10, characterized in that The processing module is specifically used for: Fusing the radar point cloud data and the image point cloud data to obtain depth values ​​of multiple pixel points; The depth values ​​of the multiple pixel points are used as depth supervision for the three-dimensional construction of the three-dimensional construction network.

12. The device according to claim 10 or 11, characterized in that Any one of the iterative trainings in the iterative training includes: Obtain at least one first sampling point in a first ray in a first perspective in a three-dimensional model output by a three-dimensional construction network obtained in a previous iteration; Acquire at least one second sampling point in a second ray in a second viewing angle in the three-dimensional model, where the at least one second sampling point is a point where the at least one first sampling point is projected on the second ray, and the first viewing angle and the second viewing angle are any different viewing angles in the three-dimensional model; Obtaining a difference between the first sampling point and the second sampling point; The three-dimensional construction network is updated according to the difference to obtain the three-dimensional construction network after the current iteration update.

13. The device according to claim 12, characterized in that The processing module is specifically used for: Acquire a first depth estimation value of the first sampling point and a second depth estimation value of the second sampling point; A difference between the first depth estimation value and the second depth estimation value is obtained.

14. The device according to claim 10 or 11, characterized in that The three-dimensional construction network includes: a coding module, a first geometry module, a first color module, a second geometry module and a second color module; The encoding module is used to encode the values ​​of the input pixel points and then input them into the first geometry module, the second geometry module and the second color module respectively; The first geometry module and the second geometry module are used to perform geometric construction based on input data and output a geometric structure of a three-dimensional scene, and the output accuracy of the second geometry module is higher than the output accuracy of the first geometry module; The first color module and the second color module are used to construct colors based on input data and output the color value of each pixel in the three-dimensional scene. The output accuracy of the second color module is higher than the output accuracy of the first color module. The input of the first color module includes the input of the first geometry module.

15. The device according to claim 14, characterized in that The inputs of the first color module and the second color module also include depth values, and the depth values ​​include values ​​obtained according to the radar point cloud data.

16. The device according to claim 10 or 11, characterized in that The device also includes: Input module, used to obtain input pose and camera parameters; An output module is used to obtain an output image from the three-dimensional scene based on the input posture and the camera parameters.

17. The device according to claim 10 or 11, characterized in that The three-dimensional model output by the three-dimensional construction network is applied to the intelligent driving of the vehicle.

18. A three-dimensional model generating device, characterized in that: include: An input module, used to obtain information of input viewing angle; An output module is used to use the information of the input viewing angle as an input of a three-dimensional construction network, and output a three-dimensional model, wherein the three-dimensional model is obtained through multiple frames of images, shooting parameters, radar point cloud data, and a three-dimensional construction network, wherein the shooting parameters are parameters used by a camera device when shooting the multiple frames of images, the multiple frames of images include images shot from multiple viewing angles, the radar point cloud data are data collected by a radar, and a collection scene when the radar collects the radar point cloud data has an intersection with a collection scene when the camera device shoots the multiple frames of images; the three-dimensional construction network includes: an encoding module, a first geometry module, a first color module, a second geometry module, and a second color module; The encoding module is used to encode the values ​​of the input pixel points and then input them into the first geometry module, the second geometry module and the second color module respectively; The first geometry module and the second geometry module are used to perform geometry construction based on input data and output a geometry structure of a three-dimensional model, and the output accuracy of the second geometry module is higher than the output accuracy of the first geometry module; The first color module and the second color module are used to construct colors based on input data and output the color value of each pixel in the three-dimensional model. The output accuracy of the second color module is higher than the output accuracy of the first color module. The input of the first color module includes the input of the first geometry module.

19. A three-dimensional construction network training device, characterized in that: The method comprises a processor, wherein the processor is coupled to a memory, wherein the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the method according to any one of claims 1 to 8 are implemented.

20. A three-dimensional model generating device, characterized in that: The method comprises a processor, wherein the processor is coupled to a memory, wherein the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the steps of the method according to claim 9 are implemented.

21. A computer-readable storage medium comprising a program, which, when executed by a processing unit, performs the steps of the method according to any one of claims 1 to 9.

22. A computer program product, characterized in that The computer program product comprises software code for executing the steps of the method according to any one of claims 1 to 9.

Images