Model training method, three-dimensional point cloud acquisition method and electronic device

Abstract

The present application provides a model training method, a three-dimensional point cloud acquisition method and an electronic device, the model training method comprising: acquiring a training data set, the training data set comprising: multiple viewing angle pictures of a sample object; training a color prediction network in a neural network model through multiple viewing angle pictures to obtain a trained color prediction network; training a depth prediction network in a neural network model through the trained color prediction network and the supervisory information determined by the multiple viewing angle pictures to obtain a trained neural network model. By using the trained color prediction network and the supervisory information determined by the multiple viewing angle pictures to train the depth prediction network in the neural network model, the trained neural network model can use the depth prediction network to predict image depth information, effectively improving the problem that there is no neural network model on the market that can predict image depth information.

Description

Technical Field

[0001] The present application relates to the technical field of deep learning and computer vision, and more specifically, to a model training method, a three-dimensional point cloud acquisition method, and an electronic device. Background Art

[0002] The current training of multilayer perceptron (MLP) models mostly relies on a single image and its corresponding posture. However, the network model trained by a single image and its corresponding posture can predict the color value of the image pixel, but cannot predict the depth of the image pixel. In the specific practice process, it was found that there is no neural network model on the market that can predict the depth information of the image. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a model training method, a three-dimensional point cloud acquisition method and an electronic device, which are used to improve the problem that there is no neural network model that can predict image depth information.

[0004] The embodiment of the present application provides a model training method, including: obtaining a training data set, the training data set including: multiple viewing angle pictures of a sample object; training a color prediction network in a neural network model through multiple viewing angle pictures to obtain a trained color prediction network; training a depth prediction network in a neural network model through the trained color prediction network and the supervisory information determined by the multiple viewing angle pictures to obtain a trained neural network model. In the implementation process of the above scheme, the color prediction network in the neural network model is trained through multiple viewing angle pictures, and the depth prediction network in the neural network model is trained using the trained color prediction network and the supervisory information determined by the multiple viewing angle pictures, so that the trained neural network model can not only use the color prediction network to predict the image color information, but also use the depth prediction network to predict the image depth information, which effectively improves the problem that there is no neural network model on the market that can predict the image depth information.

[0005] Optionally, in an embodiment of the present application, the color prediction network in the neural network model is trained through multiple viewing angle pictures, including: for each viewing angle picture in the multiple viewing angle pictures, using the color prediction network in the neural network model to predict the color value of the viewing angle picture to obtain the predicted color value; obtaining the true color value corresponding to the viewing angle picture, and calculating the color loss value between the predicted color value and the true color value; training the color prediction network in the neural network model according to the color loss value to obtain the trained color prediction network. In the implementation process of the above scheme, by training the color prediction network in the neural network model according to the color loss value, the color prediction network can directly predict the color value, effectively eliminating the cumulative error introduced by the subsequent volume rendering integration process, so that a faster rendering speed can be achieved without affecting the rendering quality.

[0006] Optionally, in an embodiment of the present application, the color value of the perspective image is predicted using a color prediction network in a neural network model, including: extracting a camera center and a sampling ray from the camera center to a sample object from the perspective image; inputting the camera center and the sampling ray into the color prediction network in the neural network model to obtain a predicted color value. In the implementation of the above scheme, since the ray points from the camera center to a specific pixel point, this method can help the network to more accurately locate the details in the scene and retain these details when predicting the color. Moreover, using the camera center and sampling rays as input can increase the robustness of the model to changes in perspective, so that the model can more accurately predict colors at different perspectives. Therefore, by using sampling rays and camera center information, the color prediction network can better understand the consistency of colors between different perspectives, thereby improving the accuracy of predicted colors.

[0007] Optionally, in an embodiment of the present application, the depth prediction network in the neural network model is trained by the trained color prediction network and the supervisory information determined by the multiple perspective images, including: determining one of the multiple perspective images as the main observation perspective image; determining another perspective image in the multiple perspective images as the auxiliary observation perspective image, the main observation perspective is different from the auxiliary observation perspective, and the main observation perspective image and the auxiliary observation perspective image include the same spatial point; the depth prediction network in the neural network model is trained by the trained color prediction network, the main observation perspective image and the supervisory information determined by the auxiliary observation perspective image. In the implementation process of the above scheme, the depth prediction network in the neural network model is trained by the trained color prediction network, the supervisory information determined by the main observation perspective image and the auxiliary observation perspective image, which effectively utilizes the spatial semantic information across multiple perspectives, so that the depth prediction network in the neural network model can have the ability to predict depth information (such as depth value).

[0008] Optionally, in an embodiment of the present application, the supervision information includes: a color loss value and a depth loss value; the depth prediction network in the neural network model is trained by the supervision information determined by the trained color prediction network, the main observation perspective picture, and the auxiliary observation perspective picture, including: determining the color loss value between the main observation perspective picture and the auxiliary observation perspective picture through the trained color prediction network; determining the depth loss value of the auxiliary observation perspective picture through the depth prediction network in the neural network model; training the depth prediction network in the neural network model according to the total loss value determined by the color loss value and the depth loss value to obtain the trained depth prediction network. In the implementation process of the above scheme, the depth prediction network in the neural network model is trained according to the color loss value determined by the trained color prediction network and the depth loss value determined by the depth prediction network, and the trained color prediction network is effectively used to convert the depth information solution problem between multiple perspective pictures into the optimization problem of the color loss value, and the color loss value between different perspective pictures is cleverly used to supervise the training process of the depth prediction network, thereby improving the depth estimation effect of the depth prediction network.

[0009] Optionally, in an embodiment of the present application, the color loss value between the main observation perspective image and the auxiliary observation perspective image is determined by the trained color prediction network, including: extracting the auxiliary perspective camera center from the auxiliary observation perspective image; obtaining the auxiliary perspective sampling ray from the auxiliary perspective camera center to the sample object, and inputting the auxiliary perspective camera center and the auxiliary perspective sampling ray into the trained color prediction network to obtain the auxiliary perspective color value output by the trained color prediction network; extracting the main perspective camera center from the main observation perspective image, and obtaining the main perspective sampling ray from the main perspective camera center to the sample object; obtaining the real main perspective color value corresponding to the main perspective sampling ray, and determining the color loss value according to the difference between the auxiliary perspective color value and the real main perspective color value. In the implementation process of the above scheme, the use of auxiliary perspective information can help the model better understand the scene structure, thereby improving the model's robustness to perspective changes, so that the model can more accurately predict the color at different perspectives. By extracting the camera center and sampling rays from pictures of different perspectives, it is easier to integrate this information into a multi-perspective fusion framework, and using the camera center and sampling rays of the auxiliary perspective, the color prediction network can better understand the consistency of colors between different perspectives, thereby improving the accuracy of predicted colors.

[0010] Optionally, in an embodiment of the present application, the depth loss value of the auxiliary observation perspective image is determined by the depth prediction network in the neural network model, including: inputting the main perspective camera center and the main perspective sampling ray into the depth prediction network in the neural network model, obtaining the main perspective point spatial coordinates output by the depth prediction network, the main perspective point spatial coordinates are the three-dimensional coordinates of the spatial point on the sample object reached by the main perspective sampling ray; determining the theoretical depth value of the auxiliary perspective according to the main perspective point spatial coordinates and the auxiliary perspective camera center; inputting the auxiliary perspective camera center and the auxiliary perspective sampling ray into the depth prediction network in the neural network model, obtaining the auxiliary perspective point spatial coordinates output by the depth prediction network, the auxiliary perspective point spatial coordinates are the three-dimensional coordinates of the spatial point on the sample object reached by the auxiliary perspective sampling ray; determining the depth prediction value of the auxiliary perspective according to the auxiliary perspective camera center and the auxiliary perspective point spatial coordinates; calculating the depth loss value according to the depth prediction value of the auxiliary perspective and the depth theoretical value of the auxiliary perspective. In the implementation process of the above scheme, by using the camera centers and sampling rays of the main view and the auxiliary view, the depth prediction network can better understand the depth relationship between different viewpoints. Moreover, by comparing the theoretical depth value of the auxiliary viewpoint with the depth prediction value, the model can learn the consistency of depth between different viewpoints, thereby improving the accuracy of depth prediction.

[0011] Optionally, in an embodiment of the present application, after obtaining the trained neural network model, it also includes: obtaining a preset perspective image, extracting the camera center from the preset perspective image, and obtaining a sampling ray corresponding to the camera center; inputting the camera center and the sampling ray corresponding to the camera center into the color prediction network in the trained neural network model to obtain the color value output by the color prediction network; rendering the preset perspective image by the color value output by the color prediction network to obtain a rendered perspective image. In the implementation process of the above scheme, compared with the traditional rendering process of displaying three-dimensional models, which requires vertex calculation one by one, the above-mentioned neural network model including the color prediction network is an implicit expression model. Therefore, when rendering the preset perspective image through the neural network model, rendering only performs a forward propagation calculation of the neural network once, which effectively reduces the amount of calculation for image rendering, thereby realizing a new perspective fast rendering function for multi-perspective images and improving the efficiency of rendering preset perspective images.

[0012] Optionally, in an embodiment of the present application, after obtaining the trained neural network model, it also includes: obtaining multiple perspective images and a three-dimensional point cloud of the target object; for each perspective image in the multiple perspective images, inputting the perspective image into a depth prediction network in the trained neural network model to obtain the perspective point spatial coordinates of the perspective image; determining the depth value of the perspective image based on the camera center of the perspective image and the perspective point spatial coordinates of the perspective image; and denoising the three-dimensional point cloud using the depth value of the perspective image to obtain a denoised three-dimensional point cloud.

[0013] The embodiment of the present application also provides a method for obtaining a three-dimensional point cloud, including: obtaining multiple perspective images and a three-dimensional point cloud of a target object; for each perspective image in the multiple perspective images, inputting the perspective image into the depth prediction network in the trained neural network model as described above to obtain the perspective point space coordinates of the perspective image; determining the depth value of the perspective image according to the camera center of the perspective image and the perspective point space coordinates of the perspective image; using the depth value of the perspective image to denoise the three-dimensional point cloud to obtain the denoised three-dimensional point cloud. In the implementation process of the above scheme, by using images of multiple perspectives, the depth prediction network can observe the scene from different angles, thereby improving the accuracy of depth prediction, and using depth information to perform point cloud denoising can more effectively identify and remove noise points, thereby improving the quality of the three-dimensional point cloud. Since the use of multi-perspective information improves the robustness of the model to perspective changes, the model can more accurately predict the depth at different perspectives, and the depth information obtained at different perspectives can be mutually verified, thereby improving the consistency and reliability of the prediction.

[0014] An embodiment of the present application also provides a model training device, including: a training data set acquisition module, used to acquire a training data set, the training data set including: multiple perspective images of a sample object; a color prediction network training module, used to train the color prediction network in the neural network model through multiple perspective images to obtain a trained color prediction network; a depth prediction network training module, used to train the depth prediction network in the neural network model through the trained color prediction network and supervision information determined by the multiple perspective images to obtain a trained neural network model.

[0015] Optionally, in an embodiment of the present application, the color prediction network training module includes: a color value prediction submodule, which is used to predict the color value of each perspective image in a plurality of perspective images using the color prediction network in the neural network model to obtain a predicted color value; obtain the true color value corresponding to the perspective image, and calculate the color loss value between the predicted color value and the true color value; and a color network training submodule, which is used to train the color prediction network in the neural network model according to the color loss value to obtain a trained color prediction network.

[0016] Optionally, in an embodiment of the present application, the color value prediction submodule includes: a picture parameter extraction unit, used to extract the camera center and the sampling ray from the camera center to the sample object from the perspective picture; a color value acquisition unit, used to input the camera center and the sampling ray into the color prediction network in the neural network model to obtain the predicted color value.

[0017] Optionally, in an embodiment of the present application, a depth prediction network training module includes: a main perspective picture determination submodule, used to determine one of a plurality of perspective pictures as a main observation perspective picture; an auxiliary perspective picture determination submodule, used to determine another perspective picture among a plurality of perspective pictures as an auxiliary observation perspective picture, the main observation perspective is different from the auxiliary observation perspective, and the main observation perspective picture and the auxiliary observation perspective picture include the same spatial point; a depth prediction network training submodule, used to train the depth prediction network in the neural network model through the supervision information determined by the trained color prediction network, the main observation perspective picture and the auxiliary observation perspective picture.

[0018] Optionally, in an embodiment of the present application, the supervision information includes: a color loss value and a depth loss value; a depth prediction network training submodule, including: a color loss value determination unit, used to determine the color loss value between the main observation perspective image and the auxiliary observation perspective image through the trained color prediction network; a depth loss value determination unit, used to determine the depth loss value of the auxiliary observation perspective image through the depth prediction network in the neural network model; a depth prediction network training unit, used to train the depth prediction network in the neural network model according to the total loss value determined by the color loss value and the depth loss value, to obtain the trained depth prediction network.

[0019] Optionally, in an embodiment of the present application, a color loss value determination unit includes: a camera center extraction subunit, used to extract the auxiliary perspective camera center from the auxiliary observation perspective picture; a center ray input subunit, used to obtain the auxiliary perspective sampling ray from the auxiliary perspective camera center to the sample object, and input the auxiliary perspective camera center and the auxiliary perspective sampling ray to the trained color prediction network to obtain the auxiliary perspective color value output by the trained color prediction network; a sampling ray acquisition subunit, used to extract the main perspective camera center from the main observation perspective picture, and obtain the main perspective sampling ray from the main perspective camera center to the sample object; a color loss value determination subunit, used to obtain the real main perspective color value corresponding to the main perspective sampling ray, and determine the color loss value according to the difference between the auxiliary perspective color value and the real main perspective color value.

[0020] Optionally, in an embodiment of the present application, a depth loss value determination unit includes: a main view coordinate acquisition subunit, which is used to input the main view camera center and the main view sampling ray into the depth prediction network in the neural network model to obtain the main view point spatial coordinates output by the depth prediction network, where the main view point spatial coordinates are the three-dimensional coordinates of the spatial point on the sample object where the main view sampling ray reaches; a depth theoretical value acquisition subunit, which is used to determine the depth theoretical value of the auxiliary view according to the main view point spatial coordinates and the auxiliary view camera center; an auxiliary view coordinate acquisition subunit, which is used to input the auxiliary view camera center and the auxiliary view sampling ray into the depth prediction network in the neural network model to obtain the auxiliary view point spatial coordinates output by the depth prediction network, where the auxiliary view point spatial coordinates are the three-dimensional coordinates of the spatial point on the sample object where the auxiliary view sampling ray reaches; a depth prediction value acquisition subunit, which is used to determine the depth prediction value of the auxiliary view according to the auxiliary view camera center and the auxiliary view point spatial coordinates; a depth loss value calculation subunit, which is used to calculate the depth loss value according to the depth prediction value of the auxiliary view and the depth theoretical value of the auxiliary view.

[0021] Optionally, in an embodiment of the present application, the model training device also includes: a camera parameter extraction submodule, used to obtain a preset perspective image, extract the camera center from the preset perspective image, and obtain the sampling ray corresponding to the camera center; a color value output submodule, used to input the camera center and the sampling ray corresponding to the camera center into the color prediction network in the trained neural network model to obtain the color value output by the color prediction network; a perspective image rendering submodule, used to render the preset perspective image through the color value output by the color prediction network to obtain a rendered perspective image.

[0022] Optionally, in an embodiment of the present application, the model training device also includes: a picture point cloud acquisition submodule, which is used to acquire multiple perspective pictures and a three-dimensional point cloud of the target object; a spatial coordinate acquisition submodule, which is used to input each perspective picture in the multiple perspective pictures into the depth prediction network in the trained neural network model to obtain the spatial coordinates of the perspective point of the perspective picture; a picture depth acquisition submodule, which is used to determine the depth value of the perspective picture based on the camera center of the perspective picture and the spatial coordinates of the perspective point of the perspective picture; and a three-dimensional point cloud denoising submodule, which is used to denoise the three-dimensional point cloud using the depth value of the perspective picture to obtain a denoised three-dimensional point cloud.

[0023] The embodiment of the present application also provides a three-dimensional point cloud acquisition device, including: an image point cloud acquisition module, used to acquire multiple perspective images and three-dimensional point clouds of a target object; a spatial coordinate acquisition module, used to input each perspective image of the multiple perspective images into a depth prediction network in a trained neural network model as described above, to obtain the spatial coordinates of the perspective points of the perspective image; an image depth acquisition module, used to determine the depth value of the perspective image based on the camera center of the perspective image and the spatial coordinates of the perspective points of the perspective image; and a three-dimensional point cloud denoising module, used to denoise the three-dimensional point cloud using the depth value of the perspective image, to obtain a denoised three-dimensional point cloud.

[0024] An embodiment of the present application further provides an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions execute the method described above when executed by the processor.

[0025] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is executed.

[0026] The embodiment of the present application also provides a computer program product, including: a computer program or a computer instruction, and the computer program or the computer instruction executes the method described above when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A flow chart of a model training method provided in an embodiment of the present application is shown;

[0029] Figure 2 A schematic diagram of a two-stage training process of a neural network model provided in an embodiment of the present application is shown;

[0030] Figure 3 A schematic diagram of the network structure of a color prediction network using an MLP network provided in an embodiment of the present application is shown;

[0031] Figure 4 A schematic diagram of the process of a three-dimensional point cloud acquisition method provided in an embodiment of the present application is shown;

[0032] Figure 5A schematic diagram of the flow chart of the model training device provided in an embodiment of the present application is shown;

[0033] Figure 6 A schematic diagram of the process flow of a three-dimensional point cloud acquisition device provided in an embodiment of the present application is shown;

[0034] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0035] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. It should be understood that the drawings in the embodiment of the present application only serve the purpose of explanation and description, and are not used to limit the protection scope of the embodiment of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in the embodiment of the present application shows the operation implemented according to some embodiments of the embodiment of the present application. It should be understood that the operation of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the embodiment of the present application, or remove one or more operations from the flowchart.

[0036] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments. The components of the embodiments of the present application described and shown in the drawings generally here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the embodiments of the present application claimed, but merely represents the selected embodiments of the embodiments of the present application.

[0037] It is understandable that the "first" and "second" in the embodiments of the present application are used to distinguish similar objects. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit the difference. In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of the associated objects, indicating that there may be three relationships, such as A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship. The term "multiple" refers to more than two (including two), and similarly, "multiple groups" refers to more than two groups (including two groups).

[0038] It should be noted that the model training method provided in the embodiment of the present application can be executed by an electronic device, where the electronic device refers to a device terminal or server with the function of executing a computer program, such as a smart phone, a personal computer, a tablet computer, a personal digital assistant, or a mobile Internet device. A server refers to a device that provides computing services through a network, such as an x86 server and a non-x86 server, and non-x86 servers include mainframes, minicomputers, and UNIX servers.

[0039] In the related art, multi-layer perceptron (MLP) models are usually used to process one-dimensional data, such as classification and regression tasks, and they are usually not as effective as convolutional neural networks (CNNs) when processing image data. When it comes to predicting the depth information of image pixels, there are some inherent challenges, especially when relying only on a single image and its posture. These challenges include: recovering depth information from a single image is an ill-posed problem, because objects of the same size appear to be the same size at different distances, and it is difficult to determine their true size or distance without other information. Therefore, the posture information of a single image (such as camera position and orientation) is not sufficient to provide sufficient constraints to recover depth information, so the network model trained by a single image and its corresponding posture can predict the color value of the image pixel, but cannot predict the depth of the image pixel point.

[0040] To overcome the above challenges, see Figure 1 The flowchart of the model training method provided by the embodiment of the present application is shown; the main idea adopted by the model training method is to provide supervision information by combining multi-view images and improving the network architecture. This improved network architecture allows the neural network model to include a color prediction network (which can be expressed as Color-Net) and a depth prediction network (which can be expressed as Depth-Net) at the same time, so that the neural network model can effectively use the multiple view images of the sample object and the supervision information determined by the Color-Net trained by it to train the Depth-Net that can predict the depth information. Since the supervision information determined by the Color-Net can effectively replace the manually annotated depth label, the above-mentioned model training method enables the neural network model to have the ability to predict depth information without the training of manually annotated depth labels. The specific implementation methods of the above-mentioned model training method may include:

[0041] Step S110: Acquire a training data set, where the training data set includes: images of sample objects from multiple viewing angles.

[0042] The above-mentioned sample object can be any target object, such as a toy excavator, etc., and the multiple-view images of the sample object contain spatial semantic information of multiple viewpoints. This cross-view spatial semantic information can be effectively used in the training process of the above-mentioned neural network model. The above-mentioned multiple-view images of the sample object can be obtained by photographing the sample object from multiple observation viewpoints by a camera, and the proximity of the multiple viewpoints can be less than a threshold, so that the color values ​​presented by the same spatial point in different viewpoints are exactly the same or very close (that is, the difference in the color values ​​presented by the same spatial point in different viewpoints is less than a preset threshold).

[0043] Step S120: training the color prediction network in the neural network model through multiple viewing angle images to obtain a trained color prediction network.

[0044] See also Figure 2 A schematic diagram of the two-stage training process of the neural network model provided by an embodiment of the present application is shown; it can be understood that the above-mentioned neural network model can be a two-stage network model, in the first stage above, the color prediction network (which can be represented as Color-Net) in the neural network model is trained through multiple perspective images, so as to learn the color value of the light in the perspective image (the color value of the light here represents the color value of the surface of the object), and in the second stage below, the depth prediction network (which can be represented as Depth-Net) in the neural network model is trained through the trained color prediction network and the supervisory information determined by the multiple perspective images, so as to learn the depth value of the sampled light.

[0045] The above-mentioned color prediction network refers to a model for predicting color values. In some embodiments, an MLP network model may be used, and in other embodiments, a spatial transformer network (Spatial Transformer Networks, STNs) model or a neural radiance field (NeRF) model may be used.

[0046] Step S130: Train the depth prediction network in the neural network model using the trained color prediction network and the supervision information determined by the multiple perspective images to obtain a trained neural network model.

[0047] Similarly, the above-mentioned depth prediction network refers to a model for predicting depth information (such as depth value). In some embodiments, an MLP network model may be used, and in other embodiments, a spatial transformer network (STNs) model or a neural radiant field (NeRF) model may be used.

[0048] It is understandable that compared to the traditional explicit three-dimensional model as the supervision information input, the rendering process of the displayed three-dimensional model requires vertex calculation one by one, which results in a very large amount of calculation and difficulty in handling complex scenes. The above-mentioned neural network model including the color prediction network and the depth prediction network is an implicit expression model. Therefore, when rendering the preset perspective image through the neural network model, rendering only performs a forward propagation calculation of the neural network once, which effectively reduces the amount of calculation for image rendering, thereby improving the efficiency of predicting depth information.

[0049] In the implementation process of the above scheme, the color prediction network in the neural network model is trained by using multiple view images, and the trained color prediction network and the supervision information determined by the multiple view images are used to train the depth prediction network in the neural network model, so that the trained neural network model can not only use the color prediction network to predict the image color information, but also use the depth prediction network to predict the image depth information, which effectively improves the problem that there is no neural network model on the market that can predict image depth information.

[0050] As an optional implementation of the above step S120, the above implementation of training the color prediction network in the neural network model through multiple viewing angle pictures may include:

[0051] Step S121: for each perspective picture among the multiple perspective pictures, use the color prediction network in the neural network model to predict the color value of the perspective picture to obtain a predicted color value.

[0052] The implementation of the above step S121 is relatively complicated, so the implementation of the step S121 will be introduced in detail below.

[0053] Step S122: Obtain the true color value corresponding to the viewing angle image, and calculate the color loss value between the predicted color value and the true color value.

[0054] The implementation method of the above step S122 is, for example: obtaining the real color value corresponding to the viewing angle image through an executable program, and using the formula Calculate the color loss value between the predicted color value and the true color value; where, Represents the color loss value between the predicted color value and the true color value. represents the predicted color value output by the color prediction network, Indicates the actual color value corresponding to the image at this viewing angle.

[0055] Step S123: training the color prediction network in the neural network model according to the color loss value to obtain a trained color prediction network.

[0056] An implementation example of the above step S123 is: using the color loss value to update the weight parameters of the color prediction network in the neural network model until the accuracy of the color prediction network in the neural network model converges or the number of training iterations exceeds a preset number, thereby obtaining a trained color prediction network.

[0057] In the implementation process of the above scheme, the color prediction network in the neural network model is trained according to the color loss value, so that the color prediction network can directly predict the color value, effectively eliminating the cumulative error introduced by the subsequent volume rendering integration process, thereby achieving faster rendering speed without affecting the rendering quality.

[0058] As an optional implementation of the above step S121, using the color prediction network in the neural network model to predict the color value of the viewing angle image includes:

[0059] Step S121a: extracting the camera center and the sampling ray from the camera center to the sample object from the perspective image.

[0060] The implementation method of the above step S121a is, for example: COLMAP algorithm software or other pose estimation algorithm software can be used to extract pose information from one of the multiple perspective images, where the pose information includes: camera parameters such as the camera center, internal parameters and external parameters corresponding to the perspective image. In addition, the above pose information and pixel coordinates can also be used to determine the sampling ray starting from the camera center to a pixel center of the sample object perspective image. The sampling ray here can be understood as a ray connecting the camera center and the pixel center, also known as a ray direction or a light direction. A preset number (for example, 128 or 256) of sampling points can be equidistantly arranged on the sampling ray. If the three-dimensional coordinates of the two endpoints of the sampling ray are known, the three-dimensional coordinates of each sampling point on the sampling ray can be calculated.

[0061] Step S121b: input the camera center and the sampling ray into the color prediction network in the neural network model to obtain the predicted color value.

[0062] See also Figure 3The network structure diagram of the color prediction network using the MLP network provided in the embodiment of the present application is shown; the implementation method of the above step S121b is, for example: multiple sampling points in the sampling ray are mapped to the high-dimensional space through nonlinear transformation, the mapped sampling ray is obtained, and the camera center and the mapped sampling ray are input into the color prediction network in the neural network model to obtain the predicted color value of the pixel center corresponding to the sampling ray in the perspective picture. Optionally, the color prediction network can adopt an MLP network, wherein the MLP network can be composed of 16 main layers, and each main layer is composed of a fully connected layer, a Normalization layer and a Dropout layer. Some residual connections are also added to the MLP network, which can effectively solve the gradient problem in the deep neural network and improve the training efficiency and expression ability of the network. It can be understood that the first few main layers of the MLP network are used to extract the posture features of the high-dimensional space from the mapped sampling rays, and the latter several main layers of the MLP network are used to output the predicted color value of the sampling ray according to the posture features of the high-dimensional space.

[0063] As an optional implementation of the above step S130, the implementation of training the depth prediction network in the neural network model using the trained color prediction network and the supervisory information determined by the multiple viewing angle images may include:

[0064] Step S131: determining one of the multiple viewing angle pictures as the main observation viewing angle picture.

[0065] It can be understood that, in a specific practice, any one of the multiple viewing angle pictures can be determined as the main observation viewing angle picture.

[0066] Step S132: Determine another perspective picture among the multiple perspective pictures as an auxiliary observation perspective picture, the main observation perspective is different from the auxiliary observation perspective, and the main observation perspective picture and the auxiliary observation perspective picture include the same spatial point.

[0067] The above-mentioned main observation perspective image and auxiliary observation perspective image include the same spatial point. The same spatial point here can be understood as the same three-dimensional coordinate point. Since the color value of the same three-dimensional coordinate point is the same in adjacent perspectives within a short period of time, the color value of the same three-dimensional coordinate point in the main observation perspective image and the auxiliary observation perspective image can be used to calculate the color loss value in the supervision information, which is used to guide the training process of the depth prediction network in the neural network model.

[0068] Step S133: training the depth prediction network in the neural network model through the trained color prediction network, the main observation perspective image, and the supervision information determined by the auxiliary observation perspective image.

[0069] In the implementation process of the above scheme, the depth prediction network in the neural network model is trained through the supervised information determined by the trained color prediction network, the main observation perspective image, and the auxiliary observation perspective image, which effectively utilizes the spatial semantic information across multiple perspectives, so that the depth prediction network in the neural network model can have the ability to predict depth information (such as depth value).

[0070] As an optional implementation of the above step S133, the above supervision information may include: a color loss value and a depth loss value; training the depth prediction network in the neural network model through the supervision information determined by the trained color prediction network, the main observation angle image and the auxiliary observation angle image, including:

[0071] Step S133a: Determine the color loss value between the main observation angle image and the auxiliary observation angle image through the trained color prediction network.

[0072] The calculation of the color loss value in the above step S133a is relatively complicated, so the calculation process of the color loss value will be explained in detail below.

[0073] Step S133b: Determine the depth loss value of the auxiliary observation perspective image through the depth prediction network in the neural network model.

[0074] The calculation of the depth loss value in the above step S133b is relatively complicated, so the calculation process of the depth loss value will be explained in detail below.

[0075] Step S133c: According to the total loss value determined by the color loss value and the depth loss value, the depth prediction network in the neural network model is trained to obtain a trained depth prediction network.

[0076] The implementation method of the above step S133c is, for example, to use the formula Calculate the total loss value determined by the color loss value and the depth loss value, or use the formula The total loss value determined by the color loss value and the depth loss value is calculated, and then the weight parameters of the depth prediction network in the neural network model are updated using the total loss value until the total loss value reaches a convergence state, and the trained depth prediction network can be obtained. represents the total loss value mentioned above, represents the color loss value mentioned above, Indicates the weight value of color loss. The weight value can be adjusted during the training process. For example, you can take 80% first, then 50%, and so on. Represents the depth loss value mentioned above.

[0077] In the implementation process of the above scheme, the depth prediction network in the neural network model is trained according to the color loss value determined by the trained color prediction network and the depth loss value determined by the depth prediction network. The trained color prediction network is effectively used to transform the problem of solving the depth information between multiple perspective images into an optimization problem of the color loss value. The color loss value between images of different perspectives is cleverly used to supervise the training of the depth prediction network, thereby improving the depth estimation effect of the depth prediction network.

[0078] As an optional implementation of the above step S133a, the above implementation of determining the color loss value between the main observation angle image and the auxiliary observation angle image through the trained color prediction network may include:

[0079] Step S133d: extracting the auxiliary viewing angle camera center from the auxiliary viewing angle picture.

[0080] The implementation of the above step S133d is similar to the implementation of the above step S121a, and therefore, it will not be repeated here.

[0081] Step S133e: Obtain the auxiliary view sampling ray from the center of the auxiliary view camera to the sample object.

[0082] The implementation of the above step S133e is similar to the implementation of the above step S121a, and therefore, it will not be repeated here.

[0083] Step S133f: input the auxiliary view camera center and the auxiliary view sampling ray into the trained color prediction network to obtain the auxiliary view color value output by the trained color prediction network.

[0084] Step S133g: extract the main viewing angle camera center from the main viewing angle picture, and obtain the main viewing angle sampling ray from the main viewing angle camera center to the sample object.

[0085] The implementation of the above step S133g is similar to the implementation of the above step S121a, and therefore, it will not be repeated here.

[0086] Step S133h: Obtain the real main-view color value corresponding to the main-view sampling ray, and determine the color loss value according to the difference between the auxiliary-view color value and the real main-view color value.

[0087] It can be understood that the true main-view color value corresponding to the above-mentioned main-view sampling ray is the true value (Ground Truth, GT), that is, the real color value, and the auxiliary-view color value is the color value predicted by the trained color prediction network (Color-Net).

[0088] The implementation method of the above step S133h is, for example: obtaining the real main view color value corresponding to the main view sampling ray through an executable program, and using the formula The difference between the auxiliary viewing angle color value and the real main viewing angle color value is calculated to obtain the color loss value; wherein, represents the color loss value mentioned above, Represents the auxiliary perspective color value output by the trained color prediction network, Indicates the real main view color value corresponding to the main view sampling ray. Indicates the difference between the auxiliary perspective color value and the true main perspective color value.

[0089] As an optional implementation of the above step S133b, the above implementation of determining the depth loss value of the auxiliary observation perspective image by the depth prediction network in the neural network model may include:

[0090] Step S133i: Input the main-view camera center and the main-view sampling ray into the depth prediction network in the neural network model to obtain the main-view point spatial coordinates output by the depth prediction network. The main-view point spatial coordinates are the three-dimensional coordinates of the spatial point on the sample object where the main-view sampling ray reaches.

[0091] Optionally, when the main view sampling ray is input into the depth prediction network in the neural network model, the multiple sampling points on the main view sampling ray can be first position-encoded to enrich the data features and obtain the encoded multiple sampling points, and then the encoded multiple sampling points are input into the depth prediction network in the neural network model.

[0092] It can be understood that the sampling point weights on the main viewing angle sampling ray output by the depth prediction network represent the probability distribution of the sampling points on the sampling ray, and the actual main viewing angle point space coordinates are on the sampling ray output by the depth prediction network. Therefore, the sampling point weights can be multiplied by the sampling point coordinates to obtain the main viewing angle point space coordinates output by the above-mentioned depth prediction network.

[0093] Step S133j: Determine the theoretical depth value of the auxiliary view according to the spatial coordinates of the main view point and the center of the auxiliary view camera.

[0094] It can be understood that the auxiliary view camera center mentioned above is also a three-dimensional space coordinate. Therefore, when the main view point space coordinate is subtracted from the auxiliary view camera center, a vector can be obtained. The direction of the vector is the predicted direction of the auxiliary view sampling ray, and the modulus value of the vector is equal to the theoretical depth value of the auxiliary view.

[0095] The implementation method of the above step S133j is, for example, to use the formula The spatial coordinates of the main viewpoint and the center of the auxiliary viewpoint camera are calculated to obtain the theoretical depth value of the auxiliary viewpoint; where: Indicates the theoretical depth value of the auxiliary perspective, Indicates the first of multiple sampling points on the main viewing angle sampling ray i The spatial three-dimensional coordinates of the sampling points, Represents the sampling point weights on the main viewing angle sampling ray output by the depth prediction network, represents the spatial coordinates of the main viewing point mentioned above, Represents the spatial coordinates of the auxiliary view camera center, that is, the starting point coordinates of the auxiliary view sampling ray.

[0096] From the above content, we can know that the theoretical depth value in the auxiliary observation perspective can be calculated according to the spatial coordinates of the main viewing point. In addition, the theoretical depth value can be converted into a weighted representation of the spatial coordinates between two ray sampling points, and then the auxiliary observation perspective is combined with the depth prediction network (Depth-Net) to obtain a depth prediction value of the auxiliary perspective. Finally, the depth loss between the theoretical depth value and the depth prediction value is calculated. This process will be described in detail below.

[0097] Step S133k: Input the auxiliary view camera center and the auxiliary view sampling ray into the depth prediction network in the neural network model to obtain the auxiliary view point spatial coordinates output by the depth prediction network. The auxiliary view point spatial coordinates are the three-dimensional coordinates of the spatial point on the sample object where the auxiliary view sampling ray reaches the sample object.

[0098] As above, similar to the explanation of the above step S133i, the sampling point weights on the sampling rays output by the depth prediction network, and the actual auxiliary viewpoint space coordinates are on the sampling rays output by the depth prediction network. Therefore, the sampling point weights can be multiplied by the sampling point coordinates to obtain the auxiliary viewpoint space coordinates output by the above-mentioned depth prediction network.

[0099] Step S1331: Determine the depth prediction value of the auxiliary view according to the auxiliary view camera center and the auxiliary view point space coordinates.

[0100] It is understandable that the implementation of the above step S1331 is similar to the implementation of the above step S133j, so it is not repeated here. It should be noted that the depth theoretical value of an auxiliary view and the auxiliary view point space coordinates output by the corresponding depth prediction network have been predicted above the main observation view. At this time, the auxiliary view point space coordinates are connected with the center of another auxiliary view camera to obtain the sampling ray of the auxiliary view and the depth prediction value corresponding to the sampling ray of the auxiliary view.

[0101] Step S133m: Calculate the depth loss value according to the depth prediction value of the auxiliary view and the depth theoretical value of the auxiliary view.

[0102] The implementation method of the above step S133m is, for example, to use the formula The depth prediction value of the auxiliary view and the depth theoretical value of the auxiliary view are calculated to obtain the above-mentioned depth loss value; wherein, represents the depth loss value, is a probability distribution of the depth prediction value of the auxiliary perspective, each element Indicates that it belongs to i The probability of a class (i.e., the class represented by the weighted spatial coordinates between two ray sampling points), is the pseudo label of the theoretical depth value of the auxiliary perspective, and C is the dimension of the pseudo label.

[0103] As an optional implementation of the above step S130, after obtaining the trained neural network model, it also includes:

[0104] Step S134: obtaining a preset viewing angle image, extracting a camera center from the preset viewing angle image, and obtaining a sampling ray from the camera center to the sample object.

[0105] The implementation of the above step S134 is similar to the implementation of the above step S121a, and therefore, it will not be repeated here.

[0106] Step S135: input the camera center and the sampling ray corresponding to the camera center into the color prediction network in the trained neural network model to obtain the color value output by the color prediction network.

[0107] An implementation example of the above-mentioned step S135 is as follows: in the rendering inference stage, the color prediction network and the depth prediction network in the neural network model can be loaded with the optimal weight parameters obtained by the above model training method, and then the camera center of the viewing angle to be rendered and the sampling ray corresponding to the camera center are input into the color prediction network in the trained neural network model to obtain the color value output by the color prediction network.

[0108] Step S136: Rendering the preset viewing angle image by using the color value output by the color prediction network to obtain a rendered viewing angle image.

[0109] An implementation example of the above-mentioned step S136 is as follows: It can be understood that, for each camera center among multiple camera centers that need to render a viewing angle, after the camera center and the sampling ray corresponding to the camera center are input into the color prediction network in the trained neural network model, the above-mentioned rendered viewing angle image can be obtained.

[0110] In the implementation process of the above scheme, compared with the traditional rendering process of displaying three-dimensional models that requires vertex calculation one by one, the above-mentioned neural network model including the color prediction network is an implicit expression model. Therefore, when rendering the preset perspective image through the neural network model, rendering once only performs a forward propagation calculation of the neural network, which effectively reduces the amount of calculation for image rendering, thereby realizing a new perspective fast rendering function for multi-perspective images and improving the efficiency of rendering preset perspective images.

[0111] As an optional implementation of the above method, after obtaining the trained neural network model, it also includes:

[0112] Step S140: Acquire multiple perspective images and three-dimensional point clouds of the target object.

[0113] The implementation method of the above-mentioned step S140 is, for example: photographing the target object from multiple observation angles by a camera to obtain multiple-view images of the target object, and scanning the target object from multiple observation angles by a scanning device such as a radar to obtain a three-dimensional point cloud of the target object.

[0114] Step S150: for each perspective picture among the multiple perspective pictures, the perspective picture is input into the depth prediction network in the trained neural network model to obtain the spatial coordinates of the perspective point of the perspective picture.

[0115] An implementation example of the above step S150 is as follows: for each perspective picture among multiple perspective pictures, the camera center and sampling ray of the perspective picture are obtained, the camera center and sampling ray of the perspective picture are input into the depth prediction network in the trained neural network model, and the spatial coordinates of the perspective point of the perspective picture are obtained.

[0116] Step S160: determining the depth value of the perspective image according to the camera center of the perspective image and the spatial coordinates of the perspective point of the perspective image.

[0117] The implementation of the above step S160 is similar to the implementation of the above step S133j, and therefore, it will not be repeated here.

[0118] Step S170: De-noising the three-dimensional point cloud using the depth value of the viewing angle image to obtain a denoised three-dimensional point cloud.

[0119] An implementation example of the above step S170 is as follows: for each perspective image in different perspective images, by combining point cloud sampling technology and iterative closest point (ICP) point cloud registration and fusion technology, the depth value of the perspective image is used to denoise the three-dimensional point cloud, thereby improving the quality and efficiency of point cloud fusion and obtaining a denoised three-dimensional point cloud. Among them, point cloud sampling technology refers to the process of selecting a part of points from the original point cloud, the purpose of which is to reduce the number of point clouds, thereby reducing the computational complexity of subsequent processing. The iterative closest point (ICP) algorithm is a widely used point cloud registration method, which aims to minimize the distance between two point clouds, and is usually used to fuse point cloud data from multiple different perspectives into a unified global point cloud.

[0120] In the implementation process of the above scheme, by using pictures from multiple perspectives, the depth prediction network can observe the scene from different angles, thereby improving the accuracy of depth prediction. In addition, point cloud denoising processing using depth information can more effectively identify and remove noise points, thereby improving the quality of the three-dimensional point cloud. Since the use of multi-perspective information improves the robustness of the model to perspective changes, the model can more accurately predict depth at different perspectives, and the depth information obtained at different perspectives can be verified with each other, thereby improving the consistency and reliability of the prediction.

[0121] See also Figure 4 The flowchart of the three-dimensional point cloud acquisition method provided in the embodiment of the present application is shown; the embodiment of the present application provides a three-dimensional point cloud acquisition method, including:

[0122] Step S210: Acquire multiple perspective images and three-dimensional point clouds of the target object.

[0123] Step S220: For each perspective picture among the multiple perspective pictures, the perspective picture is input into the depth prediction network in the trained neural network model as described above to obtain the spatial coordinates of the perspective point of the perspective picture.

[0124] The implementation of the above steps S210 to S220 is similar to the implementation of the above steps S140 to S150, and therefore, they are not described in detail here.

[0125] Step S230: determining the depth value of the perspective image according to the camera center of the perspective image and the spatial coordinates of the perspective point of the perspective image.

[0126] The implementation of the above step S230 is similar to the implementation of the above step S133j, and therefore, it will not be repeated here.

[0127] Step S240: De-noising the three-dimensional point cloud using the depth value of the viewing angle image to obtain a denoised three-dimensional point cloud.

[0128] The implementation method of the above step S240 is similar to the implementation method of the above step S170, so it will not be repeated here. In the implementation process of the above scheme, by using pictures from multiple perspectives, the depth prediction network can observe the scene from different angles, thereby improving the accuracy of depth prediction, and using depth information for point cloud denoising can more effectively identify and remove noise points, thereby improving the quality of the three-dimensional point cloud. Since the use of multi-perspective information improves the robustness of the model to changes in perspective, the model can more accurately predict depth at different perspectives, and the depth information obtained at different perspectives can be verified with each other, thereby improving the consistency and reliability of the prediction.

[0129] See also Figure 5 The flowchart of the model training device provided in the embodiment of the present application is shown; the embodiment of the present application provides a model training device 300, including:

[0130] The training data set acquisition module 310 is used to acquire a training data set, where the training data set includes: multiple viewing angle images of a sample object.

[0131] The color prediction network training module 320 is used to train the color prediction network in the neural network model through multiple viewing angle images to obtain a trained color prediction network.

[0132] The depth prediction network training module 330 is used to train the depth prediction network in the neural network model through the trained color prediction network and the supervision information determined by the multiple perspective images to obtain the trained neural network model.

[0133] As an optional implementation of the above device, the color prediction network training module includes:

[0134] The color value prediction submodule is used to predict the color value of each perspective image in multiple perspective images using the color prediction network in the neural network model to obtain a predicted color value; obtain the true color value corresponding to the perspective image, and calculate the color loss value between the predicted color value and the true color value.

[0135] The color network training submodule is used to train the color prediction network in the neural network model according to the color loss value to obtain the trained color prediction network.

[0136] As an optional implementation of the above device, the color value prediction submodule includes:

[0137] A picture parameter extraction unit is used to extract a camera center and a sampling ray from the camera center to a sample object from the view picture.

[0138] The color value obtaining unit is used to input the camera center and the sampling ray into the color prediction network in the neural network model to obtain the predicted color value.

[0139] As an optional implementation of the above device, the depth prediction network training module includes:

[0140] The main viewing angle picture determination submodule is used to determine one of the multiple viewing angle pictures as the main observation viewing angle picture.

[0141] The auxiliary viewing angle picture determination submodule is used to determine another viewing angle picture among multiple viewing angle pictures as an auxiliary viewing angle picture, the main viewing angle is different from the auxiliary viewing angle, and the main viewing angle picture and the auxiliary viewing angle picture include the same spatial point.

[0142] The depth prediction network training submodule is used to train the depth prediction network in the neural network model through the supervised information determined by the trained color prediction network, the main observation perspective image and the auxiliary observation perspective image.

[0143] As an optional implementation of the above device, the supervision information includes: a color loss value and a depth loss value; the depth prediction network training submodule includes:

[0144] The color loss value determination unit is used to determine the color loss value between the main observation perspective image and the auxiliary observation perspective image through a trained color prediction network.

[0145] The depth loss value determination unit is used to determine the depth loss value of the auxiliary observation perspective image through the depth prediction network in the neural network model.

[0146] The depth prediction network training unit is used to train the depth prediction network in the neural network model according to the total loss value determined by the color loss value and the depth loss value to obtain the trained depth prediction network.

[0147] As an optional implementation manner of the above device, the color loss value determining unit includes:

[0148] The camera center extraction subunit is used to extract the auxiliary view camera center from the auxiliary observation view picture.

[0149] The center ray input subunit is used to obtain the auxiliary view sampling ray from the auxiliary view camera center to the sample object, and input the auxiliary view camera center and the auxiliary view sampling ray into the trained color prediction network to obtain the auxiliary view color value output by the trained color prediction network.

[0150] The sampling ray acquisition subunit is used to extract the main viewing angle camera center from the main viewing angle picture, and obtain the main viewing angle sampling ray starting from the main viewing angle camera center to the sample object.

[0151] The color loss value determination subunit is used to obtain the real main-view color value corresponding to the main-view sampling ray, and determine the color loss value according to the difference between the auxiliary-view color value and the real main-view color value.

[0152] As an optional implementation manner of the above device, the depth loss value determining unit includes:

[0153] The main view coordinate acquisition subunit is used to input the main view camera center and the main view sampling ray into the depth prediction network in the neural network model to obtain the main view point spatial coordinates output by the depth prediction network. The main view point spatial coordinates are the three-dimensional coordinates of the spatial point on the sample object where the main view sampling ray reaches.

[0154] The depth theoretical value obtaining subunit is used to determine the depth theoretical value of the auxiliary view according to the spatial coordinates of the main view point and the center of the auxiliary view camera.

[0155] The auxiliary view coordinate acquisition subunit is used to input the auxiliary view camera center and the auxiliary view sampling ray into the depth prediction network in the neural network model to obtain the auxiliary view point spatial coordinates output by the depth prediction network. The auxiliary view point spatial coordinates are the three-dimensional coordinates of the spatial point on the sample object where the auxiliary view sampling ray reaches the sample object.

[0156] The depth prediction value obtaining subunit is used to determine the depth prediction value of the auxiliary view according to the auxiliary view camera center and the auxiliary view point space coordinates.

[0157] The depth loss value calculation subunit is used to calculate the depth loss value according to the depth prediction value of the auxiliary perspective and the depth theoretical value of the auxiliary perspective.

[0158] As an optional implementation of the above device, the model training device further includes:

[0159] The camera parameter extraction submodule is used to obtain a preset viewing angle image, extract the camera center from the preset viewing angle image, and obtain a sampling ray corresponding to the camera center.

[0160] The color value output submodule is used to input the camera center and the sampling ray corresponding to the camera center into the color prediction network in the trained neural network model to obtain the color value output by the color prediction network.

[0161] The perspective image rendering submodule is used to render the preset perspective image through the color value output by the color prediction network to obtain the rendered perspective image.

[0162] As an optional implementation of the above device, the model training device further includes:

[0163] The image point cloud acquisition submodule is used to obtain multiple perspective images and three-dimensional point clouds of the target object.

[0164] The spatial coordinate acquisition submodule is used to input each perspective image of multiple perspective images into the depth prediction network in the trained neural network model to obtain the spatial coordinates of the perspective point of the perspective image.

[0165] The picture depth obtaining submodule is used to determine the depth value of the perspective picture according to the camera center of the perspective picture and the spatial coordinates of the perspective point of the perspective picture.

[0166] The 3D point cloud denoising submodule is used to perform denoising on the 3D point cloud using the depth value of the perspective image to obtain a denoised 3D point cloud.

[0167] See also Figure 6 The flowchart of the three-dimensional point cloud acquisition device provided in the embodiment of the present application is shown; the embodiment of the present application provides a three-dimensional point cloud acquisition device 400, including:

[0168] The image point cloud acquisition module 410 is used to acquire multiple viewpoint images and three-dimensional point clouds of the target object.

[0169] The spatial coordinate acquisition module 420 is used to input each of the multiple perspective images into the depth prediction network in the trained neural network model as described above to obtain the spatial coordinates of the perspective point of the perspective image.

[0170] The image depth obtaining module 430 is used to determine the depth value of the perspective image according to the camera center of the perspective image and the spatial coordinates of the perspective point of the perspective image.

[0171] The three-dimensional point cloud denoising module 440 is used to perform denoising on the three-dimensional point cloud using the depth value of the viewing angle image to obtain a denoised three-dimensional point cloud.

[0172] It should be understood that the device corresponds to the above-mentioned model training method embodiment and can perform the various steps involved in the above-mentioned method embodiment. The specific functions of the device can be found in the description above, and the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system (OS) of the device.

[0173] See also Figure 7 An electronic device 500 provided in an embodiment of the present application includes: a processor 510 and a memory 520, wherein the memory 520 stores machine-readable instructions executable by the processor 510, and when the machine-readable instructions are executed by the processor 510, the above method is executed.

[0174] The embodiment of the present application also provides a computer-readable storage medium 530, on which a computer program is stored, and the computer program is executed by the processor 510 to execute the above method. The computer-readable storage medium 530 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, disk or optical disk.

[0175] The embodiment of the present application also provides a computer program product, including: a computer program or a computer instruction, and the computer program or the computer instruction executes the method described above when executed by a processor.

[0176] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0177] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed device and method can also be implemented in other ways. The device embodiments described above are only schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module, a program segment or a code, and a part of a module, a program segment or a code includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also be different from the order of occurrence marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which is mainly determined by the functions involved.

[0178] In addition, each functional module of each embodiment in the embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. In addition, in the description of this specification, the description of reference terms "one embodiment", "some embodiments", "example", "specific example", "some examples", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present application. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples without contradiction.

[0179] The above description is only an optional implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims

1. A model training method, characterized in that: include: Acquire a training data set, the training data set comprising: multiple viewing angle images of a sample object; Training a color prediction network in a neural network model through the multiple perspective images to obtain a trained color prediction network; Training the depth prediction network in the neural network model using the trained color prediction network and the supervision information determined by the multiple perspective images to obtain a trained neural network model; The method of training the color prediction network in the neural network model through the multiple viewing angle pictures includes: for each viewing angle picture in the multiple viewing angle pictures, extracting the camera center and the sampling ray from the camera center to the sample object from the viewing angle picture; inputting the camera center and the sampling ray into the color prediction network in the neural network model to obtain a predicted color value; obtaining the real color value corresponding to the viewing angle picture, and calculating the color loss value between the predicted color value and the real color value; training the color prediction network in the neural network model according to the color loss value to obtain the trained color prediction network; The supervisory information includes: a color loss value and a depth loss value; the supervisory information determined by the trained color prediction network and the multiple viewing angle pictures is used to train the depth prediction network in the neural network model, including: determining one of the multiple viewing angle pictures as a main observation viewing angle picture; determining another viewing angle picture in the multiple viewing angle pictures as an auxiliary observation viewing angle picture, the main observation viewing angle is different from the auxiliary observation viewing angle, and the main observation viewing angle picture and the auxiliary observation viewing angle picture include the same spatial point; determining the color loss value between the main observation viewing angle picture and the auxiliary observation viewing angle picture through the trained color prediction network; determining the depth loss value of the auxiliary observation viewing angle picture through the depth prediction network in the neural network model; training the depth prediction network in the neural network model according to the total loss value determined by the color loss value and the depth loss value to obtain the trained depth prediction network; The color loss value between the main observation perspective image and the auxiliary observation perspective image is determined by the trained color prediction network, including: extracting the auxiliary perspective camera center from the auxiliary observation perspective image; obtaining the auxiliary perspective sampling ray from the auxiliary perspective camera center to the sample object, and inputting the auxiliary perspective camera center and the auxiliary perspective sampling ray into the trained color prediction network to obtain the auxiliary perspective color value output by the trained color prediction network; extracting the main perspective camera center from the main observation perspective image, and obtaining the main perspective sampling ray from the main perspective camera center to the sample object; obtaining the real main perspective color value corresponding to the main perspective sampling ray, and determining the color loss value according to the difference between the auxiliary perspective color value and the real main perspective color value; The method of determining the depth loss value of the auxiliary observation perspective image through the depth prediction network in the neural network model includes: inputting the main perspective camera center and the main perspective sampling ray into the depth prediction network in the neural network model to obtain the main perspective point spatial coordinates output by the depth prediction network, wherein the main perspective point spatial coordinates are the three-dimensional coordinates of the spatial point on the sample object reached by the main perspective sampling ray; determining the auxiliary perspective theoretical depth value according to the main perspective point spatial coordinates and the auxiliary perspective camera center; inputting the auxiliary perspective camera center and the auxiliary perspective sampling ray into the depth prediction network in the neural network model to obtain the auxiliary perspective point spatial coordinates output by the depth prediction network, wherein the auxiliary perspective point spatial coordinates are the three-dimensional coordinates of the spatial point on the sample object reached by the auxiliary perspective sampling ray; determining the auxiliary perspective depth prediction value according to the auxiliary perspective camera center and the auxiliary perspective point spatial coordinates; and calculating the depth loss value according to the auxiliary perspective depth prediction value and the auxiliary perspective depth theoretical value.

2. The method according to claim 1, characterized in that After obtaining the trained neural network model, the method further includes: Acquire a preset viewing angle image, extract a camera center from the preset viewing angle image, and acquire a sampling ray corresponding to the camera center; Inputting the camera center and the sampling ray corresponding to the camera center into the color prediction network in the trained neural network model to obtain the color value output by the color prediction network; The preset viewing angle image is rendered using the color value output by the color prediction network to obtain a rendered viewing angle image.

3. The method according to claim 1, characterized in that After obtaining the trained neural network model, the method further includes: Obtain multiple perspective images and 3D point clouds of the target object; For each perspective picture among the multiple perspective pictures, input the perspective picture into the depth prediction network in the trained neural network model to obtain the spatial coordinates of the perspective point of the perspective picture; Determine the depth value of the perspective image according to the camera center of the perspective image and the spatial coordinates of the perspective point of the perspective image; The three-dimensional point cloud is denoised using the depth value of the perspective image to obtain a denoised three-dimensional point cloud.

4. A three-dimensional point cloud acquisition method, characterized in that: include: Obtain multiple perspective images and 3D point clouds of the target object; For each of the multiple perspective images, input the perspective image into a depth prediction network in a neural network model trained by the model training method described in any one of claims 1 to 3 to obtain the spatial coordinates of the perspective point of the perspective image; Determine the depth value of the perspective image according to the camera center of the perspective image and the spatial coordinates of the perspective point of the perspective image; The three-dimensional point cloud is denoised using the depth value of the perspective image to obtain a denoised three-dimensional point cloud.

5. A model training device, characterized in that: include: A training data set acquisition module is used to acquire a training data set, wherein the training data set includes: multiple viewing angle images of a sample object; A color prediction network training module, used to train the color prediction network in the neural network model through the multiple perspective images to obtain a trained color prediction network; A depth prediction network training module, used to train the depth prediction network in the neural network model through the trained color prediction network and the supervision information determined by the multiple perspective images to obtain a trained neural network model; The method of training the color prediction network in the neural network model through the multiple viewing angle pictures includes: for each viewing angle picture in the multiple viewing angle pictures, extracting the camera center and the sampling ray from the camera center to the sample object from the viewing angle picture; inputting the camera center and the sampling ray into the color prediction network in the neural network model to obtain a predicted color value; obtaining the real color value corresponding to the viewing angle picture, and calculating the color loss value between the predicted color value and the real color value; training the color prediction network in the neural network model according to the color loss value to obtain the trained color prediction network; The supervisory information includes: a color loss value and a depth loss value; the supervisory information determined by the trained color prediction network and the multiple viewing angle pictures is used to train the depth prediction network in the neural network model, including: determining one of the multiple viewing angle pictures as a main observation viewing angle picture; determining another viewing angle picture in the multiple viewing angle pictures as an auxiliary observation viewing angle picture, the main observation viewing angle is different from the auxiliary observation viewing angle, and the main observation viewing angle picture and the auxiliary observation viewing angle picture include the same spatial point; determining the color loss value between the main observation viewing angle picture and the auxiliary observation viewing angle picture through the trained color prediction network; determining the depth loss value of the auxiliary observation viewing angle picture through the depth prediction network in the neural network model; training the depth prediction network in the neural network model according to the total loss value determined by the color loss value and the depth loss value to obtain the trained depth prediction network; The color loss value between the main observation perspective image and the auxiliary observation perspective image is determined by the trained color prediction network, including: extracting the auxiliary perspective camera center from the auxiliary observation perspective image; obtaining the auxiliary perspective sampling ray from the auxiliary perspective camera center to the sample object, and inputting the auxiliary perspective camera center and the auxiliary perspective sampling ray into the trained color prediction network to obtain the auxiliary perspective color value output by the trained color prediction network; extracting the main perspective camera center from the main observation perspective image, and obtaining the main perspective sampling ray from the main perspective camera center to the sample object; obtaining the real main perspective color value corresponding to the main perspective sampling ray, and determining the color loss value according to the difference between the auxiliary perspective color value and the real main perspective color value; The method of determining the depth loss value of the auxiliary observation perspective image through the depth prediction network in the neural network model includes: inputting the main perspective camera center and the main perspective sampling ray into the depth prediction network in the neural network model to obtain the main perspective point spatial coordinates output by the depth prediction network, wherein the main perspective point spatial coordinates are the three-dimensional coordinates of the spatial point on the sample object reached by the main perspective sampling ray; determining the auxiliary perspective theoretical depth value according to the main perspective point spatial coordinates and the auxiliary perspective camera center; inputting the auxiliary perspective camera center and the auxiliary perspective sampling ray into the depth prediction network in the neural network model to obtain the auxiliary perspective point spatial coordinates output by the depth prediction network, wherein the auxiliary perspective point spatial coordinates are the three-dimensional coordinates of the spatial point on the sample object reached by the auxiliary perspective sampling ray; determining the auxiliary perspective depth prediction value according to the auxiliary perspective camera center and the auxiliary perspective point spatial coordinates; and calculating the depth loss value according to the auxiliary perspective depth prediction value and the auxiliary perspective depth theoretical value.

6. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions are executed by the processor to perform any method according to claim 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is executed.

8. A computer program product, characterized in that include: A computer program or a computer instruction, wherein when the computer program or the computer instruction is executed by a processor, the method according to any one of claims 1 to 4 is executed.

Images