Sparse visual angle 3D model construction method and device, equipment and storage medium

Abstract

The invention relates to the technical field of 3D model construction, and discloses a sparse view angle 3D model construction method and device, equipment and a storage medium, and the method comprises the steps: obtaining sparse view angle images of different view angles of a target object; performing 3D modeling on the target object by using the sparse view angle image to obtain a corresponding 3D point cloud and a pose of each image; constructing a neural radiation field model used for predicting an SDF field and an RGB field, and performing joint supervision training on the neural radiation field model by adopting a volume rendering mode based on the sparse view angle image and the pose and the 3D point cloud thereof to obtain a trained neural radiation field model; and inputting a preset resolution into the trained neural radiation field model to obtain a corresponding SDF value and a corresponding RGB value, and performing explicit 3D of an implicit scene by using a preset algorithm to obtain a corresponding scene-level or individual-level 3D virtual simulation model, and the method provided by the invention can realize accurate and efficient 3D model reconstruction only by using a sparse view angle image.

Description

Technical Field

[0001] The present invention relates to the technical field of 3D model construction, and in particular to a sparse perspective 3D model construction method, device, equipment and storage medium. Background Art

[0002] 3D virtual simulation technology based on 2D images has become a new means of testing and verifying unmanned systems. Simulation model and scene construction technology is the key to large sample, extreme boundary, and small probability sample testing and verification in the virtual simulation test link, and it is also a challenging problem. At present, the method based on NeRF (Neural Radiance Fields) has become the current mainstream solution due to its high precision and other outstanding advantages. NeRF's Best Paper at the 2020 ECCV conference has pushed the new perspective synthesis task to a new height. New perspective synthesis refers to the observation of a scene under certain perspectives. The observation under the new perspective is synthesized by an algorithm to generate a three-dimensional model of the object and the scene. As a simulation model and scene construction technology, NeRF simulation model construction technology can be applied to the field of virtual testing and verification of unmanned systems.

[0003] The original NeRF algorithm has three main disadvantages: (1) the model has a large number of parameters, and the training and rendering speeds are slow; (2) the model needs to rely on multi-view images (usually about 100 views) for training; (3) NeRF uses density fields to represent 3D space, which is beneficial to the quality of new view synthesis, but will cause the surface of 3D reconstructed objects to be rough and blurred. Currently, in order to solve the problem of NeRF rendering speed, Plenoxels, instant-ngp and other works have greatly accelerated the training process. Regarding the number of training images: the original version of NeRF requires about 100 images for training, while methods such as PixelNeRF, IBRNet, and MVSNeRF have greatly reduced the number of training images.

[0004] In response to the fuzzy edge of 3D reconstruction, the academic community has proposed SDF-based methods, such as VolSDF, which uses SDF field instead of density field to realize the display modeling of the object surface, thereby achieving more accurate object surface reconstruction. However, these methods solve certain single problems in a targeted manner, but cannot solve the above problems at the same time. Summary of the invention

[0005] In view of this, the present invention provides a sparse perspective 3D model construction method, device, equipment and storage medium to solve the problem of defects in the existing NeRF algorithm mentioned in the background technology.

[0006] In a first aspect, the present invention provides a method for constructing a sparse perspective 3D model, the method comprising:

[0007] Acquire sparse view images of the target object at different viewpoints;

[0008] Use sparse view images to perform 3D modeling of the target object and obtain the corresponding 3D point cloud and the pose of each image;

[0009] Constructing a neural radiation field model for predicting the SDF field and the RGB field, and based on the sparse view image and its pose and 3D point cloud, using volume rendering to perform joint supervised training on the neural radiation field model to obtain a trained neural radiation field model;

[0010] The preset resolution is input into the trained neural radiation field model to obtain the corresponding SDF value and RGB value, and based on the obtained SDF value and RGB value, the implicit scene is made explicit using a preset algorithm to obtain a corresponding scene-level or individual-level 3D virtual simulation model.

[0011] The sparse perspective 3D model construction method provided in this embodiment uses multi-perspective images and sparse point clouds for joint supervision, obtains RGB values ​​and depth values ​​through volume rendering, and optimizes the SDF field and RGB field using RGB constraints, depth constraints, and feature constraints to obtain a trained neural radiation field model, thereby realizing accurate object surface reconstruction using sparse perspectives. Then, the trained neural radiation field model is used to query the SDF value and RGB value at the target resolution, and the implicit scene is converted into an explicit 3D simulation scene model or a 3D simulation physical individual model, thereby realizing accurate and efficient simulation model reconstruction.

[0012] In an optional implementation, acquiring sparse perspective images of a target object at different perspectives includes:

[0013] Acquire multiple images from different angles taken by an ordinary monocular camera in a real scene, or multi-view images taken by a virtual camera in a virtual scene.

[0014] The embodiment of the present invention supports multi-angle images of real scenes and virtual scenes at the same time, and can meet the construction of 3D virtual simulation models at the scene level and the individual level.

[0015] In an optional embodiment, the neural radiation field model uses trainable voxel-based neural radiation field weights to represent the SDF field and the RGB field, and uses two lightweight fully connected neural networks to respectively predict the SDF values ​​and RGB values ​​of corresponding points in the 3D space.

[0016] The embodiment of the present invention uses trainable voxel-based neural radiation field weights to represent the SDF field and the RGB field. Compared with the traditional MLP-based neural radiation field, it has the characteristics of small parameter amount and fast training speed.

[0017] In an optional implementation, the neural radiation field model is jointly supervised trained by using volume rendering based on the sparse view image and its position and posture and the 3D point cloud to obtain a trained neural radiation field model, including:

[0018] Randomly select N pixels from the sparse view image set, and emit N rays Ray into the 3D space along the pose P direction of the image to which each pixel belongs;

[0019] Sample M 3D points on each Ray to obtain M*N 3D points;

[0020] In the SDF field and RGB field, input M*N 3D points, and use trilinear sampling to obtain feature maps S with dimensions of M*N*C f and C f ;

[0021] S f and C f Input them into the corresponding fully connected neural network respectively to obtain the RGB value and SDF value corresponding to each point;

[0022] The SDF value is converted into a density value using a conversion equation, and then the RGB value and the density value are volume rendered along each ray to obtain M RGB values ​​and depth values;

[0023] The RGB loss is used to supervise the RGB values ​​obtained from volume rendering for joint optimization of the SDF field and the RGB field;

[0024] The predicted depth values ​​are supervised using depth loss for coarse optimization of the scene geometry in the SDF field;

[0025] The SDF value is constrained using the SDF gradient loss for fine-grained optimization of the surface;

[0026] The local feature loss is used to adjust the feature S of the SDF field f Supervision is performed for implicit optimization of the SDF field feature level.

[0027] The embodiment of the present invention optimizes the SDF field and the RGB field using RGB constraints, depth constraints, and feature constraints to obtain a trained neural radiation field model, thereby realizing accurate object surface reconstruction using sparse perspectives.

[0028] In an optional implementation, the local feature loss is expressed by the following formula:

[0029] L E =L n +L gt +Linv

[0030]

[0031]

[0032] Among them, the local feature loss L E The three losses (L n ,L gt ,L inv ),where L n It is a normal vector constraint, which makes the normal vector calculated by the surface point x and the surrounding local features of the object as close to the real normal vector as possible; L gt is a true value constraint, which makes the SDF value calculated by the local features of the object surface point x and its surroundings as close to 0 as possible; L inv It is an inverse constraint, which makes the SDF value of the local feature calculation point of the non-object surface point and the surrounding area in space as far away from 0 as possible; n 1 represents a point cloud set, S() represents an MLP network for predicting SDF, Indicates the SDF feature corresponding to point x, n 2 Indicates the space divided by n 1 Cos represents the cosine similarity, and θ represents an inverse constraint function, which is used to constrain the SDF value of non-surface points to a number as far away from 0 as possible.

[0033] In an optional implementation, the conversion equation is expressed as:

[0034]

[0035] Among them, s is the SDF value, β is a hyperparameter, and the range is (0,1);

[0036] After obtaining the density field σ based on the converted density value, for the M 3D points on a ray r, the RGB value C and depth value D of the 2D pixel corresponding to this ray r are calculated by volume rendering:

[0037]

[0038] in:

[0039]

[0040] in, Represents the RGB value corresponding to the point i sampled by the ray r, Indicates the intercept of the sampling point from the initial point on the ray, represents the interval between two sampling points, Represents the cumulative transmittance of point i sampled on ray r, represents the volume density of point i sampled on ray r, Represents a process variable related to volume density, which itself is related to the opacity at the current point.

[0041] In a second aspect, the present invention provides a sparse perspective 3D model construction device, the device comprising:

[0042] A sparse perspective image acquisition module, used to acquire sparse perspective images of a target object at different perspectives;

[0043] The sparse 3D scene reconstruction module is used to perform 3D modeling of the target object using sparse view images and obtain the corresponding 3D point cloud and the pose of each image;

[0044] A model training module is used to construct a neural radiation field model for predicting the SDF field and the RGB field, and based on the sparse view image and its pose and 3D point cloud, the neural radiation field model is jointly supervised trained by volume rendering to obtain a trained neural radiation field model;

[0045] The 3D model construction module is used to input the preset resolution into the trained neural radiation field model to obtain the corresponding SDF value and RGB value, and based on the obtained SDF value and RGB value, use the preset algorithm to make the implicit scene explicit, and obtain the corresponding scene-level or individual-level 3D virtual simulation model.

[0046] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the sparse perspective 3D model construction method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0047] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the sparse perspective 3D model construction method of the first aspect or any corresponding embodiment thereof.

[0048] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the sparse perspective 3D model construction method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 is a schematic diagram of a process of constructing a sparse perspective 3D model according to an embodiment of the present invention;

[0051] Figure 2 is a schematic diagram of capturing multiple images at different angles in a real scene according to an embodiment of the present invention;

[0052] Figure 3 is a schematic diagram of a 3D point cloud sparsely reconstructed using an SFM algorithm according to an embodiment of the present invention;

[0053] Figure 4 is a flowchart of another sparse perspective 3D model construction method according to an embodiment of the present invention;

[0054] Figure 5 is a flow chart of training a neural radiation field model according to an embodiment of the present invention;

[0055] Figure 6 is a structural block diagram of a sparse perspective 3D model construction device according to an embodiment of the present invention;

[0056] Figure 7 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0058] According to an embodiment of the present invention, a method for constructing a sparse perspective 3D model is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer device such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0059] In this embodiment, a sparse perspective 3D model construction method is provided, which can be used in the above-mentioned computer device. Figure 1 is a flow chart of a method for constructing a sparse perspective 3D model according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0060] Step S101, obtaining a set of sparse perspective images of a target object at different perspectives.

[0061] Specifically, the target object can be an object or scene in a real scene or a virtual scene. In a real scene, a plurality of images at different angles can be taken by an ordinary monocular camera, and in a virtual scene, a multi-view image can be taken by a virtual camera, thereby obtaining a sparse view image set I = {i 1 ,i 2 ,i 3 ,…,i m} (m is about 10-20), the present invention can collect a relatively small amount of images for subsequent model training, which can greatly reduce the dependence on multi-view images, such as Figure 2 The images shown are examples of multiple images taken at different angles in a real scene.

[0062] Step S102, reconstructing a sparse 3D model for the sparse view images to obtain the corresponding 3D point cloud and the pose of each image.

[0063] The embodiment of the present invention uses the commonly used SFM algorithm (a general term for three-dimensional reconstruction algorithms) Colmap to realize the sparse view image set I = {i 1 ,i 2 ,i 3 ,…,i m}, and obtain the reconstructed point cloud PC and the pose set P of each image = {p 1 ,p 2 ,p 3 ,…,p m}(like Figure 3 shown).

[0064] Step S103, constructing a neural radiation field model for predicting the SDF field and the RGB field, and based on the sparse view image and its pose and 3D point cloud, using volume rendering to perform joint supervised training on the neural radiation field model to obtain a trained neural radiation field model.

[0065] The embodiment of the present invention uses a set of trainable weights to represent the 3D SDF field (signed distance field) and the RGB field, wherein the SDF field is responsible for storing the 3D geometric information described by the SDF, and the RGB field is responsible for storing the RGB information in the space, thereby realizing the representation of the SDF information and color information of the 3D scene.

[0066] The embodiment of the present invention adopts a neural radiation field based on voxel (voxel, represented as a 4D tensor [x, y, z, feature], where x, y, z represent coordinates in three-dimensional space, and feature represents feature dimension) to represent a three-dimensional scene, by constructing two sets of 4D trainable voxel features with dimensions [x, y, z, c] to represent the SDF field and RGB field respectively. Compared with the traditional neural radiation field based on MLP (multi-layer perceptron), the voxel-based radiation field has the characteristics of small parameters and fast training speed. For any point in space, the RGB feature corresponding to the point can be queried from the two sets of voxel features by trilinear sampling. and SDF features (i represents the i-th 3D point), they are all vectors of dimension 1*c, and these parameters will be optimized using the back propagation method during the training process. For the SDF field and the RGB field, the embodiment of the present invention constructs a 3-layer fully connected neural network (MLP) s and MLP C ), used to predict SDF value and RGB value. In the specific implementation, for a 3D point i, after passing through the fully connected neural network, the RGB feature can be obtained and SDF features Next, in order to predict the color of the point, Connect with the coordinates of point i and the corresponding azimuth, and input the connected vector into the MLP C In the prediction of SDF, since SDF is independent of angle (only related to the position of coordinates in space), we only need to Connect with the corresponding coordinates and input into MLP C In the calculation, the SDF value (1*1) is obtained.

[0067] During the training process, multi-view images and sparse point clouds are used for joint supervision. RGB values ​​and depth values ​​are obtained through volume rendering. The SDF field and RGB field are optimized using RGB constraints, depth constraints, and feature constraints, thereby achieving accurate object surface reconstruction using sparse viewpoints.

[0068] Step S104, input the preset resolution into the trained neural radiation field model to obtain the corresponding SDF value and RGB value, and based on the obtained SDF value and RGB value, use the preset algorithm to make the implicit scene explicit, and obtain the corresponding scene-level or individual-level 3D virtual simulation model.

[0069] In practical applications, the resolution corresponding to the target 2D image is input into the trained neural radiation field model to obtain the corresponding SDF value and RGB value, and the implicit scene is converted into an explicit Mesh through the Marching Cubes algorithm to obtain a 3D simulation scene model or a 3D simulation physical individual model.

[0070] The sparse view 3D model construction method provided in this embodiment uses a relatively small number of sparse view images for 3D reconstruction to obtain the corresponding 3D point cloud and the pose of each image, and performs joint supervision based on this, obtains RGB values ​​and depth values ​​through volume rendering, and optimizes the SDF field and RGB field using RGB constraints, depth constraints, and feature constraints to obtain a trained neural radiation field model, which realizes the accurate reconstruction of the object surface using sparse view. Then, the trained neural radiation field model is used to query the SDF and RGB values ​​at the target resolution, and the implicit scene is converted into an explicit one to realize accurate and efficient 3D simulation model reconstruction.

[0071] In this embodiment, a sparse perspective 3D model construction method is provided, which can be used in the above-mentioned computer device. Figure 4 is a flow chart of a method for constructing a sparse perspective 3D model according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0072] Step S401, obtain a set of sparse view images of the target object at different view angles. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0073] Step S402: Use the sparse view images to perform 3D modeling on the target object, and obtain the corresponding 3D point cloud and the pose of each image; see Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0074] Step S403, construct a neural radiation field model for predicting the SDF field and the RGB field, and perform joint supervised training on the neural radiation field model by volume rendering based on the sparse view image and its pose and 3D point cloud to obtain a trained neural radiation field model; the embodiment of the present invention is as follows Figure 5 The process of training the neural radiation field model using the sparse perspective image information of the entity individual shown is exemplified, and specifically includes the following steps:

[0075] Step S4031, randomly selecting N pixels from the sparse view image set, and emitting N rays Ray into the 3D space along the direction of the pose P of the image to which each pixel belongs;

[0076] Step S4032, sampling M 3D points on each Ray to obtain M*N 3D points;

[0077] Step S4033: input M*N 3D points in the SDF field and the RGB field, and obtain feature maps S with dimensions of M*N*C by trilinear sampling. f and C f ;

[0078] Step S4034, S f and C f Input them into the corresponding fully connected neural network respectively to obtain the RGB value and SDF value corresponding to each point;

[0079] Step S4035, convert the SDF value into a density value using a conversion equation, and then perform volume rendering on the RGB value and the density value along each ray to obtain M RGB values ​​and a depth value; wherein the conversion equation is expressed as:

[0080]

[0081] Where s is the SDF value and β is a hyperparameter with a range of (0,1).

[0082] After obtaining the density field σ based on the converted density value, for the M 3D points on a ray r, the RGB value C and depth value D of the 2D pixel corresponding to this ray r are calculated by volume rendering:

[0083]

[0084] in:

[0085]

[0086] in, Represents the RGB value corresponding to the point i sampled by the ray r, Indicates the intercept of the sampling point from the initial point on the ray, represents the interval between two sampling points, Represents the cumulative transmittance of point i sampled on ray r, represents the volume density of point i sampled on ray r, Represents a process variable related to volume density, which itself is related to the opacity at the current point.

[0087] Step S4036, using RGB loss to supervise the RGB value obtained by volume rendering, for joint optimization of the SDF field and the RGB field; specifically, the RGB loss is expressed as:

[0088] L rgb =||C gt -C|| 2

[0089] Among them, C gt represents the true value of the current pixel color, and C represents the predicted value of the pixel color.

[0090] Step S4037, using the depth loss to supervise the predicted depth value, for rough optimization of the scene geometry in the SDF field; this step solves the geometric ambiguity problem of sparse perspective RGB information. The depth loss is expressed as:

[0091] L depth =||D gt -D|| 2

[0092] Among them, D gt By point cloud PC in image set I = {i 1 ,i 2 ,i 3 ,…,i m}Using the pose set P = {p 1 ,p 2 ,p 3 ,…,p m} is projected, and D represents the predicted value of the depth value.

[0093] Step S4038, using SDF gradient loss to constrain the SDF value for fine-grained optimization of the surface; wherein the SDF gradient loss is:

[0094]

[0095] Where x is any point in space, and S(x) represents the SDF value corresponding to the point. Indicates calculating the gradient of the current SDF value with respect to x.

[0096] Step S4039, using local feature loss to calculate the feature S of the SDF field f Supervision is used for implicit optimization of the SDF field feature level. The local feature loss is expressed using the Eikonal loss as:

[0097] L E =L n +L gt +L inv

[0098]

[0099] Among them, L E The three losses (L n ,L gt ,L inv ),where L nIt is a normal vector constraint, which makes the normal vector calculated by the surface point x and the surrounding local features of the object as close to the real normal vector as possible; L gt is a true value constraint, which makes the SDF value calculated by the local features of the object surface point x and its surroundings as close to 0 as possible; L inv It is an inverse constraint, which makes the SDF value of the local feature calculation of the non-object surface point and the surrounding area in space as far away from 0 as possible. 1 represents the collection of point cloud PC (object surface points), S() represents the MLP network (i.e. MLP) used to predict SDF s ), Represents the SDF feature corresponding to point x (, n 2 Indicates the space divided by n 1 The set of other points outside (in actual operation, n 2 is a non-object surface point randomly selected in space), Cos represents cosine similarity, and θ represents an inverse constraint function, which is used to constrain the SDF value of the non-surface point to a number as far away from 0 as possible.

[0100] Step S404: Input the preset resolution into the trained neural radiation field model to obtain the corresponding SDF value and RGB value, and use the preset algorithm to make the implicit scene explicit based on the obtained SDF value and RGB value, and obtain the corresponding scene-level or individual-level 3D virtual simulation model. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0101] In this embodiment, a sparse perspective 3D model construction device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0102] This embodiment provides a sparse perspective 3D model construction device, such as Figure 6 As shown, including:

[0103] The sparse perspective image acquisition module 601 is used to acquire sparse perspective images of the target object at different perspectives;

[0104] The sparse 3D scene reconstruction module 602 is used to perform 3D modeling of the target object using the sparse view images to obtain the corresponding 3D point cloud and the pose of each image;

[0105] A model training module 603 is used to construct a neural radiation field model for predicting the SDF field and the RGB field, and based on the sparse view image and its pose and 3D point cloud, the neural radiation field model is jointly supervised trained by using volume rendering to obtain a trained neural radiation field model;

[0106] The 3D model construction module 604 is used to input the preset resolution into the trained neural radiation field model to obtain the corresponding SDF value and RGB value, and based on the obtained SDF value and RGB value, use the preset algorithm to make the implicit scene explicit, and obtain the corresponding scene-level or individual-level 3D virtual simulation model.

[0107] In some optional implementations, the sparse view image acquisition module 601 acquires multiple images taken at different angles by an ordinary monocular camera in a real scene, or multi-view images taken by a virtual camera in a virtual scene.

[0108] In some optional embodiments, the neural radiation field model in the model training module 603 uses trainable voxel-based neural radiation field weights to represent the SDF field and RGB field, and uses two lightweight fully connected neural networks to respectively predict the SDF values ​​and RGB values ​​of corresponding points in the 3D space.

[0109] In some optional implementations, the model training module 603 includes:

[0110] A ray generation unit is used to randomly select N pixels from the sparse view image set and emit N rays Ray into the 3D space along the pose P direction of the image to which each pixel belongs;

[0111] 3D point sampling unit, used to sample M 3D points on each Ray to obtain M*N 3D points;

[0112] The feature map acquisition unit is used to input M*N 3D points in the SDF field and the RGB field, and obtain the feature map S with dimensions of M*N*C by trilinear sampling. f and C f ;

[0113] RGB value and SDF value prediction unit, used to convert S f and C f Input them into the corresponding fully connected neural network respectively to obtain the RGB value and SDF value corresponding to each point;

[0114] A volume rendering unit, used to convert the SDF value into a density value using a conversion equation, and then perform volume rendering on the RGB value and the density value along each ray to obtain M RGB values ​​and a depth value;

[0115] A first optimization unit, used for supervising the RGB values ​​obtained by volume rendering using RGB loss, and for jointly optimizing the SDF field and the RGB field;

[0116] The second optimization unit supervises the predicted depth value using the depth loss for coarse optimization of the scene geometry in the SDF field;

[0117] The third optimization unit uses the SDF gradient loss to constrain the SDF value for fine-grained optimization of the surface;

[0118] The fourth optimization unit uses local feature loss to optimize the feature S of the SDF field. f Supervision is performed for implicit optimization of the SDF field feature level.

[0119] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0120] The sparse perspective 3D model construction device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0121] The embodiment of the present invention also provides a computer device having the above Figure 6 The sparse perspective 3D model construction device shown.

[0122] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0123] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0124] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0125] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0126] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0127] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0128] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0129] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0130] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for constructing a sparse perspective 3D model, characterized in that: The method comprises: Acquire sparse view images of the target object at different viewpoints; Use sparse view images to perform 3D modeling of the target object and obtain the corresponding 3D point cloud and the pose of each image; Constructing a neural radiation field model for predicting the SDF field and the RGB field, and based on the sparse view image and its pose and 3D point cloud, using volume rendering to perform joint supervised training on the neural radiation field model to obtain a trained neural radiation field model; The preset resolution is input into the trained neural radiation field model to obtain the corresponding SDF value and RGB value, and based on the obtained SDF value and RGB value, the implicit scene is made explicit using a preset algorithm to obtain a corresponding scene-level or individual-level 3D virtual simulation model.

2. The method according to claim 1, characterized in that Obtain sparse view images of the target object at different viewpoints, including: Acquire multiple images from different angles taken by an ordinary monocular camera in a real scene, or multi-view images taken by a virtual camera in a virtual scene.

3. The method according to claim 1, characterized in that The neural radiation field model uses trainable voxel-based neural radiation field weights to represent the SDF field and the RGB field, and uses two lightweight fully connected neural networks to respectively predict the SDF values ​​and RGB values ​​of corresponding points in the 3D space.

4. The method according to claim 3, characterized in that Based on the sparse view image and its position and posture and the 3D point cloud, the neural radiation field model is jointly supervised trained by using volume rendering to obtain a trained neural radiation field model, including: Randomly select N pixels from the sparse view image set, and emit N rays Ray into the 3D space along the pose P direction of the image to which each pixel belongs; Sample M 3D points on each Ray to obtain M*N 3D points; In the SDF field and RGB field, input M*N 3D points, and use trilinear sampling to obtain feature maps S with dimensions of M*N*C f and C f ; S f and C f Input them into the corresponding fully connected neural network respectively to obtain the RGB value and SDF value corresponding to each point; The SDF value is converted into a density value using a conversion equation, and then the RGB value and the density value are volume rendered along each ray to obtain M RGB values ​​and depth values; The RGB loss is used to supervise the RGB values ​​obtained from volume rendering for joint optimization of the SDF field and the RGB field; The predicted depth values ​​are supervised using depth loss for coarse optimization of the scene geometry in the SDF field; The SDF value is constrained using the SDF gradient loss for fine-grained optimization of the surface; The feature S of the SDF field is quantized using local feature loss. f Supervision is performed for implicit optimization of the SDF field feature level.

5. The method according to claim 4, characterized in that The local feature loss is expressed by the following formula: L E =L n +L gt +L inv Among them, the local feature loss L E The three losses (L n ,L gt ,L inv ),where L n It is a normal vector constraint, which makes the normal vector calculated by the surface point x and the surrounding local features of the object as close to the real normal vector as possible; L gt is a true value constraint, which makes the SDF value calculated by the local features of the object surface point x and its surroundings as close to 0 as possible; L inv It is an inverse constraint, which makes the SDF value of the local feature calculation of the non-object surface point and the surrounding area in the space as far away from 0 as possible; n1 represents the point cloud set, S() represents the MLP network used to predict SDF, represents the SDF feature corresponding to the point x, n2 represents the set of other points in the space except n1, Cos represents the cosine similarity, and θ represents an inverse constraint function, which is used to constrain the SDF value of non-surface points to a number as far away from 0 as possible.

6. The method according to claim 4, characterized in that The conversion equation is expressed as: Among them, s is the SDF value, β is a hyperparameter, and the range is (0,1); After obtaining the density field σ based on the converted density value, for the M 3D points on a ray r, the RGB value C and depth value D of the 2D pixel corresponding to this ray r are calculated by volume rendering: in: in, Represents the RGB value corresponding to the point i sampled by the ray r, Indicates the intercept of the sampling point from the initial point on the ray, represents the interval between two sampling points, It represents the cumulative transmittance of the point i sampled on the ray r, represents the volume density of point i sampled on ray r, Represents a process variable related to volume density, which itself is related to the opacity at the current point.

7. A sparse perspective 3D model construction device, characterized in that: The device comprises: A sparse perspective image acquisition module, used to acquire sparse perspective images of a target object at different perspectives; The sparse 3D scene reconstruction module is used to perform 3D modeling of the target object using sparse view images and obtain the corresponding 3D point cloud and the pose of each image; A model training module is used to construct a neural radiation field model for predicting the SDF field and the RGB field, and based on the sparse view image and its pose and 3D point cloud, the neural radiation field model is jointly supervised trained by volume rendering to obtain a trained neural radiation field model; The 3D model construction module is used to input the preset resolution into the trained neural radiation field model to obtain the corresponding SDF value and RGB value, and based on the obtained SDF value and RGB value, use the preset algorithm to make the implicit scene explicit, and obtain the corresponding scene-level or individual-level 3D virtual simulation model.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the sparse perspective 3D model construction method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the sparse perspective 3D model construction method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the sparse perspective 3D model construction method according to any one of claims 1 to 6.

Images