NeRF model construction method and device, equipment and storage medium

Abstract

The present disclosure provides a method, device and equipment for constructing a NeRF model and a storage medium. The method comprises: obtaining a depth map of a target three-dimensional model at a reference view angle specified from a plurality of preset view angles, and a feature map corresponding to the reference view angle; inputting the depth map of the reference view angle and a prompt text input by a user into an image generation model, and obtaining a style map generated by the image generation model, which has the same depth information as the depth map and matches the style prompted by the prompt text; constructing a training sample based on each feature vector contained in the feature map of the reference view angle and a pixel value at a position corresponding to each feature vector in the style map, training a preset neural network model based on the training sample, and generating a style map corresponding to each view angle other than the reference view angle among the plurality of view angles based on the trained neural network model; and training a NeRF model based on the style map corresponding to each view angle among the plurality of view angles.

Description

Technical Field

[0001] This disclosure relates to the field of machine learning technology, and in particular to methods, apparatus, devices and storage media for constructing NeRF models. Background Technology

[0002] NeRF (Neural Radiance Fields) is a deep learning model used for 3D scene reconstruction and rendering. The main idea behind NeRF is to represent a 3D scene as a neural radiation field constructed from a deep neural network, enabling 3D scene reconstruction and high-quality image generation. In related technologies, to obtain a NeRF model for a specific target, the user needs to train the model using a set of images taken from different perspectives of that target. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides a method, apparatus, device and storage medium for constructing NeRF models.

[0004] According to a first aspect of the embodiments of this specification, a method for constructing a NeRF model is provided, the method comprising:

[0005] Obtain a depth map of the target 3D model from a reference viewpoint specified from multiple preset viewpoints; and a feature map corresponding to the reference viewpoint.

[0006] The depth map of the reference viewpoint and the prompt text input by the user are input into the image generation model, and a style map generated by the image generation model with the same depth information as the depth map and matching the style of the prompt text is obtained;

[0007] Training samples are constructed based on the feature vectors contained in the feature map of the reference view and the pixel values ​​of the feature vectors at the corresponding positions in the style map. A preset neural network model is trained based on the training samples, and style maps corresponding to each of the other views in the plurality of views other than the reference view are generated based on the trained neural network model.

[0008] A NeRF model corresponding to the target 3D model is trained based on the style map corresponding to each of the multiple perspectives.

[0009] According to a second aspect of the embodiments of this specification, an apparatus for constructing a NeRF model is provided, comprising:

[0010] The acquisition module acquires a depth map of the target 3D model from a reference viewpoint specified from multiple preset viewpoints, and a feature map corresponding to the reference viewpoint.

[0011] The style map generation module inputs the depth map of the reference viewpoint and the prompt text input by the user into the image generation model, and obtains a style map generated by the image generation model that has the same depth information as the depth map and matches the style of the prompt text.

[0012] The neural network model processing module constructs training samples based on the feature vectors contained in the feature map of the reference view and the pixel values ​​of the feature vectors at the corresponding positions in the style map, trains a preset neural network model based on the training samples, and generates style maps corresponding to each of the other views among the multiple views except the reference view based on the trained neural network model.

[0013] The NeRF model training module trains a NeRF model corresponding to the target 3D model based on the style map corresponding to each of the multiple perspectives.

[0014] According to a third aspect of the embodiments of this specification, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method embodiments described in the first aspect above.

[0015] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method embodiments described in the first aspect above.

[0016] The technical solutions provided in the embodiments of this specification may include the following beneficial effects:

[0017] In this embodiment, a style map with the same depth information as the depth map and matching style with the prompt text is generated by an image generation model based on the depth map of the target 3D model from a reference viewpoint and the prompt text input by the user. Then, a neural network model learns the correspondence between each feature vector in the feature map of the reference viewpoint and the pixel values ​​of the corresponding positions of the feature vectors in the style map, and uses the neural network model to derive style maps for other viewpoints. Thus, this embodiment can automatically simulate and generate high-quality images from multiple viewpoints without requiring actual user photography, and can quickly construct a high-quality NeRF model.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this specification and, together with the specification, serve to explain the principles of this disclosure.

[0020] Figure 1A This is a flowchart illustrating a method for constructing a NeRF model according to an exemplary embodiment of this specification.

[0021] Figure 1B This is a schematic diagram of a style diagram of a reference view of a target three-dimensional model according to an exemplary embodiment of this specification.

[0022] Figure 1C This is a schematic diagram illustrating a reference view and other views of a target three-dimensional model according to an exemplary embodiment of this specification.

[0023] Figure 2A This is a flowchart illustrating another method for constructing a NeRF model according to an exemplary embodiment of this specification.

[0024] Figure 2B This is a schematic diagram of a set of renderings of a reference viewpoint shown in this specification according to an exemplary embodiment.

[0025] Figure 2C This is a schematic diagram of a collection of renderings from other perspectives illustrated in this specification according to an exemplary embodiment.

[0026] Figure 3 This is a hardware structure diagram of a computer device containing a NeRF model construction apparatus as illustrated in this specification according to an exemplary embodiment.

[0027] Figure 4 This is a block diagram illustrating an apparatus for constructing a NeRF model according to an exemplary embodiment of this specification. Detailed Implementation

[0028] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0029] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0030] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0031] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.

[0032] In related technologies, one way for users to obtain the specific NeRF model they need is to train the NeRF model using a large amount of specific 3D data and perspectives.

[0033] Currently, image analogy technology is developing rapidly. Research has shown that NeRF models can be generated based on image analogy. For example, Deep Image Analogy is an image analogy algorithm built using deep learning methods. This method only requires the user to input the original image A and the style B' of the target image, and it can recreate the stylized original image A' and target image B quite well. However, it currently only shows good image analogy results in certain specific fields, such as images of faces and landscapes, and is difficult to apply to the generation of NeRF models.

[0034] Another approach, such as DreamFusion and its related methods, uses the StableDiffusion model or other diffusion-based text-image generation models to guide the training of NeRF models. However, generating the NeRF model in this approach is time-consuming.

[0035] This specification provides a NeRF model construction scheme. Based on the depth map of the target 3D model from a reference viewpoint and the user-input prompt text, an image generation model generates a style map with the same depth information as the depth map and matching the style of the prompt text. Then, a neural network model learns the correspondence between each feature vector in the feature map of the reference viewpoint and the corresponding pixel values ​​in the style map, and uses the neural network model to derive style maps for other viewpoints. Thus, this embodiment can automatically simulate and generate high-quality images from multiple viewpoints without requiring actual user photography, and can quickly construct a high-quality NeRF model. The following is a detailed description of this embodiment.

[0036] like Figure 1A The diagram shown is a flowchart illustrating a method for constructing a NeRF model according to an exemplary embodiment, comprising the following steps:

[0037] Step 102: Obtain the depth map of the target 3D model from a reference viewpoint specified from multiple preset viewpoints; and the feature map corresponding to the reference viewpoint.

[0038] Step 104: Input the depth map of the reference viewpoint and the prompt text input by the user into the image generation model, and obtain a style map generated by the image generation model that has the same depth information as the depth map and matches the style of the prompt text.

[0039] Step 106: Construct training samples based on the feature vectors contained in the feature map of the reference view and the pixel values ​​of the corresponding positions of the feature vectors in the style map, train a preset neural network model based on the training samples, and generate style maps corresponding to each of the other views among the multiple views except the reference view based on the trained neural network model.

[0040] Step 108: Train a NeRF model corresponding to the target 3D model based on the style map corresponding to each of the multiple perspectives.

[0041] In this embodiment, the training process of the NeRF model can be as follows: given a set of continuously captured images and poses of a specific scene, the NeRF model can try to use the light position, illumination direction, and corresponding 3D coordinates (x, y, z) as inputs, and output the density (shape) and color of the target. There are a total of five input variables, hence the term "5D radiation field". Specifically, given the spatial coordinates (x, y, z) and the observation direction (any two of d_x, d_y, d_z, the third being obtained through a cross product), the density value (i.e., the probability that the light terminates at that point) and the corresponding color value (e.g., RGB value) of that point can be calculated. After predicting the color value and calculating the loss with the corresponding input image under the current pose, optimization can be performed to gradually converge the NeRF model, resulting in a trained NeRF model.

[0042] Building a NeRF model requires images of an object from multiple perspectives. Based on the aim of this embodiment to eliminate the need for users to actually photograph the object, the images from multiple perspectives required for building the NeRF model are automatically generated through simulation. Therefore, it is necessary to ensure that the color information for the same location on the object is consistent across images from different perspectives. For example, in the simulated images of a house from different perspectives, the shape and color of the house's door need to remain consistent; if they are inconsistent—for example, the door is black in one perspective but red in another—a high-quality NeRF model cannot be built using these images. Therefore, this embodiment uses high-quality images from multiple perspectives automatically generated from steps 102 to 108 for the data used to train the NeRF model in step 108.

[0043] In this embodiment, step 102 introduces a target 3D model. Here, the 3D model refers to a mathematical representation based on a 3D coordinate system, used to describe the shape, geometric structure, and surface features of an object or scene in 3D space. It can be a collection of geometric elements such as points, lines, and surfaces, or a solid entity composed of complex curves, surfaces, or even volumes. The 3D model can be created and edited using existing professional modeling software, or a real object can be converted into a 3D model using a 3D scanner. As an example, the 3D model can be an OBJ file, etc.

[0044] As an example, the target 3D model can be user-inputted, and this input can take several forms; for example, multiple 3D models of objects can be prepared in advance for the user to choose from, and the user-selected 3D model can be used as the target 3D model; alternatively, an input interface can be provided to obtain the 3D model provided by the user. In step 104 of this embodiment, the user can also input prompt text. Thus, this embodiment can generate a NeRF model that conforms to the style indicated in the user's prompt text based on the user-input target 3D model and the prompt text. As an example, if the user inputs a 3D model of a house and the prompt text "Chinese-style house model", this embodiment can construct a Chinese-style NeRF model of the house based on the 3D model of the house.

[0045] In some cases, considering that the goal is to generate a model that matches the style of the prompt text, the target 3D model can be a 3D model without color information; for example, the user input can be a 3D model without color information, or the color information can be removed from the user input 3D model with color information to obtain a 3D model without color information.

[0046] Based on this, in step 104, the depth map from the reference viewpoint and the user-inputted prompt text are input into the image generation model, and a style map generated by the image generation model is obtained, which has the same depth information as the depth map and matches the style of the prompt text. The reference viewpoint can be any viewpoint, and can be flexibly selected as needed in practical applications; this embodiment does not limit it. As an example, the reference viewpoint can be at an angle such as 45° above, so that the depth map covers the entire object as comprehensively as possible. Of course, in practical applications, it can be adaptively adjusted according to the objects in the target 3D model.

[0047] Depth maps can be obtained in several ways. For example, a 3D model file can be loaded into modeling software. A specific viewpoint can be selected as a reference viewpoint in the software, and an image of the 3D model can be captured from this reference viewpoint. The depth information of each pixel in the image can then be extracted to obtain a depth map, which contains the depth information of every point in the 3D model from the reference viewpoint. Depth maps can be constructed using grayscale values ​​from 0 to 255. A grayscale value of 0 represents the farthest region in the image, while a grayscale value of 255 represents the closest region.

[0048] The image generation model can be flexibly selected or trained independently according to actual needs, and its specific structure can also be constructed as needed. Optionally, the image generation model may include: Generative Adversarial Network (GAN) model, flow-based model, or diffusion model, etc., which are not limited in this embodiment.

[0049] As an example, the image generation model includes an image generation model with a ControlNet, for example, a diffusion model with a ControlNet. Here, ControlNet is a neural network architecture used to control a pre-trained large-scale diffusion model to support additional input conditions. The image generation model in this embodiment can be enhanced with ControlNet to implement conditional input, where the conditional input can be the depth map in this embodiment.

[0050] The prompt text can be used to guide the model in generating output of a specific type, theme, or format. The image generation model can be guided by the depth map, and through learned model weights and parameters, generate a style map with the same depth information as the depth map and matching the style of the prompt text.

[0051] For example, such as Figure 1B As shown, Figure 1B The left side shows the depth map of the 3D model D from the reference viewpoint, while the right side shows the colorized style map 1 output by the image generation model.

[0052] Having obtained the style map from the reference viewpoint, this embodiment designs a method to generate style maps for other viewpoints based on the style map from the reference viewpoint. The viewpoints in this embodiment can be flexibly configured according to actual needs; for example, they can be sampled spatially evenly based on horizontal and vertical angles.

[0053] like Figure 1C As shown, the top and bottom images on the left illustrate the grayscale image and style map of the target 3D model from the reference viewpoint; the top image on the right is the grayscale image of the target 3D model from other viewpoints. Based on the style map from the reference viewpoint, accurate style maps for other viewpoints (i.e., the images indicated by the question marks in the figures) need to be derived. As mentioned in the previous analysis, it is necessary to ensure that the color information for the same location of the object is consistent across images from different viewpoints. Therefore, this embodiment includes step 106, where a neural network model learns the relationship between the feature vector of each feature point in the target 3D model from the reference viewpoint and the pixel value at the corresponding position in the style map. Then, the neural network model derives the pixel values ​​that the feature vector at the same position should have in other viewpoints.

[0054] In some cases, the neural network model can be flexibly selected according to actual needs. It can be a small neural network model, such as a multilayer perceptron (MLP), etc. This embodiment does not limit this.

[0055] In this embodiment, feature vectors contained in the feature map of the reference viewpoint can be extracted in various ways, and then the pixel values ​​at the corresponding positions of each feature vector in the style map are obtained to construct training samples. The input of the neural network model is the feature vector, and the output is the pixel value. Appropriate optimizers (e.g., Adam) and appropriate loss functions, such as L2 loss, can be used as needed.

[0056] In some examples, the style map corresponding to any of the other target viewpoints can be generated in the following way:

[0057] Obtain the feature map corresponding to the target viewpoint;

[0058] Each feature vector contained in the feature map corresponding to the target viewpoint is input into the trained neural network model, and the pixel values ​​corresponding to each feature vector generated by the neural network model are obtained.

[0059] Based on the obtained pixel values ​​corresponding to each feature vector, a style map corresponding to the target viewpoint is further generated.

[0060] In this way, a trained neural network model can be used to derive style maps from other perspectives, ensuring that the color information for the same position of an object is consistent in images from different perspectives.

[0061] In some examples, the feature vectors contained in the feature map of any of the multiple viewpoints may include: feature vectors composed of feature values ​​obtained by feature mapping of the coordinate values ​​of feature points contained in the target 3D model.

[0062] Thus, this embodiment can use the positional information of feature points contained in the 3D model as the basis for subsequent feature calculations, thereby improving the accuracy, robustness, and ease of use of this embodiment.

[0063] In some examples, the positional information of feature points contained in the 3D model can be encoded into color information. For example, the feature map corresponding to any one of the multiple viewpoints can be obtained by stitching together the feature maps of each rendering map contained in the rendering map set corresponding to that viewpoint. The rendering map set corresponding to any one of the multiple viewpoints can include: a grayscale rendering map corresponding to the target 3D model under that viewpoint; and a rendering map corresponding to the tinted 3D model under that viewpoint. The tinted 3D model is a 3D model obtained by mapping the coordinate values ​​of feature points contained in the target 3D model to color values.

[0064] One approach is to render the image of a target 3D model as grayscale from any viewpoint; the resulting image is called a grayscale rendering. A grayscale rendering is a rendering that converts color information into grayscale information. In a grayscale rendering, color information is represented as gray values ​​of varying brightness, without including color information. Therefore, a grayscale rendering carries the lighting and shadow information of feature points in the target 3D model, allowing subsequent feature extraction based on the grayscale rendering to extract the lighting and shadow characteristics of these feature points.

[0065] In this embodiment, the target 3D model can also be colored. By mapping the coordinate values ​​of feature points contained in the target 3D model to color values, the position information is encoded into the colored 3D model.

[0066] In some examples, the shaded 3D model may include: a shaded 3D model corresponding to each of the three coordinate axes in a 3D coordinate system; the shaded 3D model corresponding to any coordinate axis is a 3D model obtained by mapping the coordinate values ​​of each feature point in the target 3D model on that coordinate axis to color values;

[0067] The set of rendering images corresponding to any of the plurality of viewpoints includes the rendering images corresponding to the shaded 3D model under that viewpoint, which may include: the rendering images of the shaded 3D model corresponding to each of the three coordinate axes respectively.

[0068] In this embodiment, considering that the coordinates of feature points in the three-dimensional coordinate system are three-dimensional coordinates, the shading three-dimensional model in this embodiment can be three, each corresponding to one of the three coordinate axes in the three-dimensional coordinate system, such as the shading three-dimensional models corresponding to the x-axis, y-axis and z-axis respectively.

[0069] For example, the coordinate format of each feature point in the target 3D model is (x, y, z). The coordinate values ​​of the x-axis of each feature point can be mapped to color values ​​to obtain the colored 3D model of the x-axis. Similarly, the coordinate values ​​of the y-axis of each feature point can be mapped to color values ​​to obtain the colored 3D model of the y-axis. Similarly, the colored 3D model of the z-axis can be obtained.

[0070] In some examples, the colored 3D model can be a 3D model obtained by normalizing the coordinate values ​​of each feature point contained in the target 3D model and mapping the normalized coordinate values ​​to color values.

[0071] In practical applications, the mapping relationship between specific coordinate values ​​and color values ​​can be configured as needed.

[0072] In some examples, the distance between the coordinates of each feature point in the target 3D model and the origin can be positively correlated with the color value (i.e., the mapped color value) of each feature point in the shaded 3D model. Since the distances between the coordinates of different feature points in the target 3D model and the origin vary, the color values ​​corresponding to different feature points in the target 3D model will also differ to some extent in the shaded 3D model.

[0073] For example, in practical applications, the absolute value of the distance between the coordinates of a feature point and the origin can correspond to a larger color value in a shading 3D model, resulting in a darker color; conversely, a smaller absolute value can also result in a darker color. For instance, a larger absolute value can be closer to purple, while a smaller absolute value can be closer to cyan.

[0074] In this way, the position information of each feature point in the target 3D model can be encoded into the color information. In addition, the grayscale rendering image contains the light and shadow information of each feature point in the target 3D model. Therefore, when extracting features from the rendering image set, high-dimensional features containing light and shadow information and position information of each pixel can be extracted.

[0075] In practical applications, feature maps can be extracted in various ways. To extract rich and accurate features, convolutional neural networks (CNNs) can be used, such as the VGG CNN model, which contains multiple convolutional layers and can extract rich and accurate features. Alternatively, the VGG19 model can be used, which consists of 19 convolutional layers and 3 fully connected layers. These convolutional and fully connected layers are arranged sequentially and use the ReLU activation function for non-linear transformation to extract features from the input image. The VGG19 convolutional layers can be divided into several blocks, each containing one or more convolutional layers followed by a pooling layer. The convolutional layers perform convolution operations on the input using a sliding window approach, thereby capturing features at different scales. The pooling layer reduces the size of the feature map by downsampling while preserving the main features. In the VGG19 model, the final fully connected layer flattens the feature map into a one-dimensional vector, and classification is performed using the fully connected layer and the softmax function. The VGG19 model performs well in many computer vision tasks and is suitable for image classification and feature extraction.

[0076] In some examples, the feature map corresponding to any one of the plurality of viewpoints can be obtained in the following way:

[0077] Each of the rendered images in the set of rendered images from any given viewpoint is input into a convolutional neural network model containing M convolutional layers. The feature maps extracted from each of the M convolutional layers are obtained, and the obtained feature maps are concatenated to obtain M tensors that correspond one-to-one with the M convolutional layers. The tensor corresponding to any of the M convolutional layers is obtained by concatenating the feature maps extracted from each of the input rendered images by that convolutional layer.

[0078] The obtained M tensors are concatenated to obtain a feature map corresponding to any one of the viewpoints.

[0079] For example, please see the table below:

[0080]

[0081] The table above shows a set of rendering images from the reference viewpoint: grayscale rendering image At, the x-axis shading 3D model rendering image Ax from the reference viewpoint, the y-axis shading 3D model rendering image Ay from the reference viewpoint, and the z-axis shading 3D model rendering image Az from the reference viewpoint.

[0082] The convolutional neural network model contains m convolutional layers as shown in the table: convolutional layer 1, convolutional layer 2, ... and convolutional layer m.

[0083] In m convolutional layers, taking convolutional layer 1 as an example, feature maps are extracted from the rendered image At to obtain feature maps at1, at2, ..., and am2. The processing of other rendered images is similar.

[0084] For the feature maps at1, ax1, ay1, and az1 extracted from the four rendering images in convolutional layer 1, these four feature maps are concatenated to obtain tensor a1. In this way, the lighting and shadow information in the grayscale rendering image and the position information in the rendering images of the three shading 3D models can be concatenated together.

[0085] Optionally, the tensor corresponding to any of the M convolutional layers can be obtained by concatenating the feature maps extracted from each of the input rendered images by that convolutional layer along the channel dimension. The concatenation of the obtained M tensors can include: concatenating the obtained M tensors along the channel dimension.

[0086] For example, when concatenating four feature maps—at1, ax1, ay1, and az1—they can be obtained by concatenating them along the channel dimension. In this embodiment, features are extracted from the rendered image, and the channel dimension can be the RGB (Red, Green, Blue) dimension.

[0087] The same operation is performed on the feature maps extracted from the four rendered images by other convolutional layers, thus obtaining m tensors.

[0088] Finally, these m tensors are concatenated to obtain the feature map of the reference viewpoint. Optionally, the m tensors corresponding to the set of rendered images of any viewpoint can be obtained and then concatenated along the channel dimension. For example, the m tensors a1 to am mentioned above can be concatenated along the channel dimension to obtain the feature map of the reference viewpoint. Different convolutional layers can extract information from different dimensions. For example, the earlier convolutional layers in a convolutional neural network tend to extract local information in the image, while the later convolutional layers tend to extract global macroscopic information. In this embodiment, concatenating the m tensors at the end can combine all the information extracted by multiple convolutional layers.

[0089] In some examples, considering that the size of the feature maps output by different convolutional layers in a convolutional neural network model may be different, for example, the feature map output by the earlier convolutional layer is 512*512, while the feature map output by the later convolutional layer is 32*32; in this embodiment, the tensor corresponding to any of the M convolutional layers can be obtained by converting the feature maps extracted by that convolutional layer from each of the input rendering images into a preset size and then concatenating them along the channel dimension.

[0090] The preset size can be configured as needed. For example, the preset size could be the size of the style map from the reference viewpoint; for instance, if the style map size is 512*512, then the preset size would be 512*512. Algorithms such as bilinear interpolation can be used to convert the feature maps output by the convolutional layers to the preset size before concatenation. For example, the m tensors from a1 to am mentioned above can all be converted to the same preset size before concatenation.

[0091] The acquisition of feature maps from other perspectives is similar. In some examples, the feature maps from each perspective can also be processed by dimensionality reduction as needed. For example, to improve the stability and speed of subsequent comparisons, if the feature maps from each perspective have a large number of dimensions, methods such as PCA (Principal Component Analysis) can be used to reduce the feature maps from each perspective to a set number of dimensions, such as 4 dimensions. In practical applications, this can be configured as needed, and this embodiment does not limit this.

[0092] A trained neural network model can be used to generate style maps from other perspectives. For example, a feature map from another perspective contains multiple feature vectors. These feature vectors are input one by one into the neural network model, and the neural network model can output the pixel values ​​corresponding to each feature vector. Therefore, the pixel values ​​of each feature vector in the feature map from another perspective can be obtained, and these pixel values ​​can constitute an image.

[0093] In this embodiment, when using a neural network model to derive the pixel values ​​of feature vectors in feature maps from other perspectives, it is equivalent to performing a search operation on the correspondence between each feature vector and pixel value in the feature map of the reference perspective. That is, searching to find which feature vector in the feature map of the reference perspective is the same as each feature vector in the feature map of other perspectives, thereby obtaining the pixel value corresponding to the same feature vector in the feature map of the reference perspective. This embodiment designs to use a neural network model to implement this search operation, which has low search complexity.

[0094] For example, the size of the rendered image in the reference view's rendering image set is 512*512, and 256-dimensional features need to be extracted. After feature extraction processing, the feature vector of the feature map of the reference view is 512*512*256. The parameter view has a style map, and the size of the style map is 512*512.

[0095] The feature vectors of the feature maps from other perspectives are also 512*512*256. We need to obtain the pixel value of each feature vector in the feature maps from other perspectives. Assuming we don't use the neural network model scheme of this embodiment, the feature maps from other perspectives have 512*512 feature vectors of length 256. We need to compare each of these with the 512*512 feature vectors from the reference perspective to determine which feature vector in the reference perspective has the highest similarity. Thus, the search complexity is O(N). 2 Here, N refers to the image size of 512*512.

[0096] The trained neural network model in this embodiment only requires inputting 512*512 feature vectors into the neural network model. The neural network model will output the corresponding pixel value for each input feature vector of length 256. Therefore, the search time complexity of this embodiment is O(N), which significantly improves efficiency.

[0097] In some examples, the image composed of the pixel values ​​of each feature vector in the feature map of the other viewpoint can be directly used as the style map of the other viewpoint. In other examples, in order to generate a higher quality style map, the step of further generating a style map corresponding to the target viewpoint based on the obtained pixel values ​​corresponding to each feature vector may include:

[0098] The image consisting of the pixel values ​​corresponding to each feature vector, the depth map of the target 3D model in the target viewpoint, and the descriptive text of the style map corresponding to the reference viewpoint are input into the image generation model. The model then generates a style map that has the same depth information as the depth map of the target viewpoint and matches the style indicated by the descriptive text.

[0099] In this embodiment, based on the image constructed from the pixel values ​​corresponding to the feature vectors output by the neural network model in the feature maps of other viewpoints, a high-quality style map is further generated using an image generation model. The input to the image generation model here also includes the depth map of the target 3D model in that viewpoint and descriptive text of the style map from the reference viewpoint. Thus, on the one hand, the depth map of the target 3D model in that viewpoint provides the depth information of the 3D model in that viewpoint; on the other hand, to ensure style consistency with the style map of the reference viewpoint, the input in this embodiment also includes descriptive text of the style map from the reference viewpoint, enabling the image generation model to generate a style map with the same depth information as the depth map of that viewpoint and matching the style indicated by the descriptive text.

[0100] As an example, the image composed of the pixel values ​​corresponding to each feature vector can be obtained by creating an image to be assigned the same size as the style map of the reference viewpoint, and then assigning pixel values ​​to the pixel positions in the image to be assigned based on the pixel values ​​corresponding to each feature vector. Based on this, an image composed of the pixel values ​​of each feature vector in the feature map of the other viewpoint can be quickly obtained.

[0101] Optionally, in practical applications, the weights of the input image and descriptive text in the image generation model can be adjusted as needed.

[0102] Optionally, the descriptive text for the style map of the reference viewpoint can be obtained in various ways. It can be generated based on the prompt text initially input by the user when generating the style map of the reference viewpoint. Alternatively, in this embodiment, the style map of the reference viewpoint can be input into a pre-trained visual text generation model and then generated by the visual text generation model.

[0103] In this embodiment, the visual text generation model can be flexibly selected according to actual needs. This model can generate text from images, and the model outputs text based on the input image. Depending on the actual needs, the input of the model can also include prompt text. As an example, the visual text generation model can be a Blip (Bootstrapping Language-Image Pre-training for Unifified Vision-Language Understanding and Generation) model, etc., and this embodiment does not limit it.

[0104] The following will describe this embodiment again. Figure 2A The image shown is an embodiment of a method for constructing a NeRF model according to an exemplary embodiment of this specification.

[0105] The goal of the construction method in this embodiment is to construct a NeRF model corresponding to the 3D model D that conforms to the style of the prompt text, given a specified 3D model D and a prompt text entered by the user.

[0106] The 3D model D input by the user can be a 3D model without color information, or in this embodiment, the color information can be removed from the 3D model input by the user.

[0107] This embodiment can pre-prepare the following models: a deep learning model (e.g., the VGG19 model), an image generation model with a ControlNet control network (e.g., a stable diffusion model), a visual text generation model (e.g., the Blip model), a neural network model (e.g., an MLP model), and a NeRF model to be trained. The construction process of this embodiment can be as follows:

[0108] (1) Use an image generation model to generate a style map of the reference viewpoint.

[0109] For example, such as Figure 2A As shown, a depth map of a 3D model D in viewpoint 1 can be generated. The depth map and prompt text are input into the image generation model to obtain style map 1 output by the image generation model.

[0110] (2) Generate a set of rendering images for each of the multiple perspectives.

[0111] In this embodiment, three colored 3D models of 3D model D can be generated in the following way: after normalizing the coordinates of each feature point in 3D model D, the normalized coordinate values ​​are mapped to color values. Mapping to color values ​​can be understood as coloring the 3D model D. For the specific coloring process, please refer to the aforementioned embodiment.

[0112] like Figure 2A As shown, there can be n viewpoints, and the specific viewpoints can be flexibly set; this embodiment does not limit this. For example, the viewpoints can be sampled uniformly in space based on the horizontal and vertical angles.

[0113] Obtain the set of rendered images from any viewpoint from viewpoint 1 to viewpoint n, including:

[0114] ① The grayscale rendering image obtained by performing grayscale rendering on the 3D model D from this perspective;

[0115] ② The rendered image x of the shaded 3D model x from this perspective; Since the shaded 3D model x is obtained through shading, the rendered image here can be understood as the image of the shaded 3D model from this perspective.

[0116] ③ The rendered image of the shaded 3D model y from this viewpoint;

[0117] ④ The rendered image of the shaded 3D model z from this perspective.

[0118] For example, Figure 2B The image shows a set of rendered images for viewpoint 1; Figure 2C The image shows a collection of rendered images from another perspective.

[0119] (3) Extract feature maps from the set of rendered images of any viewpoint from viewpoint 1 to viewpoint n.

[0120] In this embodiment, Figure 2A Viewpoint 1 in the example is used as the reference viewpoint; in practical applications, any of the n viewpoints can be used as the reference viewpoint, and this embodiment does not limit this. Each viewpoint corresponds to a set of rendered images, and feature maps can be extracted from the set of rendered images for each viewpoint using the method described in the previous embodiment.

[0121] (4) Train the neural network model.

[0122] like Figure 2A As shown, feature map 1 and style map 1 from the reference viewpoint can be used to construct training samples for training a neural network model. The constructed samples refer to matching pairs of "feature vectors and the corresponding pixel values ​​of the feature vectors in the style map". The input to the neural network model is the feature vector, and the output is the pixel value.

[0123] (5) Use a neural network model to output style maps from other perspectives.

[0124] For the sake of example, Figure 2A The specific processing flow is not shown. Taking viewpoint 2 as an example, an image of the same size as style map 1 can be created, which is referred to as the image to be assigned in this embodiment. The pixel values ​​of each pixel in the image to be assigned are not limited. Feature map 2 of viewpoint 2 also contains multiple feature vectors. Each feature vector is input into a trained neural network model to obtain the pixel value corresponding to the feature vector output by the neural network model. The pixel value is then assigned to the corresponding pixel in the image to be assigned, thus obtaining the assigned image.

[0125] The assigned image, the depth map of the target 3D model D at viewpoint 2, and the descriptive text of style map 1 (obtained by inputting style map 1 into the Blip model) are input into the image generation model, which then generates style map 2. Optionally, in practical applications, the weights of the image and descriptive text can be adjusted as needed.

[0126] Repeat the above process to perform the same processing on viewpoints 3 to n, and obtain style maps for the other viewpoints.

[0127] (6) Train the NeRF model.

[0128] The NeRF model is trained using style maps 1 to n from n viewpoints. Specifically, the viewpoint information (i.e., camera pose information) and style map of each viewpoint can be input into the NeRF model. The model can fit the results of the style maps under all viewpoints to obtain the trained NeRF model.

[0129] As can be seen from the above embodiments, in this embodiment, by mapping the coordinate values ​​of each axis of the feature points in the 3D model to color values ​​to generate three colored 3D models, the position information is encoded into the color information, so that the position information can be used as the basis for feature calculation, thereby improving the accuracy, robustness and ease of use of the method.

[0130] This embodiment uses a convolutional neural network to extract feature maps of the model from the rendering image set of each viewpoint. Therefore, it can incorporate multiple features and eliminates the need to manually construct high-order features, avoiding additional input. In addition, when using the neural network model to derive the pixel values ​​of the feature vectors in the feature maps of other views, it can comprehensively consider various factors rather than a single dimension such as lighting information, color values, color change gradients, and other manually set features.

[0131] When using a neural network model to derive the pixel values ​​of feature vectors in feature maps from other perspectives, it is equivalent to performing a search operation on the correspondence between each feature vector and pixel value in the feature map of the reference perspective. That is, searching for which feature vector in the feature map of other perspectives is the same as which feature vector in the feature map of the reference perspective. In this embodiment, a neural network model is used to implement this search operation, and the search time complexity is O(N), where N is the number of feature vectors in the feature map. It has very good performance on high-resolution images.

[0132] This embodiment uses an image generation model to generate final style maps for other viewpoints. This method avoids the step of finding image patches from the style map of the reference viewpoint in traditional image analogy, so that the style maps of other viewpoints are no longer limited to the possible patterns in the style map of the reference viewpoint, thus improving realism.

[0133] This embodiment does not require training the NeRF model based on a large pre-trained diffusion model. Instead, it directly uses style maps of each viewpoint with known camera positions as the training basis. Since the style maps of each viewpoint are generated from images under the guidance of the depth maps of each viewpoint, they contain depth information, which makes the convergence speed of training the NeRF model faster.

[0134] The NeRF model construction method of this embodiment can run on a computer device, including but not limited to servers, cloud servers, server clusters, tablet computers, personal digital assistants (PDAs), laptops, or desktop computers. In some examples, the model involved in the NeRF model construction method of this embodiment can be deployed on a server that interfaces with the user's client. The user's client can receive user input and send it to the server, which then runs the NeRF model construction method of this embodiment and sends the result back to the user's client. In other examples, the model involved in the NeRF model construction method of this embodiment can also be deployed on the user's client; this embodiment does not impose any limitations on this.

[0135] Corresponding to the aforementioned embodiments of the NeRF model construction method, this specification also provides embodiments of the NeRF model construction apparatus and the terminal to which it is applied.

[0136] The embodiments of the NeRF model construction apparatus described in this specification can be applied to computer devices, such as servers or terminal devices. The apparatus embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logically defined apparatus, it is formed by its processor reading the corresponding computer program instructions from non-volatile memory into memory and executing them. From a hardware perspective, such as... Figure 3 The diagram shown is a hardware structure diagram of a computer device on which the NeRF model is constructed, except for... Figure 3 In addition to the processor 310, memory 330, network interface 320, and non-volatile memory 340 shown, the computer device in which the NeRF model construction device 331 is located in the embodiment may also include other hardware depending on the actual function of the computer device, which will not be described in detail here.

[0137] like Figure 4 As shown, Figure 4 This is a block diagram illustrating an apparatus for constructing a NeRF model according to an exemplary embodiment of this specification, the apparatus comprising:

[0138] The acquisition module 41 acquires a depth map of the target 3D model from a reference viewpoint specified from multiple preset viewpoints; and a feature map corresponding to the reference viewpoint.

[0139] The style map generation module 42 inputs the depth map of the reference viewpoint and the prompt text input by the user into the image generation model, and obtains a style map generated by the image generation model that has the same depth information as the depth map and matches the style of the prompt text.

[0140] The neural network model processing module 43 constructs training samples based on the feature vectors contained in the feature map of the reference view and the pixel values ​​of the feature vectors at the corresponding positions in the style map, trains a preset neural network model based on the training samples, and generates style maps corresponding to each of the other views among the multiple views except the reference view based on the trained neural network model.

[0141] NeRF model training module 44 trains a NeRF model corresponding to the target 3D model based on the style map corresponding to each of the multiple perspectives.

[0142] In some examples, the style map corresponding to any of the other target perspectives in the neural network model processing module 43 is generated in the following way:

[0143] Obtain the feature map corresponding to the target viewpoint;

[0144] Each feature vector contained in the feature map corresponding to the target viewpoint is input into the trained neural network model, and the pixel values ​​corresponding to each feature vector generated by the neural network model are obtained.

[0145] Based on the obtained pixel values ​​corresponding to each feature vector, a style map corresponding to the target viewpoint is further generated.

[0146] In some examples, the step of further generating a style map corresponding to the target viewpoint based on the obtained pixel values ​​corresponding to each feature vector includes:

[0147] The image consisting of the pixel values ​​corresponding to each feature vector, the depth map of the target 3D model in the target viewpoint, and the descriptive text of the style map corresponding to the reference viewpoint are input into the image generation model. The model then generates a style map that has the same depth information as the depth map of the target viewpoint and matches the style indicated by the descriptive text.

[0148] In some examples, the feature vectors contained in the feature map of any of the multiple viewpoints include: feature vectors composed of feature values ​​obtained by feature mapping of the coordinate values ​​of feature points contained in the target 3D model.

[0149] In some examples, the feature map corresponding to any one of the multiple perspectives is obtained by stitching together the feature maps of each rendering map contained in the rendering map set corresponding to that perspective.

[0150] The set of rendering images corresponding to any one of the plurality of perspectives includes: a grayscale rendering image corresponding to the target 3D model under that perspective; and a rendering image corresponding to the tinted 3D model under that perspective; wherein the tinted 3D model is a 3D model obtained by mapping the coordinate values ​​of feature points contained in the target 3D model to color values.

[0151] In some examples, the shading 3D model includes: a shading 3D model corresponding to each of the three coordinate axes in the 3D coordinate system; the shading 3D model corresponding to any coordinate axis is a 3D model obtained by mapping the coordinate values ​​of each feature point in the target 3D model on that coordinate axis to color values;

[0152] The set of rendering images corresponding to any of the plurality of viewpoints includes the rendering images corresponding to the shaded 3D model under that viewpoint, including: the rendering images of the shaded 3D model corresponding to each of the three coordinate axes respectively.

[0153] In some examples, the shading 3D model is a 3D model obtained by normalizing the coordinate values ​​of each feature point contained in the target 3D model and mapping the normalized coordinate values ​​to color values.

[0154] In some examples, the target 3D model includes: a 3D model without color information obtained by removing color information from a user-input 3D model.

[0155] In some examples, the feature map corresponding to any one of the plurality of viewpoints is obtained in the following manner:

[0156] Each of the rendered images in the set of rendered images from any given viewpoint is input into a convolutional neural network model containing M convolutional layers. The feature maps extracted from each of the M convolutional layers are obtained, and the obtained feature maps are concatenated to obtain M tensors that correspond one-to-one with the M convolutional layers. The tensor corresponding to any of the M convolutional layers is obtained by concatenating the feature maps extracted from each of the input rendered images by that convolutional layer.

[0157] The obtained M tensors are concatenated to obtain a feature map corresponding to any one of the viewpoints.

[0158] In some examples, the tensor corresponding to any of the M convolutional layers is obtained by concatenating the feature maps extracted by that convolutional layer from each of the input rendered images along the channel dimension.

[0159] The concatenation of the obtained M tensors includes:

[0160] The obtained M tensors are concatenated along the channel dimension.

[0161] In some examples, the tensor corresponding to any of the M convolutional layers is obtained by concatenating the feature maps extracted by that convolutional layer from each of the input rendered images in the channel dimension after converting them to a preset size.

[0162] In some examples, the convolutional neural network model includes: the VGG convolutional neural network model.

[0163] In some examples, the image formed by the pixel values ​​corresponding to each feature vector is obtained by creating an image to be assigned the same size as the style map of the reference viewpoint, and then assigning pixel values ​​to the pixel positions in the image to be assigned that correspond to each feature vector based on the pixel values ​​corresponding to each feature vector.

[0164] In some examples, the descriptive text of the style map of the reference viewpoint is a descriptive text of the image style generated by the visual generative text model after the style map of the reference viewpoint is input into the pre-trained visual generative text model.

[0165] In some examples, the pre-trained visual text generation model includes the Blip model.

[0166] In some examples, the image generation model includes a diffusion model with a control network.

[0167] In some examples, the preset neural network model includes a multilayer perceptron.

[0168] The implementation process of the functions and roles of each module in the NeRF model construction device described above is detailed in the implementation process of the corresponding steps in the NeRF model construction method described above, and will not be repeated here.

[0169] Accordingly, this specification also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned NeRF model construction method embodiment.

[0170] Accordingly, embodiments of this specification also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of an embodiment of the NeRF model construction method.

[0171] Accordingly, embodiments of this specification also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of an embodiment of a method for constructing a NeRF model.

[0172] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0173] The above embodiments can be applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. The hardware of the computer device includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0174] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), game console, interactive network television (IPTV), smart wearable device, etc.

[0175] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0176] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0177] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0178] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0179] The terms "specific example" or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with the embodiments or examples, which are included in at least one embodiment or example of this specification. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0180] Other embodiments of this specification will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this specification are indicated by the following claims.

[0181] It should be understood that this specification is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this specification is limited only by the appended claims.

[0182] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A method for constructing a NeRF model, the method comprising: Obtain the depth map of the target 3D model from a reference viewpoint specified from multiple preset viewpoints; And, a feature map corresponding to the reference viewpoint; The depth map of the reference viewpoint and the prompt text input by the user are input into the image generation model, and a style map generated by the image generation model with the same depth information as the depth map and matching the style of the prompt text is obtained; Training samples are constructed based on the feature vectors contained in the feature map of the reference view and the pixel values ​​of the feature vectors at the corresponding positions in the style map. A preset neural network model is trained based on the training samples, and style maps corresponding to each of the other views in the plurality of views other than the reference view are generated based on the trained neural network model. A NeRF model corresponding to the target 3D model is trained based on the style map corresponding to each of the multiple perspectives.

2. The method according to claim 1, wherein the style map corresponding to any of the other target viewpoints is generated in the following manner: Obtain the feature map corresponding to the target viewpoint; Each feature vector contained in the feature map corresponding to the target viewpoint is input into the trained neural network model, and the pixel values ​​corresponding to each feature vector generated by the neural network model are obtained. Based on the obtained pixel values ​​corresponding to each feature vector, a style map corresponding to the target viewpoint is further generated.

3. The method according to claim 2, wherein generating a style map corresponding to the target viewpoint based on the obtained pixel values ​​corresponding to each feature vector comprises: The image consisting of the pixel values ​​corresponding to each feature vector, the depth map of the target 3D model in the target viewpoint, and the descriptive text of the style map corresponding to the reference viewpoint are input into the image generation model. The model then generates a style map that has the same depth information as the depth map of the target viewpoint and matches the style indicated by the descriptive text.

4. The method according to claim 2, wherein the feature vectors included in the feature map of any one of the plurality of viewpoints include: A feature vector is formed by feature values ​​obtained by feature mapping of the coordinate values ​​of feature points contained in the target 3D model.

5. The method according to claim 2, wherein the feature map corresponding to any one of the plurality of views is obtained by splicing the feature maps of each rendering map contained in the rendering map set corresponding to that view; The set of rendering images corresponding to any one of the plurality of viewpoints includes: a grayscale rendering image corresponding to the target 3D model under that viewpoint; and a rendering image corresponding to the shaded 3D model under that viewpoint; wherein... The shading 3D model is a 3D model obtained by mapping the coordinate values ​​of feature points contained in the target 3D model to color values.

6. The method according to claim 5, wherein the shaded 3D model comprises: A shaded 3D model corresponding to each of the three coordinate axes in a 3D coordinate system; A colored 3D model corresponding to any coordinate axis is a 3D model obtained by mapping the coordinate value of each feature point in the target 3D model on that coordinate axis to a color value. The set of rendering images corresponding to any of the plurality of viewpoints includes the rendering images corresponding to the shaded 3D model under that viewpoint, including: the rendering images of the shaded 3D model corresponding to each of the three coordinate axes respectively.

7. The method according to claim 5, wherein the colored 3D model is a 3D model obtained by normalizing the coordinate values ​​of each feature point contained in the target 3D model and mapping the normalized coordinate values ​​to color values.

8. The method according to claim 1, wherein the target three-dimensional model comprises: The 3D model obtained by removing color information from the user-input 3D model is a 3D model that does not carry color information.

9. The method according to claim 5, wherein the feature map corresponding to any one of the plurality of viewpoints is obtained in the following manner: Each rendered image in the set of rendered images from any given viewpoint is input into a convolutional neural network model containing M convolutional layers. The feature maps extracted from each of the M convolutional layers are then obtained, and these feature maps are concatenated to obtain M tensors corresponding one-to-one with the M convolutional layers. The tensor corresponding to any of the M convolutional layers is obtained by concatenating the feature maps extracted from each of the input rendered images by that convolutional layer. The obtained M tensors are concatenated to obtain a feature map corresponding to any one of the viewpoints.

10. The method according to claim 9, wherein the tensor corresponding to any one of the M convolutional layers is obtained by concatenating the feature maps extracted by that convolutional layer from each of the input rendering images along the channel dimension; The concatenation of the obtained M tensors includes: The obtained M tensors are concatenated along the channel dimension.

11. The method according to claim 10, wherein the tensor corresponding to any one of the M convolutional layers is obtained by concatenating the feature maps extracted from each of the input rendering images by that convolutional layer in the channel dimension after converting them to a preset size.

12. The method according to claim 9, wherein the convolutional neural network model comprises: VGG convolutional neural network model.

13. The method according to claim 3, wherein the image formed by the pixel values ​​corresponding to each feature vector is obtained by creating an image to be assigned the same size as the style map of the reference viewpoint, and then assigning pixel values ​​to the pixel positions in the image to be assigned that correspond to each feature vector based on the pixel values ​​corresponding to each feature vector.

14. The method according to claim 3, wherein the descriptive text of the style map of the reference viewpoint is a descriptive text of the image style generated by the visual generation text model after the style map of the reference viewpoint is input into a pre-trained visual generation text model.

15. The method according to claim 14, wherein the pre-trained visual text generation model comprises: Blip model.

16. The method according to claim 1, wherein the image generation model comprises: A diffusion model with a control network.

17. The method according to claim 1, wherein the preset neural network model comprises: Multilayer perceptron.

18. An apparatus for constructing a NeRF model, the apparatus comprising: The acquisition module acquires a depth map of the target 3D model from a specified reference viewpoint among multiple preset viewpoints. And, a feature map corresponding to the reference viewpoint; The style map generation module inputs the depth map of the reference viewpoint and the prompt text input by the user into the image generation model, and obtains a style map generated by the image generation model that has the same depth information as the depth map and matches the style of the prompt text. The neural network model processing module constructs training samples based on the feature vectors contained in the feature map of the reference view and the pixel values ​​of the feature vectors at the corresponding positions in the style map, trains a preset neural network model based on the training samples, and generates style maps corresponding to each of the other views among the multiple views except the reference view based on the trained neural network model. The NeRF model training module trains a NeRF model corresponding to the target 3D model based on the style map corresponding to each of the multiple perspectives.

19. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 17.

20. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any one of claims 1 to 17.

Images