Image processing method and device based on neural radiation field

Abstract

The embodiments of the present specification provide an image processing method and device based on a neural radiation field, wherein the image processing method based on a neural radiation field includes: determining sampling position information, sampling viewing angle information and a target object image in response to an image processing instruction; inputting the sampling position information, the sampling viewing angle information and the target object image into a pre-trained neural radiation field model for processing, and obtaining density information, color feature values ​​and signed distance function values ​​corresponding to each sampling point output by the neural radiation field model, wherein the neural radiation field model is a machine learning model; determining at least two sampling point subsets based on the signed distance function value corresponding to each sampling point; rendering and generating a target sub-object corresponding to each sampling point subset according to the density information and color feature values ​​of the sampling points in each sampling point subset, and generating a target object based on each target sub-object.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of image processing technology, and in particular to an image processing method based on neural radiation field. One or more embodiments of this specification also relate to an image processing device based on neural radiation field, a computing device, a computer-readable storage medium, and a computer program. Background Art

[0002] With the development of computer technology, the image processing technology has also made great progress in recent years. Among them, the field of synthesizing virtual objects from different perspectives is an important branch. Low-cost editing and synthesis of realistic virtual objects from different perspectives are very important for the field of 2D virtual object synthesis. 3D virtual object synthesis is a method of editing the encoding vector corresponding to the virtual object based on a neural network. In the process of 3D virtual object synthesis, it is necessary to ensure multi-perspective consistency and identity retention, that is, to ensure that the virtual object has the same structure at any perspective and does not change at different perspectives. For example, taking the face as an example, it is necessary to ensure that the generated face has the same structure at any perspective and is the same person at different perspectives. However, in the process of current model rendering to generate 3D virtual objects, the problem of inconsistent perspectives often occurs. Therefore, how to solve the problem of inconsistent perspectives has become a problem that technicians need to solve urgently. Summary of the invention

[0003] In view of this, an embodiment of this specification provides an image processing method based on neural radiation field. One or more embodiments of this specification also relate to a face generation method based on neural radiation field, an image processing device based on neural radiation field, a face generation device based on neural radiation field, a computing device, a computer-readable storage medium and a computer program to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, there is provided an image processing method based on a neural radiation field, comprising:

[0005] determining sampling position information, sampling viewing angle information, and a target object image in response to an image processing instruction;

[0006] Inputting the sampling position information, the sampling viewing angle information and the target object image into a pre-trained neural radiation field model for processing, and obtaining density information, color feature values ​​and signed distance function values ​​corresponding to each sampling point output by the neural radiation field model, wherein the neural radiation field model is a machine learning model;

[0007] Determining at least two subsets of sampling points based on the signed distance function value corresponding to each sampling point;

[0008] The target sub-object corresponding to each sampling point subset is generated by rendering according to the density information and color feature values ​​of the sampling points in each sampling point subset, and the target object is generated based on each target sub-object.

[0009] According to a second aspect of an embodiment of this specification, a method for generating a face based on a neural radiation field is provided, comprising:

[0010] Determining sampling position information, sampling viewing angle information and a face image in response to a face generation instruction;

[0011] Inputting the sampling position information, the sampling viewing angle information and the face image into a pre-trained neural radiation field model for processing, and obtaining density information, color feature values ​​and signed distance function values ​​corresponding to each sampling point output by the neural radiation field model, wherein the neural radiation field model is a machine learning model;

[0012] Determining at least two subsets of sampling points based on the signed distance function value corresponding to each sampling point;

[0013] Based on each sampling point subset, a face subset corresponding to each sampling point subset is generated by rendering, and a target face is generated based on each face subset.

[0014] According to a third aspect of an embodiment of this specification, there is provided an image processing device based on a neural radiation field, comprising:

[0015] a determination module configured to determine sampling position information, sampling viewing angle information and a target object image in response to an image processing instruction;

[0016] An input module is configured to input the sampling position information, the sampling viewing angle information and the target object image into a pre-trained neural radiation field model for processing, and obtain density information, color feature values ​​and signed distance function values ​​corresponding to each sampling point output by the neural radiation field model, wherein the neural radiation field model is a machine learning model;

[0017] A subset determination module is configured to determine at least two sampling point subsets based on the signed distance function value corresponding to each sampling point;

[0018] The rendering module is configured to render and generate a target sub-object corresponding to each sampling point subset according to density information and color feature values ​​of the sampling points in each sampling point subset, and generate a target object based on each target sub-object.

[0019] According to a fourth aspect of the embodiments of this specification, a human face generation device based on a neural radiation field is provided, comprising:

[0020] An image determination module, configured to determine sampling position information, sampling viewing angle information and a face image in response to a face generation instruction;

[0021] A model input module is configured to input the sampling position information, the sampling viewing angle information and the face image into a pre-trained neural radiation field model for processing, and obtain density information, color feature values ​​and signed distance function values ​​corresponding to each sampling point output by the neural radiation field model, wherein the neural radiation field model is a machine learning model;

[0022] A subset determination module is configured to determine at least two sampling point subsets based on the signed distance function value corresponding to each sampling point;

[0023] The face rendering module is configured to generate a face subset corresponding to each sampling point subset based on rendering of each sampling point subset, and generate a target face based on each face subset.

[0024] According to a fifth aspect of the embodiments of this specification, an augmented reality (AR) device or a virtual reality (VR) device is provided, including:

[0025] Memory, processor and display;

[0026] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the following steps are implemented:

[0027] determining sampling position information, sampling viewing angle information, and a target object image in response to an image processing instruction;

[0028] Inputting the sampling position information, the sampling viewing angle information and the target object image into a pre-trained neural radiation field model for processing, and obtaining density information, color feature values ​​and signed distance function values ​​corresponding to each sampling point output by the neural radiation field model, wherein the neural radiation field model is a machine learning model;

[0029] Determining at least two subsets of sampling points based on the signed distance function value corresponding to each sampling point;

[0030] Rendering generates a target sub-object corresponding to each sampling point subset according to density information and color feature values ​​of the sampling points in each sampling point subset, and generating a target object based on each target sub-object;

[0031] The target object is displayed through the display of the augmented reality (AR) device or the virtual reality (VR) device.

[0032] According to the sixth aspect of the embodiments of this specification, a computing device is provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein when the processor executes the computer instructions, the steps of the image processing method based on the neural radiation field or the face generation method based on the neural radiation field are implemented.

[0033] According to the seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer instructions. When the computer instructions are executed by a processor, the steps of the image processing method based on neural radiation field or the face generation method based on neural radiation field are implemented.

[0034] According to an eighth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned image processing method based on neural radiation field or the face generation method based on neural radiation field.

[0035] The neural radiation field-based image processing method provided in the present specification determines sampling position information, sampling viewing angle information and a target object image in response to an image processing instruction; inputs the sampling position information, the sampling viewing angle information and the target object image into a pre-trained neural radiation field model for processing, and obtains density information, color feature values ​​and signed distance function values ​​corresponding to each sampling point output by the neural radiation field model, wherein the neural radiation field model is a machine learning model; determines at least two sampling point subsets based on the signed distance function value corresponding to each sampling point; renders and generates a target sub-object corresponding to each sampling point subset according to the density information and color feature values ​​of the sampling points in each sampling point subset, and generates a target object based on each target sub-object.

[0036] An embodiment of the present specification realizes a method of integrating the signed distance function value into the neural radiation field model (NeRF model), obtaining the density information, color feature value and signed distance function value (SDF value) of the sampling point according to the sampling position information, sampling perspective information and the target object image, determining the various components of the virtual object by means of the SDF value set NeRF model, rendering each component separately, and then splicing the various components generated by rendering into the target virtual object. This method ensures the geometric properties of the neural network for 3D objects through density information and color feature values, determines the component corresponding to each sampling point through the SDF value, generates the various components of the virtual object, and ensures the superiority and consistency of the virtual object rendering effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of an image processing method based on neural radiation field provided by one embodiment of this specification;

[0038] Figure 2 is a schematic diagram of obtaining sampling position information and sampling viewing angle information provided by an embodiment of this specification;

[0039] Figure 3 is a schematic diagram of SDF value analysis of a virtual face provided by an embodiment of this specification;

[0040] Figure 4a is a flowchart of a processing process of a method for generating a face based on a neural radiation field provided by an embodiment of the present specification;

[0041] Figure 4b is a schematic diagram of a face subset provided by an embodiment of this specification;

[0042] Figure 5 is a schematic diagram of the structure of an image processing device based on neural radiation field provided by an embodiment of this specification;

[0043] Figure 6 It is a structural schematic diagram of a face generation device based on neural radiation field provided by an embodiment of this specification;

[0044] Figure 7 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0045] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0046] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to any or all possible combinations of one or more associated listed items.

[0047] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0048] First, the terms involved in one or more embodiments of this specification are explained.

[0049] Image Synthesis: A method of editing the encoding vector corresponding to a face based on a neural network.

[0050] Neural Radiance Field: A unique structure for volume rendering (3D rendering).

[0051] Semantic Segmentation: Separate the face according to hair, skin, background and other parts.

[0052] Multi-View Consistent: Ensures that the face has the same structure at any viewing angle, that is, excellent identity preservation.

[0053] Identity-Preservation: The ability of a person to remain unchanged under different perspectives.

[0054] In this specification, a method for image processing based on neural radiation field is provided. One or more embodiments of this specification also involve a method for generating a face based on neural radiation field, an image processing device based on neural radiation field, a face generating device based on neural radiation field, a computing device, a computer-readable storage medium and a computer program, which are described in detail one by one in the following embodiments.

[0055] Figure 1 A flowchart of an image processing method based on neural radiation field provided according to an embodiment of the present specification is shown, including steps 102 to 108.

[0056] Step 102: Determine sampling position information, sampling viewing angle information and a target object image in response to an image processing instruction.

[0057] Among them, the image processing instructions specifically refer to instructions for generating three-dimensional virtual objects. The image processing method based on neural radiation field provided in this specification adopts a neural radiation field model to ensure the geometric properties of the neural network for 3D objects and ensure the superiority and consistency of the effects of editing and synthesizing virtual objects.

[0058] The input of the neural radiation field model is only pictures from different perspectives. There is no need for camera parameters, ambient light, and many parts involved in the 3D Mesh-based rendering process: texture, lighting, material, camera position, etc.

[0059] The sampling position information and the sampling view angle information are input parameters for inputting into the neural radiance field model. Neural radiance fields (NeRF) are a model for rendering and generating three-dimensional scenes. Conventional NeRF is a three-dimensional scene representation, and it is an implicit scene representation. It cannot directly see a three-dimensional model. Therefore, in practical applications, NeRF is somewhat inconvenient to use. The image processing method based on neural radiation fields provided in this specification adopts a neural radiation field model to ensure the geometric properties of 3D objects by the neural network, and edit virtual objects through display synthesis.

[0060] NeRF is the abbreviation of Neural Radiance Fields. Radiance Fields refers to a function, or mapping g θ , specifically, NeRF can be expressed by the following formula 1:

[0061] (σ, c) = g θ (x, d) Formula 1

[0062] The input of the mapping is x and d, where x∈R 3 is the coordinate of the three-dimensional space point, that is, the sampling position information, d∈S 2 is the observation angle, that is, the sampling perspective information. The output of the mapping is σ and c, where σ∈R + is the density information, d∈S 3 , is the RGB color.

[0063] Optionally, determine the sampling position information and sampling viewing angle information, including:

[0064] Determine sampling position information on a preset hemispherical surface;

[0065] Corresponding sampling viewing angle information is determined based on the sampling position information.

[0066] See also Figure 2 , Figure 2A schematic diagram of obtaining sampling position information and sampling viewing angle information provided by an embodiment of the present specification is shown. The preset hemisphere specifically refers to a virtual hemisphere used to cover the target object image in NeRF. A sampling position information is randomly determined in the virtual hemisphere, that is, x in the above formula 1, and then the line of sight direction is constructed based on the sampling position information, that is, d in the above formula.

[0067] The target object image specifically refers to an image of a three-dimensional object that is to be rendered and generated in the embodiments of this specification. In this specification, the target object image is a picture of the same scene taken from different positions.

[0068] In a specific implementation provided in this specification, taking the target object image as an image of a vase as an example, the sampling position information x, the sampling viewing angle information d and the target object image P are determined in response to the image processing instruction.

[0069] Step 104: Input the sampling position information, the sampling viewing angle information and the target object image into a pre-trained neural radiation field model for processing, and obtain density information, color feature values ​​and signed distance function values ​​corresponding to each sampling point output by the neural radiation field model.

[0070] Among them, the neural radiation field model is a machine learning model. In the method provided in this specification, the neural radiation field model is different from the conventional neural radiation field model. On the basis of the conventional neural radiation field model outputting the sampling point density information σ and the color eigenvalue c, it also outputs the signed distance function value (SDF value). The signed distance function (sign distance function), referred to as SDF, can also be called an oriented distance function (oriented distance function), which determines the distance from a point to the boundary of the area in a finite area in space and defines the sign of the distance at the same time: the point is positive inside the boundary of the area, negative outside, and 0 when it is on the boundary. According to the SDF value corresponding to each sampling point, it can be determined which part of the virtual object each sampling point belongs to. See Figure 3 , Figure 3 FIG. 4 shows a schematic diagram of SDF value analysis of a virtual face provided in an embodiment of the present specification. Figure 3 As shown, the color of the upper left corner area is the face area, the color of the lower left corner area is the hair area, the color of the lower right corner area is the background area, and the color of the upper right corner area is undefined.

[0071] In the method provided in this specification, the neural radiation field model not only outputs the density information and color feature value information of the conventional neural radiation field model, but also outputs the SDF value, based on which the area corresponding to each sampling point can be determined. This facilitates the subsequent splitting of the virtual object according to the SDF value of each sampling point, rendering it separately, and then merging it.

[0072] Specifically, the sampling position information, the sampling viewing angle information and the target object image are input into a pre-trained neural radiation field model for processing, including:

[0073] The sampling position information, the sampling viewing angle information and the target object image are input into a pre-trained neural radiation field model;

[0074] The neural radiation field model emits sampling point rays to sampling points of the target object image based on the sampling position information and the sampling viewing angle information;

[0075] The density information, color feature value and signed distance function value of each sampling point are determined based on the sampling point ray corresponding to each sampling point.

[0076] In practical applications, the NeRF provided in this specification uses ray casting technology to emit sampling point rays from the sampling position information based on the sampling view information to the sampling points of the target object image, and the sampling points may be pixel points in the target object image.

[0077] Ray casting is a relatively simple implementation method of volumetric rendering. Volumetric rendering refers to a rendering method that draws the projection on a 2D plane based on volume data (3D texture). Ray casting starts from the camera and directs rays to the 3D volume texture, but this only determines the direction of the light. In fact, the starting position of the light is the front intersection of the light and the volume texture, and the end position is the back intersection of the volume texture.

[0078] In the process of obtaining the density information, color feature value and SDF value of each sampling point, in addition to using ray casting technology, ray marching technology is also used. Ray marching is a general simulation method. Its rays start along the starting point, stop at a certain interval to do a calculation, and then continue to move forward. The time interval can be fixed or not.

[0079] The density information, color feature value and SDF value of each sampling point can be obtained through the ray casting technology and ray aggregation technology of the NeRF model.

[0080] In a specific implementation provided in the present specification, taking the target object image as an image of a vase as an example, the sampling position information x, the sampling viewing angle information d and the target object image P are input into a pre-trained neural radiation field model to obtain the density information σ, color eigenvalue c and signed distance function value SDF value corresponding to each sampling point output by the neural radiation field model.

[0081] Step 106: Determine at least two sampling point subsets based on the signed distance function value corresponding to each sampling point.

[0082] In the above steps, the SDF value of each sampling point is obtained, and the SDF value corresponding to each sampling point can be used to determine which sampling point subset each sampling point belongs to. Specifically, a virtual object can be composed of multiple parts, for example, a head photo of a person can be divided into face, hair, and background; a vase can be divided into vase body, flower, background, etc. The SDF value of each sampling point can be used to determine which part each sampling point belongs to.

[0083] Specifically, determining at least two sampling point subsets based on the signed distance function value corresponding to each sampling point includes:

[0084] Determine a preset signed distance function value interval corresponding to each sampling point subset;

[0085] A sampling point subset corresponding to each sampling point is determined based on the signed distance function value corresponding to each sampling point and each preset signed distance function value interval.

[0086] The preset signed distance function value interval specifically refers to the interval of SDF values ​​corresponding to each sampling point in the components of the virtual object, and the preset signed distance function value interval corresponding to each component is determined according to the SDF values ​​of the sampling points in each component of the virtual object.

[0087] According to the SDF value corresponding to each sampling point, it is determined which component of the virtual object each sampling point belongs to, for example Figure 3 As shown in the figure. The sampling points with SDF values ​​in the upper left corner belong to the face part, the sampling points with SDF values ​​in the lower left corner belong to the hair part, and the sampling points with SDF values ​​in the lower right corner belong to the background part. Figure 3 The image shown includes three sampling point subsets, namely, a face subset, a hair subset and a background subset.

[0088] Step 108: generating a target sub-object corresponding to each sampling point subset by rendering according to the density information and color feature values ​​of the sampling points in each sampling point subset, and generating a target object based on each target sub-object.

[0089] After the sampling point subsets are divided, volumetric rendering can be performed on each sampling point subset according to the density information and color feature values ​​of the sampling points in each sampling point subset to obtain the components of the virtual object corresponding to each sampling point subset.

[0090] Still Figure 3 For example, the sampling points in the face subset are volume rendered according to the density information and color feature values ​​to obtain the face part; the sampling points in the hair subset are volume rendered according to the density information and color feature values ​​to obtain the hair part; the sampling points in the background subset are volume rendered according to the density information and color feature values ​​to obtain the background part.

[0091] After obtaining multiple target sub-objects, the target sub-objects are synthesized to obtain the target object.

[0092] In practical applications, the target sub-object corresponding to each sampling point subset is generated by rendering according to the density information and color feature value of the sampling points in each sampling point subset, including:

[0093] Select a subset of target sampling points;

[0094] A target sub-object corresponding to the target sampling point subset is generated by rendering based on the density information and the color feature value of each sampling point in the target sampling point subset.

[0095] Each component corresponding to the virtual object is rendered separately. In practical applications, one of the sampling point subsets is used as an example for explanation, that is, a target sampling point subset is determined from multiple sampling point subsets, and the target sub-object corresponding to the target sampling point subset is rendered and generated according to the density information and color feature value of each sampling point in the target sampling point subset. By performing volume rendering separately, the components of the virtual object can be explicitly reflected. When multiple virtual objects are subsequently combined, multiple target sub-objects can be combined separately. Generate multiple combinations to improve the style of the virtual object obtained by rendering. For example, taking the rendering of a face as an example, the face a1 and hairstyle a2 of character A can be rendered, and the face b1 and hairstyle b2 of character B can be rendered. The face a1 and hairstyle b2 can also be combined to generate character C, and the face b1 and hairstyle a2 can be combined to generate character D.

[0096] In a specific implementation provided in this specification, generating a target object based on each target sub-object includes:

[0097] generating an initial target object based on each target sub-object;

[0098] The resolution of the initial target object is improved to obtain the target object.

[0099] In practical applications, after obtaining each target sub-object, each target sub-object can be spliced ​​to generate an initial target object. The resolution of the initial target object is usually low, and the user viewing experience is poor. Therefore, it is necessary to improve the resolution of the initial target object according to demand, that is, to improve the resolution of the initial target object, so as to obtain the target object. Specifically, improving the resolution of the initial target object to obtain the target object includes:

[0100] Inputting the initial target object into a resolution enhancement model;

[0101] A target object corresponding to the initial target object output by the resolution enhancement model is obtained.

[0102] In practical applications, the initial target image can be input into the encoder (resolution enhancement model) using StyleGan for resolution enhancement, thereby obtaining a high-resolution target object output by the resolution enhancement model. For example, the resolution of the initial target object is 64*64, and after being processed by the resolution enhancement model, a target object with a resolution of 512*512 can be obtained.

[0103] In a specific implementation provided in this specification, the neural radiation field model is trained by the following steps:

[0104] Obtaining sample sampling position, sample sampling viewing angle information and sample object image;

[0105] Input the sample sampling position, the sample sampling viewing angle information and the sample object image into the neural radiation field model, and obtain the density information, color feature value and signed distance function value corresponding to each sample sampling point output by the neural radiation field model;

[0106] Inputting the sample sampling position, the sample sampling viewing angle information and the sample object image into an object generation model to obtain a sample object output by the object generation model;

[0107] Determine at least two sample sampling point subsets according to density information, color feature values, and signed distance function values ​​corresponding to each sample sampling point, render and generate prediction sub-objects corresponding to each sample sampling point subset, and generate prediction objects based on each prediction sub-object;

[0108] Performing image segmentation on the sample object to obtain sample sub-objects corresponding to the sample object;

[0109] Calculate a first loss value according to the prediction sub-object and the sample sub-object, and calculate a second loss value according to the prediction object and the sample object;

[0110] The model parameters of the neural radiation field model are adjusted based on the first loss value and the second loss value, and the training is continued until the model training stop condition is reached.

[0111] In practical applications, the neural radiation field model provided in the embodiments of this specification is different from the conventional neural radiation field model. In addition to outputting density information and color feature values, the neural radiation field model provided in the embodiments of this specification also outputs a signed distance function value.

[0112] During the model training process, the sample sampling position, sample sampling viewing angle information and sample object image are input into the neural radiation field model to be trained, and the density information, color feature value and signed distance function value corresponding to each sample sampling point output by the neural radiation field model are obtained. At least two sample sampling point subsets are determined according to the density information, color feature value and signed distance function value corresponding to each sample sampling point, and the prediction sub-object corresponding to each sample sampling point subset is rendered and generated, and the prediction object is generated based on each prediction sub-object.

[0113] The sample sampling position, the sample sampling perspective information and the sample object image are input into the object generation model to obtain the sample object output by the object generation model, wherein the object generation model is a conventional NeRF model. After obtaining the sample object, the sample object is image segmented to obtain a plurality of sample sub-objects corresponding to the sample object.

[0114] At this point, the first loss value is calculated according to the prediction sub-object and the sample sub-object, the second loss value is calculated according to the prediction object and the sample object, and the model parameters of the neural radiation field model to be trained are jointly trained according to the first loss value and the second loss value until the model training stop condition is reached. The training stop condition of the model can be that both the first loss value and the second loss value are lower than the preset threshold, or the training round of the model training reaches the preset training round.

[0115] In practical applications, the predicted objects generated by the neural radiation field model to be trained may differ greatly from the real objects at the initial stage of training. For example, taking the generation of face images as an example, the face images generated by the neural radiation field model to be trained at the beginning of training may differ greatly from the real faces and need to be corrected. Based on this, the method further includes:

[0116] Get the reference object;

[0117] Performing image segmentation on the reference object to obtain a reference sub-object corresponding to the reference object;

[0118] Calculate a third loss value according to the reference sub-object and the predicted sub-object;

[0119] The model parameters of the neural radiation field model are adjusted according to the third loss value.

[0120] Among them, the reference object is a real virtual object, the reference object is segmented, the reference object is divided into multiple reference sub-objects, the third loss value is calculated between the reference sub-object and the predicted sub-object, and the model parameters of the neural radiation field model to be trained are adjusted in combination with the third loss value.

[0121] The neural radiation field-based image processing method provided in the present specification determines sampling position information, sampling viewing angle information and a target object image in response to an image processing instruction; inputs the sampling position information, the sampling viewing angle information and the target object image into a pre-trained neural radiation field model for processing, and obtains density information, color feature values ​​and signed distance function values ​​corresponding to each sampling point output by the neural radiation field model, wherein the neural radiation field model is a machine learning model; determines at least two sampling point subsets based on the signed distance function value corresponding to each sampling point; renders and generates a target sub-object corresponding to each sampling point subset according to the density information and color feature values ​​of the sampling points in each sampling point subset, and generates a target object based on each target sub-object.

[0122] An embodiment of the present specification realizes a method of integrating the signed distance function value into the neural radiation field model (NeRF model), obtaining the density information, color feature value and signed distance function value (SDF value) of the sampling point according to the sampling position information, sampling perspective information and the target object image, determining the various components of the virtual object by means of the SDF value set NeRF model, rendering each component separately, and then splicing the various components generated by rendering into the target virtual object. This method ensures the geometric properties of the neural network for 3D objects through density information and color feature values, determines the component corresponding to each sampling point through the SDF value, generates the various components of the virtual object, and ensures the superiority and consistency of the virtual object rendering effect.

[0123] Secondly, the resolution of the target object can be improved by using SytleGan’s resolution enhancement model.

[0124] The following combination Figure 4a Taking the application of the image processing method based on neural radiation field provided in this specification in rendering human face as an example, the image processing method based on neural radiation field is further described. Figure 4a A flowchart of a processing procedure of a method for generating a face based on a neural radiation field provided in an embodiment of the present specification is shown, and the specific steps include steps 402 to 408.

[0125] Step 402: Determine sampling position information, sampling viewing angle information and a face image in response to a face generation instruction.

[0126] Step 404: Input the sampling position information, the sampling viewing angle information and the face image into a pre-trained neural radiation field model for processing, and obtain density information, color feature values ​​and signed distance function values ​​corresponding to each sampling point output by the neural radiation field model.

[0127] Among them, the neural radiation field model is a machine learning model.

[0128] Step 406: Determine at least two sampling point subsets based on the signed distance function value corresponding to each sampling point.

[0129] Step 408: Generate a face subset corresponding to each sampling point subset by rendering based on each sampling point subset, and generate a target face based on each face subset.

[0130] See also Figure 4b , Figure 4b FIG. 1 shows a schematic diagram of a face subset provided in an embodiment of the present specification, such as Figure 4b As shown, the left part is the background set, the middle part is the face set, and the right part is the hair set.

[0131] Optionally, determine the sampling position information and sampling viewing angle information, including:

[0132] Determine sampling position information on a preset hemispherical surface;

[0133] Corresponding sampling viewing angle information is determined based on the sampling position information.

[0134] Optionally, the sampling position information, the sampling viewing angle information and the face image are input into a pre-trained neural radiation field model for processing, including:

[0135] The sampling position information, the sampling viewing angle information and the face image are input into a pre-trained neural radiation field model;

[0136] The neural radiation field model emits sampling point rays to the sampling points of the face image based on the sampling position information and the sampling viewing angle information;

[0137] The density information, color feature value and signed distance function value of each sampling point are determined based on the sampling point ray corresponding to each sampling point.

[0138] Optionally, determining at least two sampling point subsets based on the signed distance function value corresponding to each sampling point includes:

[0139] Determine a preset signed distance function value interval corresponding to each sampling point subset;

[0140] A sampling point subset corresponding to each sampling point is determined based on the signed distance function value corresponding to each sampling point and each preset signed distance function value interval.

[0141] Optionally, generating a face subset corresponding to each sampling point subset by rendering based on each sampling point subset includes:

[0142] Select a subset of target sampling points;

[0143] The target human face sub-object corresponding to the target sampling point subset is generated by rendering based on the density information and the color feature value of each sampling point in the target sampling point subset.

[0144] Optionally, generate a target face based on each face subset, including:

[0145] Generate an initial face based on each target face sub-object;

[0146] The resolution of the initial face is improved to obtain a target face.

[0147] Optionally, improving the resolution of the initial face to obtain a target face includes:

[0148] Inputting the initial human face into a resolution enhancement model;

[0149] Obtain a target face corresponding to the initial face output by the resolution enhancement model.

[0150] Optionally, the neural radiation field model is trained by the following steps:

[0151] Obtaining sample sampling position, sample sampling viewing angle information and sample face image;

[0152] Input the sample sampling position, the sample sampling viewing angle information and the sample face image into the neural radiation field model, and obtain the density information, color feature value and signed distance function value corresponding to each sample sampling point output by the neural radiation field model;

[0153] Inputting the sample sampling position, the sample sampling viewing angle information and the sample face image into a face generation model to obtain a sample face output by the face generation model;

[0154] Determine at least two sample sampling point subsets according to density information, color feature values, and signed distance function values ​​corresponding to each sample sampling point, render and generate predicted face sub-objects corresponding to each sample sampling point subset, and generate predicted faces based on each predicted face sub-object;

[0155] Performing image segmentation on the sample face to obtain a sample face sub-object corresponding to the sample face;

[0156] Calculate a first loss value according to the predicted face sub-object and the sample face sub-object, and calculate a second loss value according to the predicted face and the sample face;

[0157] The model parameters of the neural radiation field model are adjusted based on the first loss value and the second loss value, and the training is continued until the model training stop condition is reached.

[0158] Optionally, after rendering and generating the predicted face sub-object corresponding to each sample sampling point subset, the method further includes:

[0159] Get a reference face;

[0160] Performing image segmentation on the reference face to obtain a reference face sub-object corresponding to the reference face;

[0161] Calculating a third loss value according to the reference face sub-object and the predicted face sub-object;

[0162] The model parameters of the neural radiation field model are adjusted according to the third loss value.

[0163] The face generation method based on neural radiation field provided in this specification realizes the method of integrating the signed distance function value into the neural radiation field model (NeRF model), obtaining the density information, color feature value and signed distance function value (SDF value) of the sampling point according to the sampling position information, sampling perspective information and the target object image, determining the various components of the face by means of the SDF value set NeRF model, rendering each component separately, and then splicing the rendered components into a face image. This method ensures the geometric properties of the neural network for 3D objects through density information and color feature values, determines the component corresponding to each sampling point through the SDF value, generates the various components of the face image, and ensures the superiority and consistency of the virtual object rendering effect.

[0164] Secondly, the resolution of the target object can be improved by using SytleGan’s resolution enhancement model.

[0165] Corresponding to the above-mentioned image processing method embodiment based on neural radiation field, this specification also provides an image processing device embodiment based on neural radiation field, Figure 5 FIG. 2 shows a schematic diagram of a structure of an image processing device based on a neural radiation field provided by an embodiment of the present specification. Figure 5 As shown, the device comprises:

[0166] A determination module 502, configured to determine sampling position information, sampling viewing angle information and a target object image in response to an image processing instruction;

[0167] An input module 504 is configured to input the sampling position information, the sampling viewing angle information and the target object image into a pre-trained neural radiation field model for processing, and obtain density information, color feature value and signed distance function value corresponding to each sampling point output by the neural radiation field model, wherein the neural radiation field model is a machine learning model;

[0168] The subset determination module 506 is configured to determine at least two sampling point subsets based on the signed distance function value corresponding to each sampling point;

[0169] The rendering module 508 is configured to render and generate a target sub-object corresponding to each sampling point subset according to the density information and color feature values ​​of the sampling points in each sampling point subset, and generate a target object based on each target sub-object.

[0170] Optionally, the determining module 502 is further configured to:

[0171] Determine sampling position information on a preset hemispherical surface;

[0172] Corresponding sampling viewing angle information is determined based on the sampling position information.

[0173] Optionally, the input module 504 is further configured to:

[0174] The sampling position information, the sampling viewing angle information and the target object image are input into a pre-trained neural radiation field model;

[0175] The neural radiation field model emits sampling point rays to sampling points of the target object image based on the sampling position information and the sampling viewing angle information;

[0176] The density information, color feature value and signed distance function value of each sampling point are determined based on the sampling point ray corresponding to each sampling point.

[0177] Optionally, the subset determination module 506 is further configured to:

[0178] Determine a preset signed distance function value interval corresponding to each sampling point subset;

[0179] A sampling point subset corresponding to each sampling point is determined based on the signed distance function value corresponding to each sampling point and each preset signed distance function value interval.

[0180] Optionally, the rendering module 508 is further configured to:

[0181] Select a subset of target sampling points;

[0182] A target sub-object corresponding to the target sampling point subset is generated by rendering based on the density information and the color feature value of each sampling point in the target sampling point subset.

[0183] Optionally, the rendering module 508 is further configured to:

[0184] generating an initial target object based on each target sub-object;

[0185] The resolution of the initial target object is improved to obtain the target object.

[0186] Optionally, the rendering module 508 is further configured to:

[0187] Inputting the initial target object into a resolution enhancement model;

[0188] A target object corresponding to the initial target object output by the resolution enhancement model is obtained.

[0189] Optionally, the device further comprises a training module configured to:

[0190] Obtaining sample sampling position, sample sampling viewing angle information and sample object image;

[0191] Input the sample sampling position, the sample sampling viewing angle information and the sample object image into the neural radiation field model, and obtain the density information, color feature value and signed distance function value corresponding to each sample sampling point output by the neural radiation field model;

[0192] Inputting the sample sampling position, the sample sampling viewing angle information and the sample object image into an object generation model to obtain a sample object output by the object generation model;

[0193] Determine at least two sample sampling point subsets according to density information, color feature values, and signed distance function values ​​corresponding to each sample sampling point, render and generate prediction sub-objects corresponding to each sample sampling point subset, and generate prediction objects based on each prediction sub-object;

[0194] Performing image segmentation on the sample object to obtain sample sub-objects corresponding to the sample object;

[0195] Calculate a first loss value according to the prediction sub-object and the sample sub-object, and calculate a second loss value according to the prediction object and the sample object;

[0196] The model parameters of the neural radiation field model are adjusted based on the first loss value and the second loss value, and the training is continued until the model training stop condition is reached.

[0197] Optionally, the training module is further configured to:

[0198] Get the reference object;

[0199] Performing image segmentation on the reference object to obtain a reference sub-object corresponding to the reference object;

[0200] Calculate a third loss value according to the reference sub-object and the predicted sub-object;

[0201] The model parameters of the neural radiation field model are adjusted according to the third loss value.

[0202] The image processing device based on neural radiation field provided in the present specification determines sampling position information, sampling viewing angle information and target object image in response to image processing instructions; inputs the sampling position information, the sampling viewing angle information and the target object image into a pre-trained neural radiation field model for processing, and obtains density information, color feature values ​​and signed distance function values ​​corresponding to each sampling point output by the neural radiation field model, wherein the neural radiation field model is a machine learning model; determines at least two sampling point subsets based on the signed distance function value corresponding to each sampling point; renders and generates target sub-objects corresponding to each sampling point subset according to the density information and color feature values ​​of the sampling points in each sampling point subset, and generates a target object based on each target sub-object.

[0203] An embodiment of the present specification realizes a method of integrating the signed distance function value into the neural radiation field model (NeRF model), obtaining the density information, color feature value and signed distance function value (SDF value) of the sampling point according to the sampling position information, sampling perspective information and the target object image, determining the various components of the virtual object by means of the SDF value set NeRF model, rendering each component separately, and then splicing the various components generated by rendering into the target virtual object. This method ensures the geometric properties of the neural network for 3D objects through density information and color feature values, determines the component corresponding to each sampling point through the SDF value, generates the various components of the virtual object, and ensures the superiority and consistency of the virtual object rendering effect.

[0204] Secondly, the resolution of the target object can be improved by using SytleGan’s resolution enhancement model.

[0205] The above is a schematic scheme of an image processing device based on neural radiation field of this embodiment. It should be noted that the technical scheme of the image processing device based on neural radiation field and the technical scheme of the image processing method based on neural radiation field belong to the same concept, and the details not described in detail in the technical scheme of the image processing device based on neural radiation field can be referred to the description of the technical scheme of the image processing method based on neural radiation field.

[0206] Corresponding to the above-mentioned face generation method embodiment based on neural radiation field, this specification also provides a face generation device embodiment based on neural radiation field, Figure 6 FIG. 1 shows a schematic diagram of a face generation device based on a neural radiation field provided in an embodiment of the present specification. Figure 6 As shown, the device comprises:

[0207] An image determination module 602 is configured to determine sampling position information, sampling viewing angle information and a face image in response to a face generation instruction;

[0208] The model input module 604 is configured to input the sampling position information, the sampling viewing angle information and the face image into a pre-trained neural radiation field model for processing, and obtain density information, color feature value and signed distance function value corresponding to each sampling point output by the neural radiation field model, wherein the neural radiation field model is a machine learning model;

[0209] The subset determination module 606 is configured to determine at least two sampling point subsets based on the signed distance function value corresponding to each sampling point;

[0210] The face rendering module 608 is configured to generate a face subset corresponding to each sampling point subset based on rendering of each sampling point subset, and generate a target face based on each face subset.

[0211] Optionally, the image determination module 602 is further configured to:

[0212] Determine sampling position information on a preset hemispherical surface;

[0213] Corresponding sampling viewing angle information is determined based on the sampling position information.

[0214] Optionally, the model input module 604 is further configured to:

[0215] The sampling position information, the sampling viewing angle information and the face image are input into a pre-trained neural radiation field model;

[0216] The neural radiation field model emits sampling point rays to the sampling points of the face image based on the sampling position information and the sampling viewing angle information;

[0217] The density information, color feature value and signed distance function value of each sampling point are determined based on the sampling point ray corresponding to each sampling point.

[0218] Optionally, the subset determination module 606 is further configured to:

[0219] Determine a preset signed distance function value interval corresponding to each sampling point subset;

[0220] A sampling point subset corresponding to each sampling point is determined based on the signed distance function value corresponding to each sampling point and each preset signed distance function value interval.

[0221] Optionally, the face rendering module 608 is further configured to:

[0222] Select a subset of target sampling points;

[0223] The target human face sub-object corresponding to the target sampling point subset is generated by rendering based on the density information and the color feature value of each sampling point in the target sampling point subset.

[0224] Optionally, the face rendering module 608 is further configured to:

[0225] Generate an initial face based on each target face sub-object;

[0226] The resolution of the initial face is improved to obtain a target face.

[0227] Optionally, the face rendering module 608 is further configured to:

[0228] Inputting the initial human face into a resolution enhancement model;

[0229] Obtain a target face corresponding to the initial face output by the resolution enhancement model.

[0230] Optionally, the device further includes a model training module configured to:

[0231] Obtaining sample sampling position, sample sampling viewing angle information and sample face image;

[0232] Input the sample sampling position, the sample sampling viewing angle information and the sample face image into the neural radiation field model, and obtain the density information, color feature value and signed distance function value corresponding to each sample sampling point output by the neural radiation field model;

[0233] Inputting the sample sampling position, the sample sampling viewing angle information and the sample face image into a face generation model to obtain a sample face output by the face generation model;

[0234] Determine at least two sample sampling point subsets according to density information, color feature values, and signed distance function values ​​corresponding to each sample sampling point, render and generate predicted face sub-objects corresponding to each sample sampling point subset, and generate predicted faces based on each predicted face sub-object;

[0235] Performing image segmentation on the sample face to obtain a sample face sub-object corresponding to the sample face;

[0236] Calculate a first loss value according to the predicted face sub-object and the sample face sub-object, and calculate a second loss value according to the predicted face and the sample face;

[0237] The model parameters of the neural radiation field model are adjusted based on the first loss value and the second loss value, and the training is continued until the model training stop condition is reached.

[0238] Optionally, the model training module is further configured as follows:

[0239] Get a reference face;

[0240] Performing image segmentation on the reference face to obtain a reference face sub-object corresponding to the reference face;

[0241] Calculating a third loss value according to the reference face sub-object and the predicted face sub-object;

[0242] The model parameters of the neural radiation field model are adjusted according to the third loss value.

[0243] The face generation device based on neural radiation field provided in this specification realizes the method of integrating the signed distance function value into the neural radiation field model (NeRF model), obtaining the density information, color feature value and signed distance function value (SDF value) of the sampling point according to the sampling position information, sampling perspective information and the target object image, determining the various components of the face by means of the SDF value set NeRF model, rendering each component separately, and then splicing the rendered components into a face image. This method ensures the geometric properties of the neural network for 3D objects through density information and color feature values, determines the component corresponding to each sampling point through the SDF value, generates the various components of the face image, and ensures the superiority and consistency of the virtual object rendering effect.

[0244] Secondly, the resolution of the target object can be improved by using SytleGan’s resolution enhancement model.

[0245] The above is a schematic scheme of a face generation device based on neural radiation field of this embodiment. It should be noted that the technical scheme of the face generation device based on neural radiation field and the technical scheme of the face generation method based on neural radiation field belong to the same concept. For details not described in detail in the technical scheme of the face generation device based on neural radiation field, please refer to the description of the technical scheme of the face generation method based on neural radiation field.

[0246] Figure 7 The structure block diagram of a computing device 700 provided according to an embodiment of the present specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0247] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0248] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0249] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 700 may also be a mobile or stationary server.

[0250] Among them, when the processor 720 executes the computer instructions, it implements the steps of the image processing method based on neural radiation field or the face generation method based on neural radiation field.

[0251] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the image processing method based on neural radiation field or the face generation method based on neural radiation field belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the image processing method based on neural radiation field or the face generation method based on neural radiation field.

[0252] An embodiment of the present specification further provides an augmented reality (AR) device or a virtual reality (VR) device, including:

[0253] Memory, processor and display;

[0254] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the following steps are implemented:

[0255] determining sampling position information, sampling viewing angle information, and a target object image in response to an image processing instruction;

[0256] Inputting the sampling position information, the sampling viewing angle information and the target object image into a pre-trained neural radiation field model for processing, and obtaining density information, color feature values ​​and signed distance function values ​​corresponding to each sampling point output by the neural radiation field model, wherein the neural radiation field model is a machine learning model;

[0257] Determining at least two subsets of sampling points based on the signed distance function value corresponding to each sampling point;

[0258] Rendering generates a target sub-object corresponding to each sampling point subset according to density information and color feature values ​​of the sampling points in each sampling point subset, and generating a target object based on each target sub-object;

[0259] The target object is displayed through the display of the augmented reality (AR) device or the virtual reality (VR) device.

[0260] The above is a schematic scheme of an augmented reality AR device or a virtual reality VR device of this embodiment. It should be noted that the technical scheme of the augmented reality AR device or the virtual reality VR device belongs to the same concept as the technical scheme of the image processing method based on neural radiation field or the human face generation method based on neural radiation field. For the details not described in detail in the technical scheme of the augmented reality AR device or the virtual reality VR device, please refer to the description of the technical scheme of the image processing method based on neural radiation field or the human face generation method based on neural radiation field.

[0261] An embodiment of the present specification also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the image processing method based on neural radiation field or the face generation method based on neural radiation field as described above.

[0262] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the image processing method based on neural radiation field or the face generation method based on neural radiation field belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the image processing method based on neural radiation field or the face generation method based on neural radiation field.

[0263] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned image processing method based on neural radiation field or the face generation method based on neural radiation field.

[0264] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the image processing method based on neural radiation field or the human face generation method based on neural radiation field belong to the same concept, and the details not described in detail in the technical scheme of the computer program can be found in the description of the technical scheme of the image processing method based on neural radiation field or the human face generation method based on neural radiation field.

[0265] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0266] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0267] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0268] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0269] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. An image processing method based on neural radiation field, comprising: determining sampling position information, sampling viewing angle information, and a target object image in response to an image processing instruction; The sampling position information, the sampling viewing angle information and the target object image are input into a pre-trained neural radiation field model; the neural radiation field model emits sampling point rays to the sampling points of the target object image based on the sampling position information and the sampling viewing angle information; the density information, color feature value and signed distance function value of each sampling point are determined based on the sampling point rays corresponding to each sampling point, wherein the neural radiation field model is trained by the following steps: the sample sampling position, the sample sampling viewing angle information and the sample object image are input into the neural radiation field model to obtain the density information, color feature value and signed distance function value corresponding to each sample sampling point output by the neural radiation field model; the sample sampling position, the sample sampling viewing angle information and the sample object image are input into the object generation model to obtain the sample object output by the object generation model; at least two sample sampling point subsets are determined according to the density information, color feature value and signed distance function value corresponding to each sample sampling point, and a predicted sub-object corresponding to each sample sampling point subset is rendered and generated, and a predicted object is generated based on each predicted sub-object; Performing image segmentation on the sample object to obtain a sample sub-object corresponding to the sample object; training the neural radiation field model according to the predicted sub-object and the sample sub-object, and the predicted object and the sample object; Determining at least two subsets of sampling points based on the signed distance function value corresponding to each sampling point; The target sub-object corresponding to each sampling point subset is generated by rendering according to the density information and color feature values ​​of the sampling points in each sampling point subset, and the target object is generated based on each target sub-object.

2. The method according to claim 1, determining the sampling position information and the sampling viewing angle information, comprising: Determine sampling position information on a preset hemispherical surface; Corresponding sampling viewing angle information is determined based on the sampling position information.

3. The method of claim 1, wherein determining at least two subsets of sampling points based on the signed distance function value corresponding to each sampling point comprises: Determine a preset signed distance function value interval corresponding to each sampling point subset; A sampling point subset corresponding to each sampling point is determined based on the signed distance function value corresponding to each sampling point and each preset signed distance function value interval.

4. The method according to claim 1, generating a target sub-object corresponding to each sampling point subset according to density information and color feature values ​​of the sampling points in each sampling point subset by rendering, comprising: Select a subset of target sampling points; A target sub-object corresponding to the target sampling point subset is generated by rendering based on the density information and the color feature value of each sampling point in the target sampling point subset.

5. The method of claim 1, generating a target object based on each target sub-object, comprising: generating an initial target object based on each target sub-object; The resolution of the initial target object is improved to obtain the target object.

6. The method according to claim 5, wherein the resolution of the initial target object is improved to obtain the target object, comprising: Inputting the initial target object into a resolution enhancement model; A target object corresponding to the initial target object output by the resolution enhancement model is obtained.

7. The method of claim 1, training the neural radiation field model based on the prediction sub-object and the sample sub-object, and the prediction object and the sample object, comprising: Calculate a first loss value according to the prediction sub-object and the sample sub-object, and calculate a second loss value according to the prediction object and the sample object; The model parameters of the neural radiation field model are adjusted based on the first loss value and the second loss value, and the training is continued until the model training stop condition is reached.

8. The method according to claim 7, after rendering and generating the prediction sub-object corresponding to each sample sampling point subset, the method further comprises: Get the reference object; Performing image segmentation on the reference object to obtain a reference sub-object corresponding to the reference object; Calculate a third loss value according to the reference sub-object and the predicted sub-object; The model parameters of the neural radiation field model are adjusted according to the third loss value.

9. A method for generating a human face based on a neural radiation field, comprising: Determining sampling position information, sampling viewing angle information and a face image in response to a face generation instruction; The sampling position information, the sampling viewing angle information and the face image are input into a pre-trained neural radiation field model for processing, and the density information, color feature value and signed distance function value corresponding to each sampling point output by the neural radiation field model are obtained, wherein the neural radiation field model is trained by the following steps: the sample sampling position, the sample sampling viewing angle information and the sample object image are input into the neural radiation field model, and the density information, color feature value and signed distance function value corresponding to each sample sampling point output by the neural radiation field model are obtained; the sample sampling position, the sample sampling viewing angle information and the sample object image are input into an object generation model, and the sample object output by the object generation model is obtained; at least two sample sampling point subsets are determined according to the density information, color feature value and signed distance function value corresponding to each sample sampling point, and a prediction sub-object corresponding to each sample sampling point subset is rendered and generated, and a prediction object is generated based on each prediction sub-object; Performing image segmentation on the sample object to obtain a sample sub-object corresponding to the sample object; training the neural radiation field model according to the predicted sub-object and the sample sub-object, and the predicted object and the sample object; Determining at least two subsets of sampling points based on the signed distance function value corresponding to each sampling point; Based on each sampling point subset, a face subset corresponding to each sampling point subset is generated by rendering, and based on each face subset, a target face is generated.

10. An image processing device based on neural radiation field, comprising: a determination module, configured to determine sampling position information, sampling viewing angle information and a target object image in response to an image processing instruction; The input module is configured to input the sampling position information, the sampling viewing angle information and the target object image into a pre-trained neural radiation field model; the neural radiation field model emits sampling point rays to the sampling points of the target object image based on the sampling position information and the sampling viewing angle information; and determines the density information, color feature value and signed distance function value of each sampling point based on the sampling point rays corresponding to each sampling point, wherein the neural radiation field model is trained by the following steps: inputting the sample sampling position, the sample sampling viewing angle information and the sample object image into the neural radiation field model, and obtaining the density information, color feature value and signed distance function value corresponding to each sample sampling point output by the neural radiation field model. The method comprises the steps of: inputting the sample sampling position, the sample sampling viewing angle information and the sample object image into the object generation model to obtain the sample object output by the object generation model; determining at least two sample sampling point subsets according to the density information, color feature value and signed distance function value corresponding to each sample sampling point, rendering and generating the prediction sub-object corresponding to each sample sampling point subset, and generating the prediction object based on each prediction sub-object; performing image segmentation on the sample object to obtain the sample sub-object corresponding to the sample object; and training the neural radiation field model according to the prediction sub-object and the sample sub-object, and the prediction object and the sample object; A subset determination module is configured to determine at least two sampling point subsets based on the signed distance function value corresponding to each sampling point; The rendering module is configured to render and generate a target sub-object corresponding to each sampling point subset according to density information and color feature values ​​of the sampling points in each sampling point subset, and generate a target object based on each target sub-object.

11. An augmented reality (AR) device or a virtual reality (VR) device, comprising: Memory, processor and display; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the following steps are implemented: determining sampling position information, sampling viewing angle information, and a target object image in response to an image processing instruction; The sampling position information, the sampling viewing angle information and the target object image are input into a pre-trained neural radiation field model; the neural radiation field model emits sampling point rays to the sampling points of the target object image based on the sampling position information and the sampling viewing angle information; the density information, color feature value and signed distance function value of each sampling point are determined based on the sampling point rays corresponding to each sampling point, wherein the neural radiation field model is trained by the following steps: the sample sampling position, the sample sampling viewing angle information and the sample object image are input into the neural radiation field model, and the density information, color feature value and signed distance function value corresponding to each sample sampling point output by the neural radiation field model are obtained. The method comprises the steps of: inputting the sample sampling position, the sample sampling viewing angle information and the sample object image into an object generation model to obtain a sample object output by the object generation model; determining at least two sample sampling point subsets according to the density information, color feature value and signed distance function value corresponding to each sample sampling point, rendering and generating a prediction sub-object corresponding to each sample sampling point subset, and generating a prediction object based on each prediction sub-object; performing image segmentation on the sample object to obtain a sample sub-object corresponding to the sample object; and training the neural radiation field model according to the prediction sub-object and the sample sub-object, as well as the prediction object and the sample object; Determining at least two subsets of sampling points based on the signed distance function value corresponding to each sampling point; Rendering generates a target sub-object corresponding to each sampling point subset according to density information and color feature values ​​of the sampling points in each sampling point subset, and generating a target object based on each target sub-object; The target object is displayed through the display of the augmented reality (AR) device or the virtual reality (VR) device.

12. A computing device, comprising a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 8 or 9 when executing the computer instructions.

13. A computer-readable storage medium storing computer-executable instructions, wherein the computer instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 8 or 9.

Images