Methods, apparatus, devices, and media for inverse rendering of images

Abstract

Embodiments of the present disclosure disclose a method, an apparatus, an electronic device, and a storage medium for inverse rendering of an image. The method includes: inputting a processing target image into a feature prediction model, predicting geometric features and material features of the processing target image by the feature prediction model, and obtaining a geometric feature map and a material feature map of the processing target image, where the geometric feature map includes a normal map and a depth map, and the material feature map includes an albedo feature map, a roughness feature map, and a metallicity feature map; inputting the processing target image, the geometric feature map, and the material feature map into an illumination prediction model, predicting an illuminance value of the processing target image for each pixel, and obtaining an illumination feature map of the processing target image; and performing a preset process on the processing target image based on the geometric feature map, the material feature map, and the illumination feature map. The limitation of a simplified material representation for appearance acquisition in the inverse rendering process is overcome, which helps to improve the physical accuracy of the material, geometric shape, and illumination predicted by the inverse rendering.

Description

Technical Field

[0001] Cross - reference to Related Applications This disclosure claims priority to a Chinese patent application filed with the China National Intellectual Property Administration on June 17, 2022, with application number CN202210689653.X and invention title "Method, Apparatus, Device, and Medium for Inverse Rendering of Images", the entire content of which is incorporated herein by reference.

[0002] This disclosure relates to the field of computer vision, and more particularly to a method, apparatus, device, and medium for inverse rendering of images.

Background Art

[0003] Inverse rendering of images is an important application in the fields of computer graphics and computer vision, aiming to restore attributes such as the geometric shape, material, and illumination of an image from the image. In the fields of augmented reality and scene digitization, an image can be processed according to attributes such as the geometric shape, material, and illumination obtained by inverse rendering. For example, virtual objects can be generated within the image. The attributes such as the geometric shape, material, and illumination of the image obtained by inverse rendering are directly related to the fusion effect of virtual objects and the scene.

Summary of the Invention

Problems to be Solved by the Invention

[0004] Embodiments of this disclosure provide a method, apparatus, device, and medium for inverse rendering of images, which are used to improve the effect of image processing that depends on the material representation obtained by inverse rendering.

Means for Solving the Problems

[0005] A method for inverse rendering of images according to one aspect of the embodiments of this disclosure is A step of inputting an image to be processed into a feature prediction model, predicting the geometric features and material features of the image to be processed using the feature prediction model, and obtaining a geometric feature map and a material feature map of the image to be processed, wherein the geometric feature map includes a normal map and a depth map, and the material feature map includes an albedo feature map, a roughness feature map and a metallicity feature map. The steps include: inputting the image to be processed, the geometric feature map, and the material feature map into an illumination prediction model; predicting the illuminance value of the image to be processed pixel by pixel using the illumination prediction model; and obtaining an illumination feature map of the image to be processed. The step includes performing a pre-configured process on the image to be processed based on the geometric feature map, the material feature map, and the illumination feature map.

[0006] An apparatus for inverse rendering of an image according to another embodiment of the embodiments of the present disclosure is: The system is configured to input an image to be processed into a feature prediction model, predict the geometric features and material features of the image to be processed using the feature prediction model, and obtain a geometric feature map and a material feature map of the image to be processed, wherein the geometric feature map includes a normal map and a depth map, and the material feature map includes an albedo feature map, a roughness feature map and a metallicity feature map, A lighting prediction unit is configured to input the image to be processed, the geometric feature map, and the material feature map into a lighting prediction model, predict the illuminance value of the image to be processed for each pixel using the lighting prediction model, and obtain a lighting feature map of the image to be processed. The system includes an image processing unit configured to perform pre-set processing on the image to be processed based on the geometric feature map, the material feature map, and the illumination feature map.

[0007] Electronic devices according to further different embodiments of the embodiments of this disclosure include: Memory for storing computer program products, The system includes a processor that executes a computer program product stored in the memory, and when the computer program product is executed, implements a method for inverse rendering of an image, as provided by any of the above embodiments of the present disclosure.

[0008] A computer-readable storage medium according to a further different embodiment of the embodiments of the present disclosure stores program code which can be invoked by a processor to implement a method for inverse rendering of an image provided by any of the above embodiments of the present disclosure. [Effects of the Invention]

[0009] The solutions provided by the embodiments of this disclosure use a feature prediction model to predict the geometric and material features of an image to be processed, where the geometric features include normal features and depth features, and the material features include albedo, roughness, and metallicity. Subsequently, an illumination prediction model is used to predict the illuminance values ​​of the image to be processed, and pre-configured processing can be performed on the image according to the predicted geometric features, material features, and illuminance values. The depth features, albedo, roughness, and metallicity allow for more physically and accurately characterizing complex materials within the image to be processed, thereby enabling more detailed modeling of complex lighting environments such as specular reflection in subsequent processing steps. This overcomes the limitations of simplified material representation for appearance acquisition in the inverse rendering process and contributes to improved physical accuracy of predicted materials, geometric shapes, and lighting that rely on inverse rendering, as well as improved image processing effects that rely on material representations obtained by inverse rendering. For example, in the fields of mixed reality and scene digitization, this can thus improve the fusion effect between virtual objects and scenes.

[0010] The technical solutions of this disclosure will be described in more detail below through drawings and embodiments. [Brief explanation of the drawing]

[0011] The drawings, which constitute part of the specification, illustrate embodiments of the present disclosure and are used together with the description to interpret the principles of the present disclosure.

[0012] By referring to the drawings, this disclosure can be better understood from the following detailed description. Obviously, the drawings described below are only a few embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these without any creative effort. [Figure 1] This is a flowchart of one embodiment of the method for inverse rendering of an image according to the present disclosure. [Figure 2] This is a schematic diagram of one scene of a method for inverse rendering of an image in this disclosure. [Figure 3] This is a schematic diagram of the process for training a feature prediction model and an illumination prediction model in one embodiment of the method for inverse rendering of images of the present disclosure. [Figure 4] This is a schematic diagram of the process for pre-training an illumination prediction model in one embodiment of the method for inverse rendering of images of the present disclosure. [Figure 5] This is a schematic diagram of the process for calculating the spatial loss function in one embodiment of the method for inverse rendering of images of the present disclosure. [Figure 6] This is a schematic diagram of the structure of one embodiment of an apparatus for inverse rendering of images in the present disclosure. [Figure 7] This is a schematic diagram of the structure of one application embodiment of the electronic device of this disclosure. [Modes for carrying out the invention]

[0013] Hereinafter, various exemplary embodiments of this disclosure will be described in detail with reference to the drawings. The relative arrangements of components and steps, formulas, and numerical values ​​described in these embodiments are not intended to limit the scope of this disclosure unless otherwise specified.

[0014] In the embodiments of the present disclosure, it should also be understood that "a plurality" can mean two or more, and "at least one" can mean one, two or more.

[0015] Those skilled in the art can understand that terms such as "first" and "second" in the embodiments of the present disclosure are only for distinguishing different steps, devices or modules, etc., and do not represent any specific technical meaning, nor do they represent an inevitable logical order between them.

[0016] It should also be understood that any member, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more unless there is an explicit limitation or the context gives an opposite indication.

[0017] In the description of each embodiment in the present disclosure, the differences between each embodiment are emphasized, but the same points or similar points may be referred to each other, and for the sake of brevity, it should also be understood that they will not be described one by one.

[0018] The following description of at least one exemplary embodiment is in fact only exemplary and in no way limits the present disclosure and its applications.

[0019] Although it may not be possible to discuss in detail the technologies, methods and devices known to those skilled in the relevant art, where appropriate, the said technologies, methods and devices should be regarded as part of the specification.

[0020] In addition, since similar symbols and characters represent similar items in the following drawings, once an item is defined in one drawing, there is no need for further discussion in later drawings.

[0021] Furthermore, the terms "and / or" in this disclosure are solely for the purpose of describing the relationship between related objects, and indicate that there may be three types of relationships, such that A and / or B can represent three situations: A existing alone, A and B existing simultaneously, or B existing alone. Also, in this disclosure, the letter " / " generally indicates that the related objects before and after it are in an "or" relationship.

[0022] The embodiments of this disclosure may be applied to electronic devices such as terminal devices, computer systems, and servers that can operate in many other general-purpose or dedicated computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use in electronic devices such as terminal devices, computer systems, and servers include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems.

[0023] Electronic devices such as terminals, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Typically, program modules can include routines, programs, object programs, components, logic, and data structures that perform specific tasks or realize specific abstract data types. Computer systems / servers may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked over a communication network. In a distributed cloud computing environment, program modules may reside in storage media of local or remote computing systems, including storage devices.

[0024] To further clarify the technical solutions and advantages of the embodiments of this disclosure, exemplary embodiments of this disclosure will be described below with reference to the drawings, and it is clear that the embodiments described are only a selection of embodiments of this disclosure and do not encompass all embodiments. For the purposes of this explanation, the embodiments and features of the embodiments of this disclosure can be combined with each other without conflict.

[0025] The method for inverse rendering of an image according to the present disclosure is illustrated below with reference to Figure 1. Figure 1 shows a flowchart of one embodiment of the method for inverse rendering of an image according to the present disclosure, and as shown in Figure 1, this process includes the following steps: Step 110: The image to be processed is input into the feature prediction model, and the feature prediction model predicts the geometric features and material features of the image to be processed, thereby obtaining a geometric feature map and a material feature map of the image to be processed.

[0026] Here, the geometric feature map includes a normal map and a depth map, and the material feature map includes an albedo feature map, a roughness feature map, and a metallicity feature map.

[0027] In this embodiment, geometric features can characterize the geometric attributes of the image to be processed, and may include, for example, normal features and depth features. Normal features can characterize the normal vector of a pixel point, and depth features can characterize the depth of a pixel point. Material features can characterize the material attributes of the pixel points in the image to be processed, and may include, for example, albedo (base color), roughness, and metallicity. Albedo can represent the ratio of the light stream scattered in at least one direction by all illuminated portions of the object's surface to the light stream incident on the object's surface. Roughness can represent the smoothness of the object's surface and is used to describe the behavior of light when it strikes the object's surface. For example, the less rough the object's surface is, the closer the light is to specular reflection when it strikes the object's surface. Metallicity is used to characterize the degree of metallicity of the object. The higher the metallicity, the closer the object is to metal, and conversely, the closer it is to nonmetal.

[0028] A feature prediction model can characterize the correspondence between an image to be processed and its geometric and material features, predict the geometric and material features of each pixel point in the image to be processed, and is used to form a corresponding feature map based on the predicted feature values. Accordingly, the normal map, depth map, albedo feature map, roughness feature map, and metallicity feature map can represent the normal vector, depth, albedo, roughness, and metallicity of at least one pixel point in the image to be processed, respectively.

[0029] In one specific example, the feature prediction model may be any neural network model such as a convolutional neural network or residual network, for example, a multi-branch encoder / decoder based on ResNet and UNet, the encoder may be ResNet-18, and the decoder may consist of five convolutional layers with skip connections. After training the feature prediction model using sample data, the feature prediction model is used to perform processing such as feature extraction, downsampling, high-dimensional feature extraction, upsampling, decoding, layer hopping connection, and shallow feature fusion of the image to be processed. Finally, the normal features, depth features, albedo, roughness, and metallicity of each pixel point in the image to be processed are predicted, and normal feature maps, depth feature maps, albedo feature maps, roughness feature maps, and metallicity feature maps are formed based on the predicted feature values, thereby obtaining the geometric and material features of the image to be processed.

[0030] In one possible example, step 110 may be performed by the processor calling a corresponding instruction stored in memory, or by a feature prediction unit operated by the processor.

[0031] Step 120: The image to be processed, the geometric feature map, and the material feature map are input to the illumination prediction model. The illumination prediction model predicts the illuminance value of the image to be processed for each pixel, and an illumination feature map of the image to be processed is obtained.

[0032] In this embodiment, illuminance values ​​can characterize the lighting environment of a point in space. The lighting prediction model can characterize the correspondence between the processed image and its geometric and material features and the lighting environment.

[0033] In one specific example, the illumination prediction model can use any neural network model, such as a convolutional neural network or a residual network, for example, a multi-branch encoder / decoder based on ResNet and UNet. The execution entity (which may be, for example, a terminal device or a server) preprocesses by superimposing the image to be processed, geometric feature maps (including normal feature maps and depth feature maps), and material feature maps (including albedo feature maps, roughness feature maps, and metallicity feature maps) in a given number of channels. The superimposed image is then input to the illumination prediction model, which predicts the spatial illumination environment of each pixel point, i.e., the illuminance value of each pixel point, through operations such as feature extraction, encoding, and decoding, and forms a spatially continuous HDR illumination feature map based on the predicted illuminance values.

[0034] In one possible example, step 120 may be performed by the processor calling a corresponding instruction stored in memory, or by a lighting prediction unit operated by the processor.

[0035] Step 130: Based on the geometric feature map, material feature map, and lighting feature map, the pre-configured processing is performed on the image to be processed.

[0036] In this embodiment, steps 110 and 120 enable inverse rendering of the image to be processed, thereby obtaining the geometric and material characteristics of the image to be processed. The pre-configured processing represents subsequent processing performed on the image to be processed based on the geometric and material characteristics obtained by inverse rendering. For example, in the field of mixed reality, a real image collected by a camera can be used as the image to be processed, and a virtual image can be inserted into the real image to realize the fusion of the real world and a virtual image. Furthermore, for example, a virtual object can be generated within the image to be processed by dynamic virtual object synthesis based on the geometric and material characteristics of the image to be processed. Also, for example, the material of an object within the image to be processed can be edited based on the geometric and material characteristics of the image to be processed to present an object with a different material.

[0037] The following describes, with reference to the scene shown in Figure 2, the method for inverse rendering of an image in this embodiment. As shown in Figure 2, the image to be processed 210 is an LDR panoramic image, and the geometric feature map 230 and material feature map 240 of the image to be processed 210 can be predicted using the feature prediction model 220. The geometric feature map includes a normal feature map 231 and a depth feature map 232, and the material feature map includes an albedo feature map 241, a roughness feature map 242, and a metallicity feature map 243. Subsequently, the image to be processed 210, the geometric feature map 230, and the material feature map 240 are input to the second prediction model 250 to obtain an illumination feature map 260. Then, based on the geometric feature map 230 and the material feature map 240, virtual objects 271, 272, and 273 are generated within the image to be processed 210 to obtain the processed image 270.

[0038] In one possible example, step 130 may be performed by the processor calling a corresponding instruction stored in memory, or by an image processing unit operated by the processor.

[0039] The method for inverse rendering of an image provided by this embodiment uses a feature prediction model to predict the geometric and material features of the image to be processed, where the geometric features include normal features and depth features, and the material features include albedo, roughness, and metallicity. Subsequently, an illumination prediction model is used to predict the illuminance value of the image to be processed, and pre-configured processing can be performed on the image according to the predicted geometric features, material features, and illuminance value. The depth features, albedo, roughness, and metallicity allow for more physically and accurately characterizing complex materials within the image to be processed, thereby enabling more detailed modeling of complex lighting environments such as specular reflection in subsequent processing. This overcomes the limitations of simplified material representation for appearance acquisition in the inverse rendering process and contributes to improved physical accuracy of material, geometric shape, and illumination predicted by inverse rendering, as well as improved image processing effects that rely on material representation obtained by inverse rendering.

[0040] In some selectable embodiments of this embodiment, step 120 may further include the steps of processing a target image, a geometric feature map, and a material feature map using an illumination prediction model to predict the illuminance values ​​of pixel points in the target image, generating panoramic images corresponding to the pixel points based on the predicted illuminance values, and stitching the panoramic images corresponding to the pixel points in the target image to obtain an illumination feature map.

[0041] In this embodiment, the illumination prediction model can predict the illumination environment of each pixel point in space by processing the image to be processed, a geometric feature map, and a material feature map. Since a point in space can receive light emitted from any angle in space, a 360° panoramic image can be used to characterize the illumination environment of a point. Then, according to the position of the pixel point in the image to be processed, the panoramic image corresponding to at least one pixel point is stitched into the illumination feature map.

[0042] In this embodiment, the illumination characteristics of the image to be processed can be more accurately characterized by predicting the illuminance values ​​of pixel points in the image to be processed using an illumination prediction model and characterizing the illuminance values ​​of pixel points using a panoramic image.

[0043] Next, referring to Figure 3, which shows a schematic diagram of the process for training a feature prediction model and an illumination prediction model in one embodiment of the method for inverse rendering of images of the present disclosure, as shown in Figure 3, this process includes the following steps: Step 310: The sample image is input into a pre-trained feature prediction model to predict the geometric and material features of the sample image, and a sample geometric feature map and a sample material feature map of the sample image are obtained.

[0044] In this embodiment, the pre-trained feature prediction model represents a feature prediction model that can complete prediction operations on the input image through training.

[0045] For example, a virtual dataset can be used to pre-train a feature prediction model. The virtual dataset can include virtual images obtained by a forward rendering process, virtual geometric feature maps, and virtual material feature maps generated during the forward rendering process. Subsequently, a pre-trained feature prediction model can be obtained by training the initial feature prediction model using the virtual images as input and the virtual geometric feature maps and virtual material feature maps as desired outputs.

[0046] In one possible example, step 310 may be performed by the processor calling a corresponding instruction stored in memory, or by a model training unit operated by the processor.

[0047] Step 320: The sample image, sample geometric feature map, and sample material feature map are input into a pre-trained illumination prediction model to predict the illuminance values ​​of pixel points in the sample image and obtain a sample illumination feature map of the sample image.

[0048] In this embodiment, the pre-trained illumination prediction model represents an illumination prediction model that, through training, can complete prediction operations on sample images, sample geometric feature maps, and sample material feature maps.

[0049] For example, a virtual dataset can be used to pre-train a lighting prediction model. This virtual dataset may include virtual images obtained through a forward rendering process, virtual geometric feature maps, virtual material feature maps, and virtual lighting feature maps generated during the forward rendering process. By using the virtual images, virtual geometric feature maps, and virtual material feature maps as inputs and the virtual lighting feature map as the desired output, a pre-trained feature prediction model can be obtained by training an initial lighting feature prediction model.

[0050] In one possible example, step 320 may be performed by the processor calling a corresponding instruction stored in memory, or by a model training unit operated by the processor.

[0051] Step 330: Use the Differentiable Rendering Module to generate a rendered image based on the sample geometric feature map, sample material feature map, and sample illumination feature map.

[0052] In related technologies, when generating images through rendering, the rendering process becomes non-differentiable because the relationship between the light rays received by the camera and the entire scene cannot be determined during the ray tracing stage. Since the reverse conduction of the neural network is realized through derivation, it is not possible to impose constraints on the neural network in a non-differentiable rendering process.

[0053] In this embodiment, the sample geometric feature map, sample material feature map, and sample illumination feature map obtained by inverse rendering are images obtained by mapping feature values ​​into camera space. The differentiable rendering module calculates shading values ​​using the sample geometric feature map, sample material feature map, and sample illumination feature map directly without performing ray tracing, and thereby generates a rendered image through a differentiable rendering process.

[0054] For example, the differentiable rendering module can determine the normal vector, albedo, roughness, and metallicity of each pixel point from a sample geometric feature map, a sample material feature map, and a sample illumination feature map. It then substitutes these values ​​into the rendering equation, solves the rendering equation using a Monte Carlo sampling method, and determines the shading value for that pixel point. Here, to generate more detailed specular reflections, a Monte Carlo integral can be calculated using an importance sampling method.

[0055] The following equations (1) to (6) represent the differentiable rendering process in this example, where equation (1) is the rendering equation.

number

number

number

number

number

number

number

[0056] In one possible example, step 330 may be performed by the processor calling a corresponding instruction stored in memory, or by a model training unit operated by the processor.

[0057] Step 340: Based on the differences between the sample image and the rendered image, the parameters of the pre-trained feature prediction model and the pre-trained illumination prediction model are adjusted until the pre-set training completion conditions are met (i.e., the pre-trained feature prediction model and the pre-trained illumination prediction model are trained) to obtain the feature prediction model and the illumination prediction model.

[0058] For example, the pre-set training completion condition may be that the loss function converges, or that the number of iterations of steps 310 to 240 reaches a pre-set number.

[0059] For example, the execution body can use the L1 or L2 function as the rendering loss function, and then determine the value of the rendering loss function based on the difference between the sample image and the rendered image. Subsequently, by utilizing the back conduction properties of the neural network to derive the rendering loss function, the parameters of the pre-trained feature prediction model and the pre-trained illumination prediction model can be adjusted until the function value of the rendering loss function converges, thereby obtaining the feature prediction model and the illumination prediction model.

[0060] Furthermore, for example, when the number of iterations of steps 310 to 340 reaches a predetermined number, training can be terminated, and a feature prediction model and an illumination prediction model can be obtained.

[0061] In this embodiment, a rendered image is generated by a differentiable rendering process based on geometric features, material features, and lighting features obtained by inverse rendering. By adjusting the parameters of a pre-trained feature prediction model and a pre-trained lighting prediction model based on the differences between the rendered image and the sample image, physical constraints can be imposed on the feature prediction model and the lighting prediction model. This improves the accuracy of the feature prediction model and the lighting prediction model, which helps to improve the accuracy of the attributes obtained by inverse rendering.

[0062] In one possible example, step 340 may be performed by the processor calling a corresponding instruction stored in memory, or by a model training unit operated by the processor.

[0063] In some of the selectable embodiments of the above examples, the pre-training process for the illumination feature prediction model can employ the process shown in Figure 4, which, as shown in Figure 4, includes the following steps: Step 410: An initial illumination feature map is obtained by processing the sample data with an initial illumination feature prediction model.

[0064] For example, sample data may include a virtual image obtained by a forward rendering process, a virtual geometric feature map, a virtual material feature map, and a virtual illumination feature map generated during the forward rendering process. Here, the virtual image, virtual geometric feature map, and virtual material feature map may be used as inputs, and the virtual illumination feature map may be used as sample labels.

[0065] In one possible example, step 410 may be performed by the processor calling a corresponding instruction stored in memory, or by a pre-training unit operated by the processor.

[0066] Step 420: Determine the value of the prediction loss function based on the difference between the initial illumination feature map and the sample labels.

[0067] In this embodiment, the prediction loss function characterizes the degree of difference between the output of the initial illumination prediction model and the sample labels, and for example, the L1 function or the L2 function can be used as the prediction loss function.

[0068] In one possible example, step 420 may be performed by the processor calling a corresponding instruction stored in memory, or by a pre-training unit operated by the processor.

[0069] Step 430: Determine the value of the spatial continuity loss function based on the difference in illuminance values ​​and the difference in depth between adjacent pixel points in the initial illumination feature map.

[0070] Typically, the lighting environment between two adjacent points in space is similar, and conversely, the lighting environment between two points far apart is significantly different. After mapping these two points to an image, the distance between them in space can be represented by the depth between pixels.

[0071] In this embodiment, the spatial continuity loss function can represent the difference in illumination environment between adjacent pixel points. When the difference in depth between two adjacent pixel points is small, it indicates that their illumination environments are similar, and in this case, the value of the spatial continuity loss function is also small. Conversely, when the difference in depth between two adjacent pixel points is large, it indicates that their illumination environments can be significantly different, and in this case, the value of the spatial continuity loss function is also large.

[0072] In one possible example, step 430 may be performed by the processor calling a corresponding instruction stored in memory, or by a pre-training unit operated by the processor.

[0073] Step 440: Based on the values ​​of the prediction loss function and the spatial continuity loss function, the initial illumination feature prediction model is trained to obtain a pre-trained illumination feature prediction model.

[0074] In one possible example, step 440 may be performed by the processor calling a corresponding instruction stored in memory, or by a pre-training unit operated by the processor.

[0075] In this embodiment, the execution unit iteratively performs steps 410 to 440 and adjusts the parameters of the initial illumination feature prediction model based on the values ​​of the prediction loss function and the spatial continuity loss function until the prediction loss function and the spatial continuity loss function converge or the number of iterations of steps 410 to 440 reaches a predetermined number, thereby terminating the training and obtaining a pre-trained illumination prediction model.

[0076] The embodiment shown in Figure 4 embodies the step of constraining the pre-training of the illumination prediction model using a prediction loss function and a spatial continuity loss function. The spatial continuity loss function provides an overall constraint on the local illumination of the image to be processed, preventing illumination mutations. This constrains the pre-training of the illumination prediction model, improves the accuracy of the illumination prediction model, and helps to more accurately obtain the illumination features of the image to be processed.

[0077] In some selectable embodiments of the example shown in Figure 4, the value of the spatial continuous loss function can be determined by the process shown in Figure 5, which, as shown in Figure 5, includes the following steps: Step 510: Project the illuminance value of a pixel point in the initial illumination feature map onto an adjacent pixel point to obtain the projected illuminance value of the pixel point in the initial illumination feature map, and the difference between the illuminance value of the pixel point in the initial illumination feature map and the projected illuminance value. value To decide.

[0078] In this embodiment, the difference between the illuminance value and projected illuminance value of a pixel point in the initial illumination feature map. value This can characterize the differences in lighting environments between adjacent pixel points.

[0079] For example, the implementing entity can perform the projection of illuminance values ​​using a projection operator. By projecting the illuminance value of each pixel point onto adjacent pixel points in a predetermined direction, the projected illuminance value of each pixel point can be obtained. Subsequently, the difference between the illuminance value of each pixel point and the projected illuminance value can be calculated. value It is possible to make a decision.

[0080] In one possible example, step 510 may be performed by the processor calling a corresponding instruction stored in memory, or by a pre-training unit operated by the processor.

[0081] Step 520: Determine the scaling factor based on the depth gradient of the pixel points in the initial illumination feature map and the pre-configured continuity weight parameter.

[0082] Here, the scaling factor and the depth gradient are positively correlated.

[0083] In this embodiment, the depth gradient of a pixel point can represent the spatial distance between adjacent pixel points. The value of the continuity weight parameter may typically be set empirically.

[0084] For example, the implementing entity can first predict the depth gradient between two adjacent pixel points, and then determine a scaling factor based on the depth gradient and continuity weight parameters. The scaling factor allows for a certain deviation in the illumination environment between at least one pixel point.

[0085] In one possible example, step 520 may be performed by the processor calling a corresponding instruction stored in memory, or by a pre-training unit operated by the processor.

[0086] Step 530, difference value Based on the scaling factor, the value of the spatial continuity loss function is determined.

[0087] For example, the execution body is the difference corresponding to each pixel point. value Then, by multiplying these by the corresponding scaling factors, the average of the sum of the products corresponding to all pixel points can be used as the value of the spatial continuity loss function.

[0088] For example, the spatial continuity loss function in this embodiment can be given by the following equation (7):

number

number

number

number

[0089] In one selectable example, step 530 may be performed by the processor calling a corresponding instruction stored in memory, or by a pre-training unit operated by the processor.

[0090] In the process shown in Figure 5, the difference between the lighting environments of adjacent pixel points is determined by the difference between the illuminance value of the pixel point and the projected illumination. value The scaling factor is determined based on the depth gradient and continuity weight parameter of the pixel point, and the difference between the illuminance value of the pixel point and the projected illumination is expressed as follows: value By determining the value of the spatial continuity loss function based on scaling factors, the differences between lighting environments at points located at different positions in space can be represented more accurately. For example, the lighting environments of distant points can be significantly different, and the lighting environments of nearby points can be similar. This constrains the pre-training process of the lighting prediction model, allowing it to learn the potential correlation between the location of a point in space and its lighting environment, thereby improving prediction accuracy.

[0091] In some selectable embodiments of the above embodiment, after obtaining the geometric feature map and material feature map of the image to be processed through step 110, the albedo feature map and roughness feature map can also be processed as follows: the image to be processed, the geometric feature map and material feature map are input into a guided filtering model, filtering parameters are determined, and the albedo feature map and roughness feature map are smoothed based on the filtering parameters.

[0092] In this embodiment, the albedo feature map and roughness feature map can be smoothed using a guided filtering model to improve the image quality of the albedo feature map and roughness feature map. By inputting the smoothed albedo feature map and roughness feature map into the illumination prediction model, it is possible to improve the prediction accuracy of illumination features. At the same time, by using the smoothed albedo feature map and roughness feature map and performing pre-configured processing on the image to be processed, the image quality of the processed image can be improved.

[0093] For example, a guided filtering model may be a convolutional neural network with an embedded guided filtering layer.

[0094] Furthermore, the filtering parameters are obtained as follows: an input image is generated based on the image to be processed, geometric feature map, and material feature map; the resolution of the input image is lower than the resolution of the image to be processed; an initial filtering parameter for the input image is predicted using a guided filtering model; the initial filtering parameter is upsampled to obtain a filtering parameter that matches the resolution of the image to be processed.

[0095] For example, the resolution of the image to be processed, the geometric feature map, and the material feature map can be reduced to half of their original resolution, then input into a guided filtering model to obtain initial filtering parameters at half the resolution, and then these initial filtering parameters can be upsampled to obtain filtering parameters that match the original resolution.

[0096] In this embodiment, filtering parameters can be obtained more quickly by first reducing the resolution of the input image to obtain initial filtering parameters, and then obtaining filtering parameters that match the input image by upsampling. This helps to improve the efficiency of image smoothing processing by the guided filtering model.

[0097] The methods for inverse rendering of images provided by embodiments of this disclosure may be performed by any suitable device with data processing capabilities, including but not limited to terminal devices and servers. Alternatively, the methods for inverse rendering of images provided by embodiments of this disclosure may be performed by a processor, for example, by calling corresponding instructions stored in memory, the processor performing the methods for inverse rendering of images described in embodiments of this disclosure. Repeated explanations are omitted below.

[0098] Those skilled in the art will understand that all or some of the steps of the embodiments of the above method may be completed by a program that instructs the relevant hardware, the program may be stored in a computer-readable storage medium, and when the program is executed, the steps of the embodiments of the above method are executed, the storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks or optical disks.

[0099] Refer to Figure 6 below, which is a schematic diagram of the structure of one embodiment of the apparatus for inverse rendering of images of the present disclosure. The apparatus of the embodiment can be used to implement embodiments of each of the above methods of the present disclosure. As shown in Figure 6, the apparatus comprises a feature prediction unit 610 configured to input an image to be processed into a feature prediction model, which predicts the geometric and material features of the image to be processed into the feature prediction model, and to obtain a geometric feature map and a material feature map of the image to be processed, wherein the geometric feature map includes a normal map and a depth map, and the material feature map includes an albedo feature map, a roughness feature map, and a metallicity feature map; an illumination prediction unit 620 configured to input the image to be processed, the geometric feature map, and the material feature map into an illumination prediction model, which predicts the illuminance value of the image to be processed pixel by pixel, and to obtain an illumination feature map of the image to be processed; and an image processing unit 630 configured to perform pre-set processing on the image to be processed based on the geometric feature map, the material feature map, and the illumination feature map.

[0100] In one embodiment, the illumination prediction unit 620 further includes a prediction module configured to process a target image, a geometric feature map, and a material feature map using an illumination prediction model, predict the illuminance values ​​of pixel points in the target image, and generate a panoramic image corresponding to the pixel points based on the predicted illuminance values; and a stitching module configured to stitch the panoramic images corresponding to the pixel points in the target image to obtain an illumination feature map.

[0101] In one embodiment, the device further comprises a model training unit configured to input a sample image into a pre-trained feature prediction model, predict the geometric and material features of the sample image, obtain a sample geometric feature map and a sample material feature map of the sample image, input the sample image, the sample geometric feature map and the sample material feature map into a pre-trained illumination prediction model, predict the illuminance values ​​of pixel points in the sample image, obtain a sample illumination feature map of the sample image, use a differentiable rendering module to generate a rendered image based on the sample geometric feature map, the sample material feature map and the sample illumination feature map, and adjust the parameters of the pre-trained feature prediction model and the pre-trained illumination prediction model based on the differences between the sample image and the rendered image until a pre-set training completion condition is met, thereby obtaining a feature prediction model and an illumination prediction model.

[0102] In one embodiment, the device further comprises a pre-training unit configured to obtain an initial illumination feature map obtained by processing sample data with an initial illumination feature prediction model, determine the value of a prediction loss function based on the difference between the initial illumination feature map and the sample labels, determine the value of a spatial continuous loss function based on the difference in illuminance values ​​and the difference in depth between adjacent pixel points in the initial illumination feature map, train the initial illumination feature prediction model based on the value of the prediction loss function and the value of the spatial continuous loss function, and obtain a pre-trained illumination feature prediction model.

[0103] In one embodiment, the pre-training unit projects the illuminance value of a pixel point in the initial illumination feature map onto an adjacent pixel point, obtains the projected illuminance value of the pixel point in the initial illumination feature map, and the difference between the illuminance value of the pixel point in the initial illumination feature map and the projected illuminance value. valueDetermine the scaling factor based on the depth gradient of pixel points in the initial illumination feature map and a pre-set continuity weight parameter, and if the scaling factor and depth gradient are positively correlated, then the difference value It further includes a loss function module configured to determine the value of a spatially continuous loss function based on a scaling factor.

[0104] In one embodiment, the apparatus further comprises a filtering unit configured to input an image to be processed, a geometric feature map, and a material feature map into a guided filtering model, determine filtering parameters, and smooth an albedo feature map and a roughness feature map based on the filtering parameters.

[0105] In one embodiment, the apparatus further comprises a parameter determination unit configured to generate an input image based on an image to be processed, a geometric feature map and a material feature map, predict initial filtering parameters for the input image using a guided filtering model, and upsample the initial filtering parameters to obtain filtering parameters that match the resolution of the image to be processed, provided that the resolution of the input image is lower than the resolution of the image to be processed.

[0106] Furthermore, the embodiments of this disclosure are Memory for storing computer programs, Further providing is an electronic device comprising a processor for executing a computer program stored in the memory, and, once the computer program is executed, for implementing a method for inverse rendering of an image as described in any of the above embodiments of this disclosure.

[0107] Furthermore, embodiments of the present disclosure provide a computer-readable storage medium on which computer program instructions are stored, and which, when executed by a processor, can realize a method for inverse rendering of an image in any of the above embodiments.

[0108] Hereinafter, an electronic device according to an embodiment of the present disclosure will be described with reference to Figure 7.

[0109] Figure 7 shows a block diagram of an electronic device according to an embodiment of the present disclosure.

[0110] As shown in Figure 7, the electronic device comprises one or more processors and memory.

[0111] The processor may be a central processing unit (CPU), or other form of processing unit having data processing capability and / or instruction execution capability, which can control other components in the electronic device to perform a desired function.

[0112] The memory can store one or more computer program products, and the memory may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache). The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program products may be stored in the computer-readable storage media, and the processor may execute the computer program products to implement the methods for inverse rendering of images and / or other desired functions in each of the above embodiments of the present disclosure.

[0113] As one example, the electronic device may further include input and output devices, and these components may be interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0114] Furthermore, the input device may further include, for example, a keyboard and a mouse.

[0115] The output device can output various information to the outside, including determined distance information, direction information, and so on. The output device may include, for example, a display, speaker, printer, communication network, and remote output device connected thereto.

[0116] Naturally, for the sake of simplification, Figure 7 shows only some of the components relevant to this disclosure in the electronic device, and components such as buses and input / output interfaces are omitted. Furthermore, depending on the specific application, the electronic device may be further equipped with any other suitable components.

[0117] In addition to the methods and apparatus described above, embodiments of the present disclosure may also be computer program products including computer program instructions, where, when the computer program instructions are executed by a processor, the processor performs steps of the methods for inverse rendering of images according to various embodiments of the present disclosure, as described in the above portion of this specification.

[0118] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure, and such programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as the C language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, as a single standalone software package, partially on the user's computing device and a remote computing device, or fully on a remote computing device or a server.

[0119] Furthermore, embodiments of the present disclosure may also be computer-readable storage media on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the processor performs steps of the methods for inverse rendering of images according to various embodiments of the present disclosure, as described in the above portion of this specification.

[0120] The computer-readable storage medium may be any combination of one or more readable storage media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium includes, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any combination thereof. More specific examples (non-exclusive list) of readable storage media include electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0121] The basic principles of this disclosure have been explained above along with specific embodiments. However, it should be noted that the advantages, advantages, and effects mentioned in this disclosure are merely examples and not limiting, and it is not considered necessary for each embodiment of this disclosure. Furthermore, the above specific details in this disclosure are for illustrative purposes only, to facilitate understanding, and are not limiting, and the above details do not limit the fact that this disclosure must be realized using the above specific details.

[0122] Each embodiment in this specification is described progressively, and the main points described in each embodiment are the differences from other embodiments. Parts that are the same or similar between embodiments should be referenced to one another. System embodiments are described briefly, as they essentially correspond to method embodiments; relevant details should be referred to the partial description of the method embodiments.

[0123] The block diagrams of the devices, apparatus, equipment, and systems relating to this disclosure are illustrative examples only and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. Those skilled in the art will recognize that these devices, apparatus, equipment, and systems may be connected, arranged, and configured in any manner. For example, words such as “inclusive,” “includes,” and “have” are open vocabulary and are interchangeable with each other, meaning “including, but not limited to…”. The vocabulary “or” and “and” as used herein are interchangeable with each other, unless the context clearly indicates otherwise. The vocabulary “for example” as used herein is interchangeable with the phrase “for example,…, but not limited to…”.

[0124] The methods and apparatus of this disclosure can be implemented in many ways. For example, the methods and apparatus of this disclosure can be implemented in software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the methods is for illustrative purposes only, but the steps of the methods of this disclosure are not limited to the order described above unless otherwise specifically stated. In some embodiments, the disclosure can also be implemented as a program recorded on a recording medium, which includes machine-readable instructions for implementing the methods of this disclosure. Thus, the disclosure also includes a recording medium for storing a program for performing the methods of this disclosure.

[0125] It should also be noted that in the apparatus, equipment and methods of this disclosure, each component or step is disassembled and / or reassembled. These disassembly and / or reassembly should be considered equivalent solutions of this disclosure.

[0126] The above description relating to the disclosed aspects is provided so that a person skilled in the art can create or use this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the embodiments shown herein, but rather to the broadest extent consistent with the principles and novel features disclosed herein.

[0127] The above description is for illustrative and explanatory purposes only. Furthermore, this description is not intended to limit the embodiments of this disclosure to those disclosed herein. While several exemplary embodiments and examples have been discussed above, several variations, modifications, changes, additions, and subcombinations thereof will be recognizable to those skilled in the art.

Claims

1. A method for inverse rendering of an image, A step of inputting an image to be processed into a feature prediction model, predicting the geometric features and material features of the image to be processed using the feature prediction model, and obtaining a geometric feature map and a material feature map of the image to be processed, wherein the geometric feature map includes a normal map and a depth map, and the material feature map includes an albedo feature map, a roughness feature map and a metallicity feature map. The steps include: inputting the image to be processed, the geometric feature map, and the material feature map into an illumination prediction model; predicting the illuminance value of the image to be processed pixel by pixel using the illumination prediction model; and obtaining an illumination feature map of the image to be processed. The step includes performing a pre-configured process on the image to be processed based on the geometric feature map, the material feature map, and the illumination feature map, The steps of inputting the image to be processed, the geometric feature map, and the material feature map into an illumination prediction model, predicting the illuminance value of the image to be processed pixel by pixel using the illumination prediction model, and obtaining an illumination feature map of the image to be processed are as follows: The steps include: processing the image to be processed, the geometric feature map, and the material feature map using the illumination prediction model; predicting the illuminance value of a pixel point in the image to be processed; and generating a panoramic image corresponding to the pixel point based on the predicted illuminance value. A method for inverse rendering an image, characterized by including the step of stitching together panoramic images corresponding to pixel points in the image to be processed to obtain the illumination feature map.

2. The process further includes the step of obtaining the feature prediction model and the illumination prediction model, the steps of which are as follows: A sample image is input into a pre-trained feature prediction model to predict the geometric and material features of the sample image, and a sample geometric feature map and a sample material feature map of the sample image are obtained. The sample image, the sample geometric feature map, and the sample material feature map are input into a pre-trained illumination prediction model to predict the illuminance values ​​of pixel points in the sample image and to obtain a sample illumination feature map of the sample image. Using the differentiable rendering module, a rendered image is generated based on the sample geometric feature map, the sample material feature map, and the sample illumination feature map. The method according to claim 1, characterized in that, based on the differences between the sample image and the rendered image, the parameters of the pre-trained feature prediction model and the pre-trained illumination prediction model are adjusted until a pre-set training completion condition is met, thereby obtaining the feature prediction model and the illumination prediction model.

3. The process further includes the step of obtaining the aforementioned pre-trained lighting feature prediction model, the step of which is as follows: We obtain an initial illumination feature map by processing the sample data with an initial illumination feature prediction model. Based on the difference between the initial illumination feature map and the sample labels, the value of the prediction loss function is determined. Based on the difference in illuminance values ​​of adjacent pixel points and the difference in depth of adjacent pixel points in the initial illumination feature map, the value of the spatial continuity loss function is determined. The method according to claim 2, characterized in that the initial illumination feature prediction model is trained based on the value of the prediction loss function and the value of the spatial continuity loss function to obtain the pre-trained illumination feature prediction model.

4. The step of determining the value of the spatial continuity loss function based on the difference in illuminance values ​​of adjacent pixel points and the difference in depth of adjacent pixel points in the initial illumination feature map is: The steps include: projecting the illuminance value of a pixel point in the initial illumination feature map onto an adjacent pixel point to obtain the projected illuminance value of the pixel point in the initial illumination feature map, and determining the difference between the illuminance value of the pixel point in the initial illumination feature map and the projected illuminance value; A step of determining a scaling coefficient based on the depth gradient of pixel points in the initial illumination feature map and a pre-set continuity weight parameter, wherein the scaling coefficient and the depth gradient are positively correlated. The method according to claim 3, characterized by comprising the step of determining the value of the spatial continuous loss function based on the difference and the scaling coefficient.

5. After obtaining the geometric feature map and material feature map of the image to be processed, the method proceeds as follows: The steps include inputting the image to be processed, the geometric feature map, and the material feature map into a guided filtering model and determining the filtering parameters, The method according to any one of claims 1 to 4, further comprising the step of smoothing the albedo feature map and the roughness feature map based on the filtering parameters.

6. The method further includes the step of obtaining the filtering parameters, the step of which is as follows: Based on the image to be processed, the geometric feature map, and the material feature map, an input image is generated, and the resolution of the input image is lower than the resolution of the image to be processed. The method according to claim 5, characterized in that it uses the guided filtering model to predict the initial filtering parameters of the input image, upsamples the initial filtering parameters, and obtains filtering parameters that match the resolution of the image to be processed.

7. A device for inverse rendering of images, The system is configured to input an image to be processed into a feature prediction model, predict the geometric features and material features of the image to be processed using the feature prediction model, and obtain a geometric feature map and a material feature map of the image to be processed, wherein the geometric feature map includes a normal map and a depth map, and the material feature map includes an albedo feature map, a roughness feature map and a metallicity feature map, A lighting prediction unit is configured to input the image to be processed, the geometric feature map, and the material feature map into a lighting prediction model, predict the illuminance value of the image to be processed for each pixel using the lighting prediction model, and obtain a lighting feature map of the image to be processed. The system includes an image processing unit configured to perform pre-set processing on the image to be processed based on the geometric feature map, the material feature map, and the illumination feature map, The aforementioned lighting prediction unit is A prediction module is configured to process the image to be processed, the geometric feature map, and the material feature map using the illumination prediction model, predict the illuminance value of a pixel point in the image to be processed, and generate a panoramic image corresponding to the pixel point based on the predicted illuminance value. An apparatus for inverse rendering of an image, characterized by including a stitching module configured to stitch panoramic images corresponding to pixel points in the image to be processed to obtain the illumination feature map.

8. The aforementioned device is A sample image is input into a pre-trained feature prediction model to predict the geometric and material features of the sample image, and a sample geometric feature map and a sample material feature map of the sample image are obtained. The sample image, the sample geometric feature map, and the sample material feature map are input into a pre-trained illumination prediction model to predict the illuminance values ​​of pixel points in the sample image and to obtain a sample illumination feature map of the sample image. Using the differentiable rendering module, a rendered image is generated based on the sample geometric feature map, the sample material feature map, and the sample illumination feature map. The apparatus according to claim 7, further comprising a model training unit configured to obtain the feature prediction model and the illumination prediction model by adjusting the parameters of the pre-trained feature prediction model and the pre-trained illumination prediction model based on the differences between the sample image and the rendered image until a pre-set training completion condition is met.

Images