Image processing method and device, equipment, storage medium and program product

Abstract

The invention relates to an image processing method and device, equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence. The method comprises the following steps: inputting a training sample into a prediction model to obtain a prediction inverse rendering result of an image sample included in the training sample; the prediction inverse rendering result is used for indicating the physical attribute of the object in the image sample; according to the predicted inverse rendering result, executing image rendering operation through a ray tracing rendering pipeline to obtain a rendered image; obtaining a first loss function value based on the difference between the image sample and the rendered image; performing parameter updating on the prediction model based on the first loss function value; wherein the trained prediction model is used for outputting an inverse rendering result of the target image included in the input data based on the input data; the inverse rendering result is used for indicating the physical attribute of the object in the target image. According to the invention, the accuracy of the inverse rendering result predicted by the prediction model can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, device, storage medium, and program product. Background Art

[0002] Inverse rendering technology refers to the technology of decomposing geometric, material, lighting and other analytical components from an image.

[0003] In related technologies, inverse rendering can be achieved using a pre-trained machine learning model. For example, developers pre-set two model branches. During model training, the first model branch is used to predict the analytical component from the image sample, and the second model branch is used to reconstruct the analytical component predicted by the first model branch into an image. Afterwards, the model parameters of the two model branches can be updated based on the difference between the image sample and the reconstructed image. During application, the first model branch can be used to predict the analytical component from the input target image.

[0004] However, the model training accuracy in the above solution is low, which leads to inaccurate analytical components predicted by the model. Summary of the Invention

[0005] This application provides an image processing method, apparatus, device, storage medium, and program product; the technical solution is as follows:

[0006] According to one aspect of the present application, there is provided an image processing method, the method comprising:

[0007] Inputting a training sample into a prediction model to obtain a predicted inverse rendering result of an image sample included in the training sample; the predicted inverse rendering result is used to indicate a physical property of an object in the image sample;

[0008] Performing an image rendering operation through a ray tracing rendering pipeline according to the predicted inverse rendering result to obtain a rendered image;

[0009] Obtaining a first loss function value based on a difference between the image sample and the rendered image;

[0010] Updating parameters of the prediction model based on the first loss function value;

[0011] The prediction model after training is used to output an inverse rendering result of the target image included in the input data based on the input data; the inverse rendering result is used to indicate the physical properties of the object in the target image.

[0012] According to one aspect of the present application, there is provided an image processing apparatus, the apparatus comprising:

[0013] A prediction module, configured to input a training sample into a prediction model to obtain a predicted inverse rendering result of an image sample included in the training sample; the predicted inverse rendering result is used to indicate a physical property of an object in the image sample;

[0014] A rendering module, configured to perform an image rendering operation through a ray tracing rendering pipeline according to the predicted inverse rendering result to obtain a rendered image;

[0015] A first loss acquisition module, configured to acquire a first loss function value based on a difference between the image sample and the rendered image;

[0016] A first parameter updating module, configured to update parameters of the prediction model based on the first loss function value;

[0017] The prediction model after training is used to output an inverse rendering result of the target image included in the input data based on the input data; the inverse rendering result is used to indicate the physical properties of the object in the target image.

[0018] In some embodiments, the rendering module is used to input the predicted inverse rendering result into the ray tracing rendering pipeline based on the Monte Carlo ray tracing algorithm for rendering to obtain the rendered image.

[0019] In some embodiments, the rendering module is configured to input the predicted inverse rendering result multiple times into the ray tracing rendering pipeline based on the Monte Carlo ray tracing algorithm for rendering to obtain multiple candidate rendering images;

[0020] The rendering module is used to obtain similarities between the plurality of candidate rendering images and the image sample respectively;

[0021] The rendering module is configured to determine the candidate rendering image with the highest similarity as the rendered image.

[0022] In some embodiments, the prediction model includes a diffusion model and a decoder; the training sample further includes a first random noise;

[0023] The prediction module is configured to input the image sample and the first random noise into the diffusion model to obtain a first predicted noise map;

[0024] The prediction module is configured to input the first predicted noise map into the decoder to obtain the predicted inverse rendering result.

[0025] In some embodiments, the first parameter updating module is configured to update parameters of the diffusion model and the decoder based on the first loss function value.

[0026] In some embodiments, the apparatus further comprises: a second loss acquisition module, configured to acquire a second loss function value based on a difference between a first noise map and the first predicted noise map; the first noise map being an image obtained by superimposing the first random noise on the image sample;

[0027] The device also includes: a second parameter updating module, configured to update the parameters of the prediction model based on the second loss function value.

[0028] In some embodiments, the first parameter updating module is configured to update parameters of the diffusion model based on the second loss function value.

[0029] In some embodiments, the training sample further includes a prompt word sample; the prompt word sample includes an identifier of a first parsed component, and the first parsed component is one or more of a plurality of parsed components;

[0030] The predicted inverse rendering result includes a predicted result of the first analytical component of the image sample.

[0031] In some embodiments, the prompt word sample further includes a first component feature of the image sample; the first component feature is used to indicate a feature of a first analytical component of the image sample; the apparatus further includes: a feature acquisition module for acquiring a labeling result of the first analytical component of the image sample before inputting the image sample and the first random noise into the diffusion model to obtain a first predicted noise map;

[0032] The first component feature is obtained based on the labeling result of the first parsed component of the image sample.

[0033] In some embodiments, the prediction model further comprises an encoder;

[0034] The feature acquisition module is configured to input the labeling result of the first parsed component of the image sample into the encoder to obtain the first component feature.

[0035] In some embodiments, the first parameter updating module is used to update the parameters of the encoder based on the first loss function value.

[0036] In some embodiments, the device also includes: a second parameter update module, used to update the parameters of the encoder based on a second loss function value; the second loss function value is obtained based on the difference between the first noise map and the first predicted noise map; the first noise map is an image obtained by superimposing the first random noise on the image sample.

[0037] In some embodiments, the input data further includes a second random noise, and the apparatus further includes: an inference application module configured to input the target image and the second random noise into the diffusion model to obtain a second predicted noise map;

[0038] The inference application model is used to input the second predicted noise map into the decoder to obtain the inverse rendering result of the target image.

[0039] In some embodiments, the input data further includes a target prompt word; the target prompt word includes an identifier of a second parsed component, and the second parsed component is one or more of a plurality of parsed components;

[0040] The inverse rendering result includes a prediction result of the second analytical component of the target image.

[0041] In some embodiments, the analytical components include at least one or more of the following: a base color map, a normal vector map, a metalness map, a roughness map, and an irradiance map.

[0042] According to one aspect of the present application, there is provided an image processing method, the method comprising:

[0043] Get input data;

[0044] Inputting the input data into a prediction model to obtain an inverse rendering result of a target image included in the input data; the inverse rendering result is used to indicate a physical property of an object in the target image;

[0045] In which, the prediction model is a machine learning model obtained by updating parameters through a first loss function value, and the first loss function value is a loss function value obtained based on the difference between the image sample and the rendered image; the rendered image is an image obtained by performing an image rendering operation through a ray tracing rendering pipeline based on the predicted inverse rendering result of the image sample, and the predicted inverse rendering result is used to indicate the physical properties of the object in the image sample; the predicted inverse rendering result is output by the prediction model after processing the training sample including the image sample.

[0046] According to one aspect of the present application, there is provided an image processing apparatus, the apparatus comprising:

[0047] Acquisition module, used to obtain input data;

[0048] A prediction module, configured to input the input data into a prediction model to obtain an inverse rendering result of a target image included in the input data; the inverse rendering result is used to indicate physical properties of an object in the target image;

[0049] In which, the prediction model is a machine learning model obtained by updating parameters through a first loss function value, and the first loss function value is a loss function value obtained based on the difference between the image sample and the rendered image; the rendered image is an image obtained by performing an image rendering operation through a ray tracing rendering pipeline based on the predicted inverse rendering result of the image sample, and the predicted inverse rendering result is used to indicate the physical properties of the object in the image sample; the predicted inverse rendering result is output by the prediction model after processing the training sample including the image sample.

[0050] In some embodiments, the prediction model includes a diffusion model and a decoder, and the input data further includes a second random noise;

[0051] The prediction module is configured to input the target image and the second random noise into the diffusion model to obtain a second predicted noise map;

[0052] The prediction module is configured to input the second predicted noise map into the decoder to obtain the inverse rendering result of the target image.

[0053] In some embodiments, the input data further includes a target prompt word; the target prompt word includes an identifier of a second parsed component, and the second parsed component is one or more of a plurality of parsed components;

[0054] The inverse rendering result includes a prediction result of the second analytical component of the target image.

[0055] According to another aspect of the present application, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the image processing method described above.

[0056] According to another aspect of the present application, a computer-readable storage medium is provided, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the image processing method described above.

[0057] According to another aspect of the present application, a computer program product is provided, which includes computer instructions stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the image processing method described above.

[0058] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0059] A solution is provided for using ray tracing to assist in training a prediction model. During the training process, after obtaining a predicted inverse rendering result of an image sample through the prediction model, the predicted inverse rendering result is rendered into an image through a ray tracing algorithm, and then the prediction model is updated based on the difference between the rendered image and the input image sample. The image rendering operation through the ray tracing rendering pipeline can more completely retain the information of the analytical components in the predicted inverse rendering result, especially can more completely retain the lighting-related information (such as specular highlight-related information) in the predicted inverse rendering result. Therefore, the rendered image is more closely matched with the predicted inverse rendering result, thereby improving the accuracy of the rendered image during the training process of the prediction model, ensuring the accuracy of the prediction model training, and further improving the accuracy of the inverse rendering result predicted by the prediction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0061] Figure 1 is a schematic diagram of an implementation environment of an image processing method provided by an exemplary embodiment of the present application;

[0062] Figure 2 is a flow chart of an image processing method provided by an exemplary embodiment of the present application;

[0063] Figure 3 is a flow chart of an image processing method provided by an exemplary embodiment of the present application;

[0064] Figure 4 is a schematic diagram of an analytical component provided by an exemplary embodiment of the present application;

[0065] Figure 5 is a flow chart of an image processing method provided by an exemplary embodiment of the present application;

[0066] Figure 6 is a flowchart of training and reasoning of a prediction model provided by an exemplary embodiment of the present application;

[0067] Figure 7 This is a complete pipeline diagram of a generative inverse rendering method based on ray tracing, shown in an exemplary embodiment of the present application;

[0068] Figure 8is a schematic diagram of a ray tracing module according to an exemplary embodiment of the present application;

[0069] Figure 9 is a block diagram of an image processing apparatus shown in an exemplary embodiment of the present application;

[0070] Figure 10 is a block diagram of an image processing apparatus shown in an exemplary embodiment of the present application;

[0071] Figure 11 It is a structural block diagram of a computer device shown in an exemplary embodiment of the present application.

[0072] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application. DETAILED DESCRIPTION

[0073] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0074] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0075] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0076] It should be understood that although the terms first, second, etc. may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter without departing from the scope of this disclosure. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0077] The following are some definitions of terms involved in this application:

[0078] 1) Rendering: The process of converting a 3D model or scene into a 2D image. This process includes, but is not limited to, modeling, texture mapping, lighting calculations, projection transformations, and viewpoint transformations. The goal of rendering is to produce a 2D image that is as realistic as possible so that users can view the 3D scene on devices such as monitors.

[0079] 2) Inverse Rendering: This is the reverse of the rendering process, aiming to extract the original information of a 3D model or scene, such as its geometry, material properties, and lighting conditions, from an existing image or video. Inverse rendering analyzes the image to infer the parameters of the 3D model or scene necessary to generate it. This is a complex task, and in most cases, inferring 3D information from a single 2D image presents multiple solutions.

[0080] 3) Ray Tracing: A rendering algorithm that simulates the propagation of real-world light. Widely used in computer graphics, it can produce images that closely resemble the real world. The core of ray tracing is to simulate the propagation path of real-world light in a virtual scene, thereby calculating the effects of light interacting with the surfaces of objects in the scene, such as refraction, reflection, and shadows.

[0081] Please refer to Figure 1 , which shows a schematic diagram of an implementation environment of an image processing method provided by an exemplary embodiment of the present application. The implementation environment may include: a terminal 110 and a server 120.

[0082] The terminal 110 includes, but is not limited to, a smartphone, a tablet computer, an intelligent voice interaction device, a smart home appliance, an in-vehicle terminal, a laptop or desktop computer, etc. The terminal 110 may run a client that provides image processing functions. The client may be an image processing application, an animation processing application, an instant messaging application, a music playback application, a reading application, etc. The specific type of the client is not limited in this embodiment of the application.

[0083] In the embodiment of the present application, the terminal 110 provides an image processing function client that can output a predicted analytical component of the target image based on the input target image.

[0084] Exemplarily, the terminal 110 can obtain the target image 101; thereafter, the terminal 110 can input the target image 101 into the trained component prediction model to obtain the predicted analytical component 102 of the target image 101 output by the component prediction model; the analytical component 102 is one or more components of geometry, material, and lighting decomposed from the target image 101, which can indicate the physical properties of the object in the target image 101.

[0085] Server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, as well as big data and artificial intelligence platforms.

[0086] In the embodiment of the present application, server 120 is a backend server that provides a client-side image processing function in terminal 110. Optionally, server 120 can output predicted analytical components of the target image based on an input target image; or, server 120 can train a component prediction model using image samples, and the component prediction model is used to output predicted analytical components of the target image based on the input target image.

[0087] Exemplarily, the server 120 can obtain an image sample 103; thereafter, the server 120 can input the image sample 103 into a component prediction model to be trained to obtain a predicted analytical component 104 of the image sample 103 output by the component prediction model, where the analytical component 104 is one or more components of geometry, material, and lighting decomposed from the image sample 103, and can indicate the physical properties of the object in the image sample 103; thereafter, the server 120 can perform an image rendering operation on the predicted analytical component 104 of the image sample 103 through a ray tracing rendering pipeline to obtain a rendered image 105; thereafter, the server 120 can obtain a first loss function value 106 based on the difference between the image sample 103 and the rendered image 105; thereafter, the server 120 can update the parameters of the component prediction model based on the first loss function value 106.

[0088] Optionally, the training process of the component prediction model may also be performed by the terminal 110 .

[0089] The terminal 110 and the server 120 communicate data via a communication network. Optionally, the communication network may be a wired network or a wireless network, and the communication network may be at least one of a local area network, a metropolitan area network, and a wide area network.

[0090] Please refer to Figure 2, which shows a flow chart of an image processing method provided by an exemplary embodiment of the present application. The method is executed by a computer device, which may be Figure 1 The server 120 or terminal 110 in the system shown; or, the computer device can also be other electronic devices with computing capabilities. Figure 2 As shown, the method may include the following steps 210 , 220 , 230 and 240 .

[0091] Step 210: Input the training samples into the prediction model to obtain the predicted inverse rendering results of the image samples included in the training samples; the predicted inverse rendering results are used to indicate the physical properties of the objects in the image samples.

[0092] The image samples may be images from a training sample used to train the prediction model. The predicted inverse rendering results may be graphs indicating analytical components of the physical properties of objects in the image. For example, the physical properties of objects in the image may include, but are not limited to, one or more of the following:

[0093] 1) Geometric attributes: used to describe the shape and surface features of an object, such as the shape, size, position, posture, etc.; specifically, the geometric structure can include the normal vector of the object's surface.

[0094] 2) Material properties: used to describe the physical properties of an object's surface, which can affect the object's appearance under different lighting conditions; specifically, material properties can include base color, metallicity, roughness, etc.

[0095] 3) Lighting properties: These describe the effects of a light source's lighting parameters on an object. These parameters can include the light source's position, intensity, color, and direction. For example, the light source's position affects the contrast between light and dark areas on an object's surface, as well as the position of specular highlights. Another example is the light source's intensity, which affects the brightness of specular highlights. A stronger light source reflects more light energy from the object's surface, resulting in brighter specular highlights. Conversely, a weaker light source reduces the brightness of specular highlights. Specifically, lighting properties can include irradiance.

[0096] Optionally, the above-mentioned image sample may include annotated inverse rendering results, that is, the physical properties of the object in the image sample are known; or, the above-mentioned image sample may not include annotated inverse rendering results, that is, the physical properties of the object in the image sample are unknown.

[0097] Step 220: Based on the predicted inverse rendering result, perform image rendering operations through the ray tracing rendering pipeline to obtain a rendered image.

[0098] The ray tracing rendering pipeline can completely preserve the scene's lighting-related information by simulating the path of light propagation in the scene, thereby generating highly realistic images. Exemplarily, the scene's lighting-related information includes light source information (such as light source type, position, color, etc.), specular reflection information (such as specular reflection coefficient, specular highlights, etc.), ambient lighting (such as ambient lighting color and intensity), and other information, which is not limited in this application.

[0099] Exemplarily, the ray tracing rendering pipeline can be implemented as a complete ray tracing rendering pipeline that conforms to the rendering equation. A complete ray tracing rendering pipeline refers to a process that implements image rendering entirely through a ray tracing algorithm. For example, the complete ray tracing rendering pipeline can be a ray tracing pipeline that starts from a viewpoint (such as a virtual camera) and calculates the pixel value of each pixel in the image by tracing the path of rays to generate a rendered image.

[0100] For example, the ray tracing rendering pipeline can be a rendering pipeline based on full ray tracing. Full ray tracing simulates the complete propagation path of light in a three-dimensional scene, accurately calculates the interaction between light and objects, and can completely preserve lighting-related information in the scene (such as information related to specular highlights), thereby generating extremely realistic image effects.

[0101] It should be noted that the above-mentioned ray tracing rendering pipeline can also be implemented as other types of ray tracing rendering pipelines, and this application does not limit this.

[0102] The above rendering equation is based on physical principles and can simulate natural lighting phenomena to generate realistic images. For example, the rendering equation can be expressed as the following integral form:

[0103] L o (p,ω o )=L e (p,ω o )+∫ Ω f r (p,ω i ,ω o )L i (p,ω i )(ω i ·n)dω i

[0104] Among them, L o (p,ω o ) represents the direction from point p to ω o The observed radiance, L e (p,ω o) indicates that point p is in the direction ω o The radiant brightness emitted on the r (p,ω i ,ω o ) is the bidirectional reflectance distribution function (BRDF), which describes the reflection from the direction ω i To direction ω o The reflection characteristics, L i (p,ω i ) is the point p in the direction ω i The brightness of the incident radiation received on i n is the direction of the incident light ω i The dot product with the surface normal n at point p indicates whether the incident light is directly incident on the surface, ∫ Ω Indicates that in all possible incident directions ω i The integral over Ω is all directions on the unit sphere.

[0105] Step 230: Obtain a first loss function value based on the difference between the image sample and the rendered image.

[0106] For example, the computer device may input the above-mentioned image sample and the rendered image into a preset first loss function to obtain the above-mentioned first loss function value output by the loss function.

[0107] Exemplarily, the first loss function may include one or more of the following loss functions:

[0108] 1) Absolute Error:

[0109] Also known as L1 loss, it is used to calculate the average of the absolute difference in pixel values ​​between two images (such as a rendered image and an image sample). The smaller the absolute error, the more similar the two images are. For example, the calculation formula of L1 loss is as follows:

[0110]

[0111] in, is the value of the absolute error, y i is the true value of each pixel in the image, is the predicted value of each pixel in the image, and N is the number of pixels in the image.

[0112] 2) Mean Squared Error (MSE):

[0113] Also known as L2 loss, it is used to calculate the average of the squared difference in pixel values ​​between two images (such as a rendered image and an image sample). The smaller the mean square error, the more similar the two images are. For example, the calculation formula of L2 loss is as follows:

[0114]

[0115] in, is the mean square error, y i is the true value of each pixel in the image, is the predicted value for each pixel in the image, and n is the number of pixels in the image.

[0116] 3) Structural Similarity Index (SSIM):

[0117] SSIM is a metric that measures the structural similarity between two images (such as a rendered image and an image sample), including brightness, contrast, and structural information. The SSIM value ranges from -1 to 1, with the closer it is to 1, the more similar the two images are.

[0118] Step 240: Update the parameters of the prediction model based on the first loss function value.

[0119] The trained prediction model is used to output an inverse rendering result of a target image included in the input data based on the input data.

[0120] During initialization, the developer can set the parameters of the prediction model to random initial values. The computer device can then update the parameters of the prediction model based on the obtained first loss function value. For example, the computer device can update the parameters of the prediction model using a backpropagation algorithm based on the first loss function value to improve the prediction performance of the prediction model.

[0121] For example, the computer device may repeat steps 210, 220, 230, and 240 to iteratively update and train the prediction model until a preset condition of the prediction model is met. For example, the developer may set the preset condition to a specified number of training iterations or a specified convergence condition (e.g., the accuracy of the model reaches a certain threshold).

[0122] In the embodiment of the present application, the trained prediction model can be used to predict the inverse rendering result of the target image. Specifically, the computer device can input the target image to be predicted into the above prediction model to obtain the inverse rendering result of the target image output by the prediction model.

[0123] For example, the inverse rendering results of the target image can be applied to operations such as relighting, image editing, and image synthesis on the target image to enhance the image's realism and naturalness. Relighting refers to changing the lighting conditions in the target image; image editing refers to modifying the target image to achieve a specific visual effect, such as replacing materials or adjusting colors; and image synthesis refers to combining multiple objects from different target images to form a new image.

[0124] In summary, the solution of the embodiment of the present application provides a solution for assisting in training a prediction model through ray tracing. During the training process, after obtaining the predicted inverse rendering result of the image sample through the prediction model, the predicted inverse rendering result is rendered into an image through a ray tracing algorithm, and then the prediction model is updated based on the difference between the rendered image and the input image sample. The image rendering operation through the ray tracing rendering pipeline can more completely retain the information of the analytical component in the predicted inverse rendering result, especially the lighting-related information (such as specular highlight-related information) in the predicted inverse rendering result. Therefore, the rendered image is more closely matched with the predicted inverse rendering result, thereby improving the accuracy of the rendered image during the training process of the prediction model, ensuring the accuracy of the prediction model training, and further improving the accuracy of the inverse rendering result predicted by the prediction model.

[0125] Compared with existing technologies, this solution does not require training multiple models. Instead, it combines machine learning and ray tracing technologies to train a prediction model, which can reduce the amount of computation and complexity during training.

[0126] Based on the above Figure 2 In the solution in the embodiment shown, in a possible implementation solution, the above step 220 can be implemented as step 220a.

[0127] Step 220a: Input the predicted inverse rendering result into a ray tracing rendering pipeline based on the Monte Carlo ray tracing algorithm for rendering to obtain a rendered image.

[0128] Among them, the above-mentioned Monte Carlo ray tracing algorithm is a ray tracing algorithm based on probability statistics. It can estimate the propagation path of light in the image scene through random sampling, thereby calculating the final lighting effect, and then can handle complex lighting phenomena, such as global illumination, specular highlight components, multiple bounces of light and ambient occlusion.

[0129] Optionally, the ray tracing rendering pipeline based on the Monte Carlo ray tracing algorithm can be implemented as a Monte Carlo ray tracing module. In other words, the computer can input the predicted inverse rendering result into the Monte Carlo ray tracing module to obtain the rendered image.

[0130] However, because the Monte Carlo ray tracing algorithm uses random sampling, the generated rendered image contains noise, such as grainy or speckled visual effects. Therefore, in order to improve the image quality of the rendered image, the computer device can perform noise reduction processing on the rendered image obtained by the ray tracing rendering pipeline based on the Monte Carlo ray tracing algorithm to reduce the noise in the rendered image. Exemplarily, the computer device can implement noise reduction processing of the rendered image through a denoiser. That is, the computer device can input the rendered image into the denoiser to obtain a rendered image after noise reduction processing, thereby improving the image quality of the rendered image.

[0131] The embodiments of the present application provide a feasible solution for obtaining a rendered image, which may specifically include: rendering the image through a Monte Carlo ray tracing algorithm, which can completely retain the lighting information in the analytical components, such as the specular highlight component, thereby improving the quality of the rendered image.

[0132] Based on the solutions shown in the above embodiments of the present application, the above step 220a can be implemented as follows:

[0133] The predicted inverse rendering results are input into the ray tracing rendering pipeline based on the Monte Carlo ray tracing algorithm for rendering multiple times to obtain multiple candidate rendering images;

[0134] Obtaining the similarity between multiple candidate renderings and the image sample;

[0135] The candidate rendering image with the highest similarity is determined as the rendered image.

[0136] The similarity between the multiple candidate renderings and the image sample can be obtained by one or more of the following methods: mean square error (MSE) and structural similarity index (SSIM). The execution methods of mean square error (MSE) and structural similarity index (SSIM) are described above and will not be repeated here.

[0137] Optionally, the developer can pre-set a threshold for the similarity between the candidate renderings and the image sample. Accordingly, determining the candidate rendering with the highest similarity as the rendering result can be implemented by determining the candidate rendering with the highest similarity, and whose similarity is not less than the threshold, as the rendering result. If the similarity between multiple candidate renderings and the image sample is less than the threshold, the computer device retrieves multiple candidate renderings.

[0138] In this embodiment of the present application, since the Monte Carlo ray tracing algorithm is a ray tracing algorithm based on random sampling technology, in order to obtain a rendered image with a high degree of similarity to the image sample, the computer device may render the predicted inverse rendering result multiple times to obtain multiple candidate rendering images. The computer device may then determine the candidate rendering image with the highest similarity to the image sample as the rendered image, thereby obtaining a rendered image with a high degree of similarity to the image sample. Optionally, the computer device may also perform noise reduction processing on the rendered image to obtain a rendered image with higher image quality.

[0139] Based on the above Figure 2 In the embodiment shown, in a possible implementation, the prediction model includes a diffusion model and a decoder, and the training sample also includes a first random noise. Figure 3 , which shows a flow chart of an image processing method provided by an exemplary embodiment of the present application. Figure 3 As shown, the above step 210 can be implemented as step 210a and step 210b.

[0140] Step 210a: Input the image sample and the first random noise into the diffusion model to obtain a first predicted noise map.

[0141] The diffusion model is a generative model that defines a forward diffusion process from data to noise and a reverse generation process from noise back to data to learn the data distribution. In the forward diffusion process, the original data is gradually added with noise and becomes unrecognizable. For example, the original data sample X0 is gradually added with Gaussian noise to obtain a series of noise samples {X1, X2...X T}, where X T The state is almost completely noisy, T is the number of diffusion steps, and noise is added in each diffusion step according to a predefined probability distribution (such as Gaussian distribution). In the reverse generation process, the noise of the noise sample is gradually removed to restore data similar to the original data. For example, the diffusion model can learn how to convert X to T Gradually restore to X0. After the training is completed, a new image can be generated by forward diffusion and reverse generation process for an image and a random noise.

[0142] In an embodiment of the present application, the computer device can gradually add noise to the image sample through the forward diffusion process of the diffusion model, so that the original image sample is gradually masked by the noise, and obtain a predicted noise map (i.e., the first predicted noise map mentioned above).

[0143] The first random noise and the noise added in the forward diffusion process of the diffusion model may be Gaussian noise.

[0144] Step 210b: Input the first predicted noise map into the decoder to obtain a predicted inverse rendering result.

[0145] The decoder can be a neural network model. Exemplarily, the decoder can include an input layer, a hidden layer, and an output layer. The input layer is used to receive the first predicted noise map output by the diffusion model; the hidden layer can gradually increase the size of the first predicted noise map through a deconvolution layer, and then extract features from the first predicted noise map through a convolution layer; the output layer is used to further process the features extracted by the convolution layer to output a predicted inverse rendering result of the image sample.

[0146] The embodiment of the present application provides two components of a prediction model and the workflow of the two components. The diffusion model is used to gradually add noise to image samples, and the decoder is used to obtain the predicted inverse rendering results, which can improve the accuracy of the predicted inverse rendering results output by the prediction model.

[0147] Based on the above Figure 3 In the embodiment shown, in one possible implementation, step 240 may be implemented as follows:

[0148] Parameters of the diffusion model and the decoder are updated based on the first loss function value.

[0149] Since the first loss function value is a loss function value obtained based on image samples and rendered images, it can be used to measure the prediction performance of the diffusion model and decoder. Therefore, the computer device can update the parameters of the diffusion model and decoder according to the first loss function value to optimize the performance of the diffusion model and decoder.

[0150] Based on the above Figure 3 In the solution in the embodiment shown, in a possible implementation solution, the above image processing method may further include step 250 and step 260.

[0151] Step 250: Obtain a second loss function value based on a difference between the first noise map and the first predicted noise map; the first noise map is an image obtained by superimposing the first random noise on the image sample;

[0152] Step 260: Update the parameters of the prediction model based on the second loss function value.

[0153] The first predicted noise map is a noise map obtained by gradually adding noise to the image samples using the diffusion model, and the first noise map is a noise map obtained by directly adding noise to the image samples. The smaller the difference between the first predicted noise map and the first noise map, the better the prediction performance of the diffusion model.

[0154] For example, the computer device may input the first noise map and the first predicted noise map into a preset second loss function to obtain the second loss function value output by the loss function. For example, the second loss function and the first loss function may be the same loss function or different loss functions.

[0155] For example, the developer can set the weights corresponding to the first loss function value and the second loss function value in the computer device to define the importance of the two loss function values, so that the computer device can update the parameters of the prediction model according to the weights of the two loss function values.

[0156] Exemplarily, the computer device may repeat the above steps 210, 220, 230, 240, 250, and 260 for iterative updating and training until the preset conditions of the prediction model are met.

[0157] In an embodiment of the present application, the computer device can compare the first predicted noise map with the first noise map to obtain a second loss function value, and then the computer device can update the parameters of the prediction model based on the two loss function values ​​of the second loss function value. The above scheme uses the difference between the first predicted noise map output by the diffusion model and the original input noise map as a constraint, which can avoid large deviations in the process of the diffusion model generating the noise map, thereby improving the accuracy of the extended model and further improving the accuracy of the prediction of the prediction model.

[0158] Based on the solutions shown in the above embodiments of the present application, in a possible implementation solution, the above step 260 can be implemented as follows:

[0159] The parameters of the diffusion model are updated based on the second loss function value.

[0160] Since the second loss function value is a loss function value obtained based on the first noise map and the first predicted noise map, it can be used to measure the prediction performance of the diffusion model. Therefore, the computer device can update the parameters of the diffusion model according to the second loss function value to optimize the performance of the diffusion model.

[0161] Based on the solutions shown in the above embodiments of the present application, in a possible implementation solution, the training sample further includes a prompt word sample; the prompt word sample includes an identifier of a first parsed component, and the first parsed component is one or more of a plurality of parsed components;

[0162] The predicted inverse rendering result includes a predicted result of the first parsing component of the image sample.

[0163] As described above, the predicted inverse rendering result may indicate various physical properties of the object in the image sample, such as the geometric structure, material properties, and lighting effects; accordingly, the first analytical component may correspond to one or more of these physical properties.

[0164] Optionally, the identifier of the first parsed component may be a component name corresponding to one or more parsed components; accordingly, the identifier of the first parsed component may indicate the number of parsed components in the predicted inverse rendering result.

[0165] Exemplarily, the identifier of the first parsed component may include one or more of an identifier of a geometric structure component, an identifier of a material attribute component, and an identifier of a lighting impact component. For example, if the identifier of the first parsed component is an identifier of a material attribute component, the computer device may accordingly obtain the parsed component corresponding to the material attribute. For another example, if the identifier of the first parsed component is a geometric structure component and a material attribute classification, the computer device may accordingly obtain the parsed components corresponding to the geometric structure and material attributes. In other words, based on the identifier of the first parsed component of the input prompt word sample, the computer device may obtain the parsed component corresponding to the identifier of the first parsed component.

[0166] In an embodiment of the present application, the output of the decoder is a complete data structure containing multiple parsed components. When the computer device obtains the first parsed component, it can obtain one or more parsed sub-components corresponding to the identifier of the first parsed component based on the identifier of the first parsed component contained in the prompt word sample.

[0167] Please refer to Figure 4 , which shows a schematic diagram of the analytical component provided by an exemplary embodiment of the present application. Figure 4 As shown, taking the five analytical components of base color, normal vector, metalness, roughness, and irradiance as an example, the output of the decoder is five fixed sub-data, and the five sub-data correspond to the five analytical components respectively.

[0168] Case ①: The identifier of the first analytical component contained in the prompt word sample is the identifier of the three components of base color, normal vector, and metalness; accordingly, the computer device can extract the three analytical components of base color, normal vector, and metalness from the output of the decoder.

[0169] Case ②: The identifier of the first parsed component contained in the prompt word sample is the identifier of the roughness component; accordingly, the computer device can extract the parsed component corresponding to the roughness from the output of the decoder.

[0170] Case ③: The identifier of the first analytical component contained in the prompt word sample is the identifier of the five components of base color, normal vector, metalness, roughness, and irradiance; accordingly, the computer device can extract the five analytical components of base color, normal vector, metalness, roughness, and irradiance from the output of the decoder.

[0171] It should be noted that, when the first analytical component includes multiple analytical components, during step 220, the computer device can perform image rendering operations according to one analytical component in turn; or, the computer device can also perform image rendering operations according to the complete multiple analytical components.

[0172] According to the solution of the embodiment of the present application, during the training process, the prediction model can, based on the indication information of the prompt word sample, specifically process and output the predicted inverse rendering result corresponding to the indication information; accordingly, when the prediction model is applied for inference, the user can input the specified prompt word into the prediction model so that the prediction model can output the predicted inverse rendering result desired by the user; this solution can not only improve the training efficiency of the prediction model, but also improve the flexibility of the prediction model when applying inference.

[0173] Based on the solutions shown in the above-mentioned embodiments of the present application, in a possible implementation scheme, the prompt word sample also includes the first component feature of the image sample; the first component feature is a feature used to indicate the first analytical component of the image sample; before the above-mentioned step 210a, the image processing method also includes steps 202 and 204.

[0174] Step 202: Obtain the annotation result of the first analytical component of the image sample.

[0175] Step 204: Obtain first component features based on the annotation results of the first analytical component of the image sample.

[0176] The annotation result of the first analytical component of the image sample can be the actual value or expected value of the first analytical component of the image sample. The purpose of training the prediction model is to narrow the difference between the prediction result of the first analytical component of the image sample and the annotation result of the first analytical component of the image sample.

[0177] Because the annotation results of the first analytical component of the image sample are high-dimensional image data, to reduce computational complexity, the computer device can extract features from the annotation results of the first analytical component of the image sample using a dimensionality reduction method to obtain low-dimensional first component features. For example, the computer device can obtain the first component features using an encoder, singular value decomposition, or other methods.

[0178] That is to say, the first component feature is a low-dimensional feature obtained by performing dimensionality reduction on the labeling result of the first analytical component of the image sample.

[0179] In the solution of the embodiment of the present application, the prompt word sample may also include the features of the first analytical component of the image sample. The computer device reduces the dimension of the annotation results of the first analytical component of the image sample to obtain low-dimensional first component features, and then uses the first component features as training data for the diffusion model and the decoder; on the one hand, during the training process, by adding the features of the analytical component to be predicted to the prompt word sample, the diffusion model can better process the features related to the analytical component to be extracted, thereby ensuring the accuracy of the trained model; on the other hand, the low-dimensional analytical component features can reduce the computational complexity of the diffusion model and the decoder during training, thereby improving the training efficiency of the diffusion model and the decoder.

[0180] Based on the solutions shown in the above embodiments of the present application, in a possible implementation, the prediction model further includes an encoder; the above step 202 can be implemented as follows:

[0181] The annotation result of the first parsed component of the image sample is input into the encoder to obtain the first component feature.

[0182] Among them, the encoder can convert the input image into a low-dimensional implicit feature to capture the essential characteristics of the image.

[0183] In an embodiment of the present application, the encoder may be a neural network model, specifically including an input layer, a hidden layer, and an output layer. The input layer is configured to receive the annotation results of the first analytical component of the image sample, the hidden layer may perform feature extraction and dimensionality reduction on the annotation results of the first analytical component of the image sample, and the output layer may further process the features extracted and reduced by the hidden layer to output a low-dimensional feature vector, namely, the first component features.

[0184] The solution of the embodiment of the present application can reduce the dimension of the annotation results of the first analytical component of the image sample through the encoder, which can reduce the computational complexity of the diffusion model and the decoder, thereby improving the generalization ability of the prediction model.

[0185] Based on the solutions shown in the above embodiments of the present application, in a possible implementation solution, the above step 240 can be implemented as follows:

[0186] The encoder parameters are updated based on the first loss function value.

[0187] In an embodiment of the present application, during the model training process, the computer device can also update the above-mentioned encoder based on the difference between the image sample and the rendered image, so that during the training process, the accuracy of the component features contained in the prompt word sample can be gradually improved, thereby ensuring the prompt effect of the prompt word sample on the extended model, and thereby improving the accuracy of the model training.

[0188] Based on the solutions shown in the above embodiments of the present application, in a possible implementation solution, the above image processing method further includes:

[0189] The encoder parameters are updated based on a second loss function value; the second loss function value is obtained based on the difference between the first noise map and the first predicted noise map; the first noise map is an image obtained by superimposing a first random noise on the image sample.

[0190] In an embodiment of the present application, during the model training process, the computer device can also update the above-mentioned encoder based on the difference between the first noise map and the first predicted noise map, so that during the training process, the accuracy of the component features contained in the prompt word sample can be gradually improved, thereby ensuring the prompt effect of the prompt word sample on the extended model, and thereby improving the accuracy of the model training.

[0191] Based on the solutions shown in the above embodiments of the present application, in a possible implementation scheme, the analytical components include at least one or more of the following: a base color map, a normal vector map, a metalness map, a roughness map, and an irradiance map.

[0192] The following describes the above analytical components:

[0193] Base Color Map: The pixel values ​​of each pixel in the base color map are the values ​​of the base colors corresponding to each pixel in the original image. Base colors refer to the colors of objects without additional lighting conditions. They are the inherent colors of the object's surface and are unaffected by ambient light.

[0194] Normal map: The pixel values ​​in the normal map are the values ​​of the normal vectors corresponding to each pixel in the original image. A normal vector is a vector perpendicular to a point on an object's surface and is used to determine the direction of light reflection at that point. In 3D modeling, each vertex can have a normal vector, and each polygon face can also have a normal vector assigned to it.

[0195] Metalness map: The pixel values ​​in the metalness map are the metalness values ​​corresponding to each pixel in the original image. Metalness is a scalar value that indicates whether an object's surface has a metallic texture. Surfaces with high metalness have higher reflectivity, and the color of the reflected light is more consistent with the color of the light source.

[0196] Roughness map: The pixel values ​​of each pixel in the roughness map represent the roughness values ​​corresponding to each pixel in the original image. Roughness describes the smoothness of an object's surface. Smooth surfaces produce specular reflections, while rough surfaces produce diffuse reflections. The higher the roughness value, the less smooth the surface.

[0197] Irradiance map: The pixel value of each pixel in the irradiance map is the irradiance value corresponding to each pixel in the original image. Irradiance is used to describe the luminous flux per unit area of ​​the object's surface.

[0198] The embodiments of the present application provide the specific content of the analytical components, that is, the prediction model can predict the base color map, normal vector map, metalness map, roughness map, and irradiance map from the image, thereby expanding the diversity of application scenarios of this solution.

[0199] Based on the solutions shown in the above embodiments of the present application, in a possible implementation solution, the input data further includes a second random noise, and the above image processing method further includes step 270 and step 280.

[0200] Step 270: Input the target image and the second random noise into the diffusion model to obtain a second predicted noise map;

[0201] Step 280: Input the second predicted noise map into the decoder to obtain an inverse rendering result of the target image.

[0202] That is to say, the diffusion model and decoder in the prediction model can be applied to the prediction of inverse rendering results.

[0203] The target image is an image for which an inverse rendering result is to be predicted. The second random noise may be Gaussian noise.

[0204] The embodiment of the present application provides an inference application process of the above-mentioned prediction model. Through the trained diffusion model and decoder, this solution can predict the inverse rendering result corresponding to the target image from the target image; this solution can reduce the computational complexity of the prediction model inference application.

[0205] Based on the solutions shown in the above embodiments of the present application, in a possible implementation solution, the input data further includes a target prompt word; the target prompt word includes an identifier of a second parsed component, and the second parsed component is one or more of a plurality of parsed components;

[0206] The inverse rendering result includes a prediction result of the second parsing component of the target image.

[0207] Optionally, the identifier of the second parsed component may be a component name corresponding to one or more parsed components; accordingly, the identifier of the second parsed component may indicate the number of parsed components in the inverse rendering result.

[0208] That is to say, when predicting the inverse rendering result of the target image, the specified target cue word can be input into the diffusion model, and the decoder can output the second parsing component corresponding to the target cue word according to the indication information of the target cue word, so that the prediction model can obtain one or more parsing components in a targeted manner during a prediction process. Different parsing components can be predicted through different prediction processes, thereby ensuring the reliability of each prediction process and then ensuring the accuracy of the prediction.

[0209] The solution of the embodiment of the present application can improve the accuracy of the prediction model through target prompt words during the reasoning application process of the prediction model.

[0210] Please refer to Figure 5 , which shows a flow chart of an image processing method provided by an exemplary embodiment of the present application. The method is executed by a computer device, which can be implemented as a model application device; wherein, the model application device can be the same device as the computer device of steps 210 to 240 above, or it can be a different device. Optionally, the computer device can also Figure 1 The terminal 110 in the system shown; alternatively, the computer device may be Figure 1 Alternatively, the computer device may be any other electronic device with computing capabilities. Figure 5 As shown, the method may include the following steps 510 and 520.

[0211] Step 510: Obtain input data.

[0212] Step 520: Input the input data into the prediction model to obtain an inverse rendering result of the target image included in the input data; the inverse rendering result is used to indicate the physical properties of the object in the target image.

[0213] Among them, the prediction model is a machine learning model obtained by updating parameters through the first loss function value, and the first loss function value is a loss function value obtained based on the difference between the image sample and the rendered image; the rendered image is an image obtained by performing an image rendering operation through a ray tracing rendering pipeline based on the predicted inverse rendering result of the image sample, and the predicted inverse rendering result is used to indicate the physical properties of the object in the image sample; the predicted inverse rendering result is output after the prediction model processes the training sample including the image sample.

[0214] Referring to the above steps 270 and 280, the above prediction model includes a diffusion model and a decoder, which will not be described in detail here.

[0215] Regarding the above-mentioned reasoning method in the embodiment of the present application, the specific manner and technical effects of executing the operations in each step have been described in detail in the embodiment of the above-mentioned training method and will not be elaborated here.

[0216] Please refer to Figure 6 , which shows a training and reasoning flow chart of a prediction model provided by an exemplary embodiment of the present application. The prediction model includes an encoder A, a diffusion model B, and a decoder C.

[0217] like Figure 6 As shown in part (a) of the training process, the training process is as follows:

[0218] Step T1: Obtain an image sample 601 and a labeling result 602 of a first analytical component of the image sample 601;

[0219] Step T2: input the annotation result 602 of the first parsed sub-component of the image sample 601 into the encoder A to obtain the first component feature 603;

[0220] Step T3: input the image sample 601, the first random noise 604, and the prompt word sample 605 including the first component feature 603 and the identifier of the first analytical component into the diffusion model B to obtain a first predicted noise map 606;

[0221] Step T4: input the first predicted noise map 606 into the decoder C to obtain the predicted inverse rendering result 607 of the image sample 601;

[0222] Step T5: input the predicted inverse rendering result 607 of the image sample 601 into the ray tracing module D to obtain a rendered image 608;

[0223] Step T6: Obtaining a first loss function value based on the difference between the image sample 601 and the rendered image 608;

[0224] Obtaining a second loss function value based on a difference between a first noise map obtained by superimposing the first random noise 604 on the image sample 601 and the first predicted noise map 606;

[0225] Step T7: Based on the first loss function value, update the parameters of encoder A, diffusion model B, and decoder C;

[0226] Based on the second loss function value, the parameters of the diffusion model B are updated.

[0227] like Figure 6 As shown in part (b) of the above, the reasoning process is as follows:

[0228] Step I1: Acquire a target image 610;

[0229] Step I2: Input the target image 610 , the second random noise 611 , and the target prompt word 612 into the diffusion model B to obtain a second predicted noise map 613 ;

[0230] Step I3: Input the second predicted noise map 613 into the decoder C to obtain an inverse rendering result 614 of the target image 610 .

[0231] Inverse rendering is a major challenge in the field of computer vision. Inverse rendering, simply put, is to recover the geometry, material, lighting, shading and other components from an image. The reason why inverse rendering is difficult is that the geometry, material, lighting, shading and other components are coupled. Specifically, an image can include several lighting effects, such as shadows, occlusion, global illumination, material shading, etc. Related technologies can decompose some of these components, such as decomposing the geometric components (such as the normal vector N) from the image; for material components, the base color A component can be decomposed. Of course, with the development of deep learning in recent years, the metalness M and roughness R components can also be decomposed.

[0232] In order to solve the problem of inverse rendering, an image processing system can be constructed based on the above embodiments of the present application. Figure 7 , which shows a complete pipeline diagram of a generative inverse rendering method based on ray tracing according to an exemplary embodiment of the present application. Figure 7 As shown, the system includes an encoder 710, a diffusion model 720, a decoder 730 and a ray tracing module 740. The system combines ray tracing with a generative model and can be applied to a generative inverse rendering method based on ray tracing. The method can extract geometric components, material components and lighting components from a single image, providing a basis for downstream applications (such as relighting, image editing, and image synthesis).

[0233] The complete pipeline of the above-mentioned ray tracing-based generative inverse rendering method is described as follows:

[0234] 1) Construct training samples:

[0235] The training sample consists of three parts: ① an input image I; ② the annotation results of the analytical components corresponding to the input image I. The analytical components include one or more of the base color (Albedo, A), normal vector (N), metallic map (M), roughness map (R), and irradiance map (E). The analytical components are also called ground truth (GT value); ③ the component name ∈ {"Albedo", "Normal", "Metallic", "Roughness", "Irradiance"} corresponding to the input image I. The above component names correspond one-to-one to the above analytical components.

[0236] 2) Training:

[0237] During the training process, the computer device can encode the GT value through the encoder 710, that is, the analytical component corresponding to the input image is input into the encoder 710. After processing by the encoder 710, the encoder 710 outputs the reduced-dimensional implicit feature z, which is a vector.

[0238] Afterwards, the computer device can send the input image, the first random noise, and the text prompt word (Prompt, P) to the diffusion model 720 to be pre-trained. After processing by the diffusion model 720, the diffusion model 720 outputs a first predicted noise map; wherein the text prompt word serves as a switch, and the text prompt word contains the above-mentioned implicit feature z and the component name corresponding to the parsed component.

[0239] Afterwards, the computer device can input the first predicted noise map into the decoder 730. After processing by the decoder 730, the decoder 730 outputs the predicted inverse rendering result of the input image obtained by training decoding. The predicted inverse rendering result includes the predicted results of one or more analytical components corresponding to the component names. For example, the predicted inverse rendering result obtained by training decoding is one or more of a base color map, a normal vector map, a metalness map, a roughness map, and an irradiance map.

[0240] The purpose of training is to make the difference between the predicted inverse rendering result output by the decoder 730 and the input GT images smaller and smaller. The training process is constrained by the L2 loss between the first predicted noise map and the original noise map, that is, Figure 7 The loss function Lm term in . Among them, the original noise map is the noise map obtained by adding the first random noise to the input image.

[0241] At the same time, the computer device can add a ray tracing module 740 at the end of the decoder 730. The predicted inverse rendering result output by the decoder 730 is fed into the ray tracing module 740. After processing by the ray tracing module 740, the ray tracing module 740 outputs a rendered image. The ray tracing module 740 is a complete ray tracing pipeline that conforms to the rendering equation.

[0242] The gap between the rendered image and the input image is constrained by the L2 loss, which constructs the second loss function, namely the above Figure 7 Therefore, the loss function can be expressed as L = Lm + ωLr, where ω is a hyperparameter.

[0243] Specifically, this method describes the illumination including the irradiance map. Therefore, this method proposes to add a ray tracing module at the end of the pipeline to supplement the illumination results, making the final generated image more realistic and as close to the input image as possible. Figure 8, which shows a schematic diagram of a ray tracing module shown in an exemplary embodiment of the present application. Figure 8 As shown, the input of the ray tracing module is the predicted inverse rendering result output by the decoder 730, such as the predicted base color, predicted normal vector, etc. The ray tracing module includes a Monte Carlo ray tracing module 7401 and a denoiser 7402. The Monte Carlo ray tracing method is a physically based method that can simulate relatively accurate lighting results. However, due to insufficient sampling, it often generates noise. Therefore, the ray tracing result image is sent to the denoiser 7402 for image noise reduction, ultimately producing a rendered image.

[0244] 3) Reasoning Application:

[0245] During the inference application stage, the computer device can input the target image to be predicted, the second random noise, and the target prompt word into the diffusion model 720 after training. After processing by the diffusion model 720, the diffusion model 720 outputs a second predicted noise map; wherein the target prompt word is used to indicate the component name corresponding to the parsed component.

[0246] Afterwards, the computer device may input the second predicted noise map into the decoder 730 after training. After being processed by the decoder 730, the decoder 730 outputs one or more parsed components corresponding to the target prompt word.

[0247] For example, the above-mentioned generative inverse rendering method based on ray tracing can fine-tune an open-source diffusion model in related technologies. The trained diffusion model in related technologies is used for image generation, so the implicit variable layer of the diffusion model in related technologies is a 4-dimensional vector. In the above-mentioned training process of this method, the features learned by the implicit variable layer of the generative model are the various features decomposed from the input image, such as the base color (a∈R h*w*3 ), normal vector (n∈R h *w*3 ), metallicity (m∈R h*w ), roughness (r∈R h*w ), irradiance (E∈R h*w*3 ); where h and w represent height and width, respectively, and R is a commonly accepted set of real numbers. As can be seen, the dimensions of each feature are different, and the lengths of the implicit feature z output by the encoder vary. This method uses the text prompt as a switch, allowing switching within the same diffusion model without fine-tuning five diffusion models individually. In other words, this method maximizes the capabilities of the diffusion model by fine-tuning an already open-source diffusion model using a specific set of data. This reduces training complexity and improves the model's representational power and generalization.

[0248] above Figure 7The complete pipeline shown here combines a generative model with a ray tracing inverse rendering pipeline. This approach is a generative inverse rendering method combined with ray tracing. By adding a ray tracing module at the end of the pipeline, specular highlights can be added, improving the lighting estimation. Fine-tuning the diffusion model allows for even better decomposition. The ray tracing module is used only for training and not for lighting estimation; its purpose is to fine-tune the network's weights.

[0249] Please refer to Figure 9 , which shows a block diagram of an image processing device shown in an exemplary embodiment of the present application, which can be used to perform the following Figure 2 or Figure 3 In the method shown, all or part of the steps performed by the computer device are as follows Figure 9 As shown, the device includes:

[0250] The prediction module 901 is used to input the training samples into the prediction model to obtain the predicted inverse rendering results of the image samples included in the training samples; the predicted inverse rendering results are used to indicate the physical properties of the objects in the image samples;

[0251] A rendering module 902 is configured to perform an image rendering operation through a ray tracing rendering pipeline according to the predicted inverse rendering result to obtain a rendered image;

[0252] A first loss acquisition module 903 is configured to acquire a first loss function value based on a difference between the image sample and the rendered image;

[0253] A first parameter updating module 904 is configured to update parameters of the prediction model based on the first loss function value;

[0254] The trained prediction model is used to output an inverse rendering result of a target image included in the input data based on the input data; the inverse rendering result is used to indicate the physical properties of the object in the target image.

[0255] In some embodiments, the rendering module 902 is configured to input the predicted inverse rendering result into a ray tracing rendering pipeline based on a Monte Carlo ray tracing algorithm for rendering to obtain a rendered image.

[0256] In some embodiments, the rendering module 902 is configured to input the predicted inverse rendering results multiple times into a ray tracing rendering pipeline based on a Monte Carlo ray tracing algorithm for rendering to obtain multiple candidate rendering images;

[0257] A rendering module 902 is configured to obtain similarities between a plurality of candidate rendering images and the image sample;

[0258] The rendering module 902 is configured to determine the candidate rendering image with the highest similarity as the rendered image.

[0259] In some embodiments, the prediction model includes a diffusion model and a decoder; the training sample further includes a first random noise;

[0260] Prediction module 901, configured to input an image sample and a first random noise into a diffusion model to obtain a first predicted noise map;

[0261] The prediction module 901 is configured to input the first predicted noise map into a decoder to obtain a predicted inverse rendering result.

[0262] In some embodiments, the first parameter updating module 904 is configured to update parameters of the diffusion model and the decoder based on the first loss function value.

[0263] In some embodiments, the apparatus further includes: a second loss acquisition module, configured to acquire a second loss function value based on a difference between the first noise map and the first predicted noise map; the first noise map is an image obtained by superimposing the first random noise on the image sample;

[0264] The device also includes: a second parameter updating module, used to update the parameters of the prediction model based on the second loss function value.

[0265] In some embodiments, the first parameter updating module 904 is configured to update parameters of the diffusion model based on the second loss function value.

[0266] In some embodiments, the training sample further includes a prompt word sample; the prompt word sample includes an identifier of a first parsed component, and the first parsed component is one or more of a plurality of parsed components;

[0267] The predicted inverse rendering result includes a predicted result of the first parsing component of the image sample.

[0268] In some embodiments, the prompt word sample further includes a first component feature of the image sample; the first component feature is used to indicate a feature of the first analytical component of the image sample; the apparatus further includes: a feature acquisition module for acquiring a labeling result of the first analytical component of the image sample before inputting the image sample and the first random noise into the diffusion model to obtain the first predicted noise map;

[0269] A first component feature is obtained based on a labeling result of the first parsed component of the image sample.

[0270] In some embodiments, the prediction model further comprises an encoder;

[0271] The feature acquisition module is used to input the annotation result of the first analytical component of the image sample into the encoder to obtain the first component feature.

[0272] In some embodiments, the first parameter updating module 904 is configured to update the parameters of the encoder based on the first loss function value.

[0273] In some embodiments, the device also includes: a second parameter update module, used to update the parameters of the encoder based on a second loss function value; the second loss function value is obtained based on the difference between the first noise map and the first predicted noise map; the first noise map is an image obtained by superimposing the first random noise on the image sample.

[0274] In some embodiments, the input data further includes a second random noise, and the apparatus further includes: an inference application module, configured to input the target image and the second random noise into the diffusion model to obtain a second predicted noise map;

[0275] The inference application model is used to input the second predicted noise map into the decoder to obtain an inverse rendering result of the target image.

[0276] In some embodiments, the input data further includes a target prompt word; the target prompt word includes an identifier of a second parsed component, and the second parsed component is one or more of a plurality of parsed components;

[0277] The inverse rendering result includes a prediction result of the second parsing component of the target image.

[0278] In some embodiments, the analytical components include at least one or more of the following: a base color map, a normal vector map, a metalness map, a roughness map, and an irradiance map.

[0279] Please refer to Figure 10 , which shows a block diagram of an image processing device shown in an exemplary embodiment of the present application, which can be used to perform the following Figure 5 In the method shown, all or part of the steps performed by the computer device are as follows Figure 10 As shown, the device includes:

[0280] Acquisition module 1001, used to acquire input data;

[0281] Prediction module 1002, configured to input input data into a prediction model and obtain an inverse rendering result of a target image included in the input data; the inverse rendering result is used to indicate physical properties of an object in the target image;

[0282] Among them, the prediction model is a machine learning model obtained by updating parameters through the first loss function value, and the first loss function value is a loss function value obtained based on the difference between the image sample and the rendered image; the rendered image is an image obtained by performing an image rendering operation through a ray tracing rendering pipeline based on the predicted inverse rendering result of the image sample, and the predicted inverse rendering result is used to indicate the physical properties of the object in the image sample; the predicted inverse rendering result is output after the prediction model processes the training sample including the image sample.

[0283] In some embodiments, the prediction model includes a diffusion model and a decoder, and the input data further includes a second random noise;

[0284] The prediction module 1002 is configured to input the target image and the second random noise into the diffusion model to obtain a second predicted noise map;

[0285] The prediction module 1002 is configured to input the second predicted noise map into a decoder to obtain an inverse rendering result of the target image.

[0286] In some embodiments, the prediction module 1002 is configured to receive input data further including a target prompt word; the target prompt word includes an identifier of a second parsed component, and the second parsed component is one or more of a plurality of parsed components;

[0287] The inverse rendering result includes a prediction result of the second parsing component of the target image.

[0288] It should be noted that the device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example to implement its functions. In actual applications, the above-mentioned functions can be assigned to different functional modules according to actual needs, that is, the content structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0289] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method; the technical effects achieved by each module performing operations are the same as the technical effects in the embodiment of the method, and will not be elaborated here.

[0290] Please refer to Figure 11, which shows a structural block diagram of a computer device shown in an exemplary embodiment of the present application. The computer device can be implemented as the server in the above-mentioned solution of the present application. The computer device 1100 includes a central processing unit (CPU) 1101, a system memory 1104 including a random access memory (RAM) 1102 and a read-only memory (ROM) 1103, and a system bus 1105 connecting the system memory 1104 and the central processing unit 1101. The computer device 1100 also includes a mass storage device 1106 for storing an operating system 1109, application programs 1110, and other program modules 1111.

[0291] The mass storage device 1106 is connected to the central processing unit 1101 via a mass storage controller (not shown) connected to the system bus 1105. The mass storage device 1106 and its associated computer-readable media provide non-volatile storage for the computer device 1100. In other words, the mass storage device 1106 may include a computer-readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0292] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electronically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage medium is not limited to the above-mentioned ones. The above-mentioned system memory 1104 and mass storage device 1106 can be collectively referred to as memory.

[0293] According to various embodiments of the present disclosure, the computer device 1100 may also be connected to a remote computer on a network such as the Internet for operation. That is, the computer device 1100 may be connected to the network 1108 via the network interface unit 1107 connected to the system bus 1105, or the network interface unit 1107 may be used to connect to other types of networks or remote computer systems (not shown).

[0294] The memory further includes at least one computer program, which is stored in the memory. The central processing unit 1101 implements all or part of the steps in the methods shown in the above embodiments by executing the at least one computer program.

[0295] In an exemplary embodiment, a chip is further provided. The chip includes a programmable logic circuit and / or program instructions. When the chip is run on a computer device, it is used to implement the image processing method in the above aspect.

[0296] In an exemplary embodiment, a computer program product is also provided. The computer program product includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to implement the image processing methods provided in the above-described method embodiments.

[0297] In an exemplary embodiment, a computer-readable storage medium is further provided. The computer-readable storage medium stores a computer program. The computer program is loaded and executed by a processor to implement the image processing methods provided by the above-mentioned method embodiments.

[0298] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0299] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0300] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Inputting a training sample into a prediction model to obtain a predicted inverse rendering result of an image sample included in the training sample; the predicted inverse rendering result is used to indicate a physical property of an object in the image sample; Performing an image rendering operation through a ray tracing rendering pipeline according to the predicted inverse rendering result to obtain a rendered image; Obtaining a first loss function value based on a difference between the image sample and the rendered image; Updating parameters of the prediction model based on the first loss function value; The prediction model after training is used to output an inverse rendering result of the target image included in the input data based on the input data; the inverse rendering result is used to indicate the physical properties of the object in the target image.

2. The method according to claim 1, characterized in that The step of performing an image rendering operation through a ray tracing rendering pipeline according to the predicted inverse rendering result to obtain a rendered image includes: The predicted inverse rendering result is input into the ray tracing rendering pipeline based on the Monte Carlo ray tracing algorithm for rendering to obtain the rendered image.

3. The method according to claim 2, characterized in that Inputting the predicted inverse rendering result into a Monte Carlo ray tracing algorithm for rendering to obtain the rendered image includes: Inputting the predicted inverse rendering results multiple times into the ray tracing rendering pipeline based on the Monte Carlo ray tracing algorithm for rendering to obtain multiple candidate rendering images; Obtaining similarities between a plurality of candidate renderings and the image sample respectively; The candidate rendering image with the highest similarity is determined as the rendered image.

4. The method according to claim 1, wherein The prediction model includes a diffusion model and a decoder; the training sample also includes a first random noise; The step of inputting the training samples into the prediction model to obtain the predicted inverse rendering results of the image samples included in the training samples includes: Inputting the image sample and the first random noise into the diffusion model to obtain a first predicted noise map; The first predicted noise map is input into the decoder to obtain the predicted inverse rendering result.

5. The method according to claim 4, characterized in that The updating of parameters of the prediction model based on the first loss function value includes: Parameters of the diffusion model and the decoder are updated based on the first loss function value.

6. The method according to claim 4, characterized in that The method further comprises: Obtaining a second loss function value based on a difference between a first noise map and the first predicted noise map, wherein the first noise map is an image obtained by superimposing the first random noise on the image sample; Parameters of the prediction model are updated based on the second loss function value.

7. The method according to any one of claims 4 to 6, characterized in that: The training sample further includes a prompt word sample; the prompt word sample includes an identifier of a first parsed component, and the first parsed component is one or more of a plurality of parsed components; The predicted inverse rendering result includes a predicted result of the first analytical component of the image sample.

8. The method according to claim 7, characterized in that The prompt word sample further includes a first component feature of the image sample; the first component feature is used to indicate a feature of a first analytical component of the image sample; Before inputting the image sample and the first random noise into the diffusion model to obtain the first predicted noise map, the method further includes: Obtaining a labeling result of a first analytical component of the image sample; The first component feature is obtained based on the labeling result of the first parsed component of the image sample.

9. The method according to claim 8, characterized in that The prediction model also includes an encoder; The obtaining of the first component feature based on the annotation result of the first parsed component of the image sample includes: The labeling result of the first parsed component of the image sample is input into the encoder to obtain the first component feature.

10. The method according to claim 9, characterized in that The method further comprises: The encoder parameters are updated based on a second loss function value; the second loss function value is obtained based on the difference between the first noise map and the first predicted noise map; the first noise map is an image obtained by superimposing the first random noise on the image sample.

11. The method according to any one of claims 4 to 6, characterized in that: The input data further includes second random noise, and the method further includes: Inputting the target image and the second random noise into the diffusion model to obtain a second predicted noise map; The second predicted noise map is input into the decoder to obtain the inverse rendering result of the target image.

12. An image processing method, characterized in that: The method comprises: Get input data; Inputting the input data into a prediction model to obtain an inverse rendering result of a target image included in the input data; the inverse rendering result is used to indicate a physical property of an object in the target image; In which, the prediction model is a machine learning model obtained by updating parameters through a first loss function value, and the first loss function value is a loss function value obtained based on the difference between the image sample and the rendered image; the rendered image is an image obtained by performing an image rendering operation based on a ray tracing rendering pipeline according to the predicted inverse rendering result of the image sample, and the predicted inverse rendering result is used to indicate the physical properties of the object in the image sample; the predicted inverse rendering result is output by the prediction model after processing the training sample including the image sample.

13. An image processing device, characterized in that: The device comprises: A prediction module, configured to input a training sample into a prediction model to obtain a predicted inverse rendering result of an image sample included in the training sample; the predicted inverse rendering result is used to indicate a physical property of an object in the image sample; A rendering module, configured to perform an image rendering operation through a ray tracing rendering pipeline according to the predicted inverse rendering result to obtain a rendered image; A first loss acquisition module, configured to acquire a first loss function value based on a difference between the image sample and the rendered image; A first parameter updating module, configured to update parameters of the prediction model based on the first loss function value; The prediction model after training is used to output an inverse rendering result of the target image included in the input data based on the input data; the inverse rendering result is used to indicate the physical properties of the object in the target image.

14. An image processing device, characterized in that: The device comprises: Acquisition module, used to obtain input data; A prediction module, configured to input the input data into a prediction model to obtain an inverse rendering result of a target image included in the input data; the inverse rendering result is used to indicate physical properties of an object in the target image; In which, the prediction model is a machine learning model obtained by updating parameters through a first loss function value, and the first loss function value is a loss function value obtained based on the difference between the image sample and the rendered image; the rendered image is an image obtained by performing an image rendering operation through a ray tracing rendering pipeline based on the predicted inverse rendering result of the image sample, and the predicted inverse rendering result is used to indicate the physical properties of the object in the image sample; the predicted inverse rendering result is output by the prediction model after processing the training sample including the image sample.

15. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer instruction, and the at least one computer instruction is loaded and executed by the processor to implement the image processing method according to any one of claims 1 to 12.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer instruction, and the computer instruction is loaded and executed by a processor to implement the image processing method according to any one of claims 1 to 12.

17. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; the computer instructions are read and executed by a processor of a computer device to implement the image processing method according to any one of claims 1 to 12.

Images