Image rendering method and apparatus, and related device

Abstract

Disclosed are an image rendering method and apparatus, and a related device, which belong to the technical field of computer vision. In the method, a plurality of first images of a first rendering object are input into a first implicit object model training network, so as to generate a first implicit object model of the first rendering object; rendering pose information is received; the rendering pose information is input into the first implicit object model, so as to obtain first rendering data of the first rendering object; light source configuration information is determined; and illumination treatment is performed on the first rendering data on the basis of the light source configuration information, so as to obtain a target image. The implicit object model provided in the embodiments of the present application can output rendering data that matches a geometry buffer in a rendering pipeline, and therefore an output of the implicit object model is combined with a rendering pipeline for an explicit object model. Thus, the rendering effect in a scenario where a light source environment dynamically changes is improved, while the realism of rendering is ensured.

Description

Image rendering method, device and related equipment Technical Field

[0001] The embodiments of the present application relate to the field of computer vision technology, and in particular to an image rendering method, apparatus, and related equipment. Background Art

[0002] In computer vision technology, a three-dimensional object model for a certain rendering object can be constructed first, and then a two-dimensional image of the rendering object under a certain rendering perspective position can be rendered using the three-dimensional object model.

[0003] In the related art, the pre-built three-dimensional object model can be an implicit object model, such as a neural radiance field (NeRF) model. When a certain rendering perspective pose is input into the implicit object model, the implicit object model can output a two-dimensional image corresponding to the rendering perspective pose. Among them, the relevant implicit object model renders the two-dimensional image according to the fixed light source information learned in advance when generating the two-dimensional image, resulting in the inability to modify the light source information during the image rendering process through the implicit object model, thereby making the rendering effect of this image rendering method not very ideal in scenes with dynamically changing light source environments. In addition, the way of rendering the implicit object model is quite different from the way of rendering the explicit object model.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide an image rendering method, apparatus, and related devices that can improve rendering effects in scenes with dynamically changing light source environments while ensuring rendering realism. The technical solution is as follows:

[0006] In a first aspect, an image rendering method is provided, in which a plurality of first images of a first rendering object are input into a first implicit object model training network to generate a first implicit object model of the first rendering object, the plurality of first images being images of the first rendering object in a plurality of first poses, the first implicit object model being a neural network model, and the first rendering object being located in a target scene; rendering pose information is received, the rendering pose information being used to indicate the pose of a rendering perspective of the target scene; the rendering pose information is input into the first implicit object model to obtain first rendering data of the first rendering object, the first rendering data including one or more of normal information, depth information, color information, and material information of the first rendering object under the rendering perspective; light source configuration information is determined, the light source configuration information being used to indicate the lighting conditions of the target scene; and lighting processing is performed on the first rendering data based on the light source configuration information to obtain a target image, the target image including an image of the first rendering object under the rendering perspective and lighting conditions.

[0007] For the sake of convenience, the implicit object model that directly outputs a two-dimensional image after inputting the rendering perspective pose without lighting processing is called the relevant implicit object model.

[0008] In the image rendering method provided in an embodiment of the present application, an implicit object model is generated by multiple images, and the implicit object model can output rendering data that matches the geometric buffer in the rendering pipeline, such as normal information, depth information, color information or material information, so that the rendering pipeline can perform lighting processing on the model based on the output. Since the light source configuration information in the lighting processing operation of the rendering pipeline can be flexibly set, the rendering effect in the scene with dynamically changing light source environment can be improved while ensuring the rendering realism. At the same time, the rendering process can be combined with the rendering pipeline for the explicit object model to improve the flexibility of rendering.

[0009] Based on the method provided in the first aspect, in one possible implementation, before inputting multiple first images into the first implicit object model training network, a first neural radiation field NeRF model is also obtained, and the first NeRF model is used to predict arbitrary perspective images of the first rendered object; based on the first NeRF model, multiple first images in multiple first poses are obtained.

[0010] In an embodiment of the present application, the first implicit object model can be trained using the output of the first NeRF model. This process can be referred to as an implicit model migration process. That is, the relevant implicit object model is migrated to an implicit object model that can output rendering data that matches the geometry buffer.

[0011] Based on the method provided in the first aspect, in one possible implementation, the first rendering data also includes initial light source information. In this method, a light source configuration interface is also provided; the initial light source information is displayed on the light source configuration interface; in response to the user's editing operation on the light source configuration interface, the light source information after adjusting the initial light source information is obtained, and the adjusted light source information is used as the light source configuration information.

[0012] In an embodiment of the present application, the user can edit the initial light source information in the target scene where the first object is located obtained by inferring the first implicit object model to adjust the initial light source information, thereby performing lighting processing operations based on the adjusted light source information, and further improving the rendering effect in scenes with dynamically changing light source environments.

[0013] Based on the method provided in the first aspect, in one possible implementation, the first rendering data also includes initial light source information. In this method, a light source configuration interface is also provided; the initial light source information is displayed on the light source configuration interface; and in response to the user's confirmation operation on the light source configuration interface, the initial light source information is determined as the light source configuration information.

[0014] In the embodiment of the present application, the initial light source information in the target scene where the first object is located can be directly inferred based on the first implicit object model to perform lighting processing operations, thereby improving the efficiency of image rendering.

[0015] Based on the method provided in the first aspect, in one possible implementation, in this method, an explicit object model of a second rendering object is further obtained, where the second rendering object is located in the target scene; based on the rendering pose information, the explicit object model is geometrically processed to obtain second rendering data of the second rendering object, where the second rendering data includes one or more of normal information, depth information, color information, and material information of the second rendering object.

[0016] Accordingly, the implementation method of performing lighting processing on the first rendering data based on the light source configuration information to obtain the target image can be: performing lighting processing on the first rendering data and the second rendering data based on the light source configuration information to obtain the target image, the target image including images of the first rendering object and the second rendering object under the rendering perspective and lighting conditions.

[0017] In an embodiment of the present application, when it is necessary to render the implicit object model and the explicit object model at the same time, the rendering data matching the geometry buffer can be obtained through the two object models respectively, and then the rendering data in the geometry buffer can be uniformly processed through the lighting processing operation in the rendering pipeline. Therefore, the embodiment of the present application also provides an image rendering method that highly integrates the rendering process of the implicit object model and the rendering process of the explicit object model.

[0018] Based on the method provided in the first aspect, in a possible implementation, in this method, multiple second images of the third rendering object are also input into a second implicit object model training network to generate a second implicit object model of the third rendering object, the multiple second images are images of the third rendering object in multiple second poses, the second implicit object model is a neural network model, and the third rendering object is located in the target scene; the rendering pose information is input into the second implicit object model to obtain third rendering data of the third rendering object, and the third rendering data includes one or more of normal information, depth information, color information and material information of the third rendering object in the rendering pose information.

[0019] Correspondingly, the implementation method of performing lighting processing on the first rendering data based on the light source configuration information to obtain the target image can be: performing lighting processing on the first rendering data and the third rendering data based on the light source configuration information to obtain the target image, the target image including images of the first rendering object and the third rendering object under the rendering perspective and lighting conditions.

[0020] In an embodiment of the present application, when multiple implicit object models need to be rendered simultaneously, rendering data that matches the geometry buffer can be obtained through each implicit object model separately, and then the rendering data in the geometry buffer can be uniformly processed through the lighting processing operation in the rendering pipeline, thereby further improving the flexibility of image rendering.

[0021] Based on the method provided in the first aspect, in a possible implementation, in this method, the implementation method of inputting multiple first images into the first implicit object model training network to generate the first implicit object model of the first rendering object can be: determining one or more of the normal information, depth information, color information and material information of each first image in the multiple first images; training the first implicit object model training network based on one or more of the normal information, depth information, color information and material information of each first image in the multiple first images to generate the first implicit object model.

[0022] After obtaining one or more of the normal information, depth information, color information and material information of each of the multiple first images, one or more of the normal information, depth information, color information and material information of each of the multiple first images can be used as supervision information to perform supervised training on the first implicit object model training network, so that the first implicit object model training network learns one of the normal information, depth information, color information and material information of the first rendered object under different viewing angles and poses during the training process, that is, the first implicit object model is trained.

[0023] In a second aspect, an image rendering device is provided, wherein the image rendering device has the function of implementing the image rendering method of the first aspect. The image rendering device includes at least one module, wherein the at least one module is used to implement the image rendering method of the first aspect.

[0024] In a third aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the image rendering method provided in the first aspect above.

[0025] In a fourth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the image rendering method provided in the first aspect.

[0026] In a fifth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the image rendering method provided in the first aspect.

[0027] The technical effects obtained in the above-mentioned second, third, fourth and fifth aspects are similar to those obtained by the corresponding technical means in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] FIG1 is a schematic diagram of the architecture of an image rendering system provided by an embodiment of the present application;

[0029] FIG2 is a schematic diagram of the architecture of another image rendering system provided in an embodiment of the present application;

[0030] FIG3 is a schematic diagram of the architecture of another image rendering system provided in an embodiment of the present application;

[0031] FIG4 is a flow chart of an image rendering method provided in an embodiment of the present application;

[0032] FIG5 is a schematic structural diagram of a first implicit object model provided in an embodiment of the present application;

[0033] FIG6 is a schematic diagram of a light source configuration interface provided in an embodiment of the present application;

[0034] FIG7 is a flow chart of another image rendering method provided by an embodiment of the present application;

[0035] FIG8 is a flow chart of another image rendering method provided by an embodiment of the present application;

[0036] FIG9 is a schematic structural diagram of an image rendering device provided in an embodiment of the present application;

[0037] FIG10 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0038] FIG11 is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application;

[0039] FIG12 is a schematic diagram of the architecture of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present application more clear, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0041] In computer graphics (CG) technology, three-dimensional (3D) object models are typically represented using explicit mesh models. A mature real-time rendering pipeline has been developed to render mesh models to produce a two-dimensional image at a specific rendering viewpoint. This rendering pipeline can generally be divided into a geometry processing stage and a lighting processing stage. In the geometry processing stage, color information, geometry information, and material information of the mesh model at a specific rendering viewpoint are calculated and stored in a set of buffers, also known as the geometry buffer (G-Buffer). In the lighting processing stage, the data cached in the geometry buffer is combined with light source information to calculate and generate the final rendering result, which is a two-dimensional image corresponding to the rendering viewpoint.

[0042] In recent years, NeRF technology has been increasingly studied and applied, enabling more realistic renderings. Unlike traditional CG techniques, object models such as those represented by NeRF are expressed using neural networks, with the object model structure stored in the neural network parameters. Therefore, these models are often referred to as implicit object models, while what-you-see-is-what-you-get (WYSIWYG) object models, such as mesh models, are often referred to as explicit object models.

[0043] The rendering pipeline for implicit object models is different from that for explicit object models. Typically, an implicit object model is trained for a specific object. Then, when a rendering viewpoint pose is directly fed into the implicit object model, it can output a 2D image corresponding to that viewpoint pose.

[0044] Among them, the method of training an implicit object model for a certain rendering object can be: first collect sample images of different perspectives for the rendering object, and determine the camera pose of each sample image. The camera pose can be understood as the pose of the camera that collects the sample image. The sample image and the corresponding camera pose are sent to the NeRF model for training, so that the NeRF model can learn the mapping relationship between the camera pose and the sample image. The trained NeRF model can then input the two-dimensional image of the rendering object under any rendering perspective pose.

[0045] For the convenience of subsequent explanation, the implicit object model that directly outputs a two-dimensional image after inputting the rendering view pose is called the relevant implicit object model.

[0046] Due to the inherent characteristics of neural network models, the rendering realism of the relevant implicit object model is higher than that of the explicit object model. However, after the relevant implicit object model learns the mapping relationship between camera pose and sample images during training, for example, after learning the mapping relationship between camera pose and the lighting conditions in the sample images, subsequent rendering of two-dimensional images is directly rendered and output based on this mapping relationship. This makes it impossible to modify the light source information during the rendering process of the relevant implicit object model, resulting in less than ideal rendering results in scenes with dynamically changing light sources.

[0047] Based on this, an embodiment of the present application provides an image rendering method. In the image rendering method provided by the embodiment of the present application, a new implicit object model is generated from multiple images. Compared with the relevant implicit object model, the implicit object model provided by the embodiment of the present application can output rendering data that matches the geometric buffer in the rendering pipeline, so as to combine the output of the implicit object model with the lighting processing operation of the rendering pipeline for the explicit object model. Because the light source configuration information in the lighting processing operation of the rendering pipeline can be flexibly set, the rendering effect in scenes with dynamically changing light source environments can be improved while ensuring rendering realism.

[0048] The implicit object models in the subsequent embodiments all refer to the new implicit object models provided in the embodiments of the present application, and will not be explained one by one in the following.

[0049] In addition, the image rendering method provided in the embodiments of the present application can be applied to cloud scenarios, such as autonomous driving scenarios. In such scenarios, the image rendering method is executed by a cloud platform, that is, the cloud platform provides cloud services to users, and the cloud services are used to implement the image rendering method provided in the embodiments of the present application. Optionally, the image rendering method provided in the embodiments of the present application can also be executed by a terminal device. The embodiments of the present application do not limit the application scenarios of the image rendering method.

[0050] The image rendering system provided in the embodiment of the present application is explained below.

[0051] FIG1 is a schematic diagram of the architecture of an image rendering system provided by an embodiment of the present application. As shown in FIG1 , the image rendering system 00 includes a pre-processing unit 01 , a rendering calculation unit 02 , and an editing and interaction unit 03 .

[0052] Among them, the preprocessing unit 01 is used to process data to obtain the implicit object model required by the embodiment of the present application.

[0053] In embodiments of the present application, the implicit object model can output data that matches the geometry buffer in the rendering pipeline. In other words, the implicit object model can output rendering data that can be cached in the geometry buffer. This rendering data includes one or more of the normal information, depth information, color information, and material information of the rendered object. This allows the implicit object model output to be flexibly processed by the lighting processing operations in the rendering pipeline to produce a rendering result.

[0054] The implicit object model can be obtained by training with sample images. In some embodiments, the sample images can be directly obtained from a related implicit object model such as a NeRF model. In this scenario, the process of obtaining the implicit object model provided by the embodiments of the present application can be referred to as a model migration process. In other words, the implicit object model required by the embodiments of the present application can be obtained by migrating a related implicit object model such as a NeRF model.

[0055] Optionally, the sample image may also be acquired through an image acquisition device, which will not be described in detail here.

[0056] The rendering calculation unit 02 is configured to output rendering data for a rendering object through an implicit object model and cache the rendering data for the rendering object in a geometry buffer. Alternatively, the rendering calculation unit 02 may output rendering data for each rendering object through multiple implicit object models and cache the rendering data for each rendering object in a geometry buffer. Alternatively, the rendering calculation unit 02 may also perform a rendering pipeline geometry processing operation on the explicit object model of a rendering object to obtain rendering data for the rendering object and cache the rendering data for the rendering object in a geometry buffer.

[0057] In other words, in an embodiment of the present application, when rendering a scene, regardless of whether the object model included in the scene is an implicit object model or an explicit object model, the rendering calculation unit 02 can obtain the rendering data corresponding to the rendering object, and cache the rendering data in the geometric buffer, and then render the rendering data of each rendering object in combination with the light source information to obtain the final rendering result.

[0058] The editing interaction unit 03 is used for human-computer interaction to provide light source information edited by the user, thereby realizing adjustment of the light source information.

[0059] Based on the image rendering system shown in FIG1 , the output of the implicit object model can be combined with the lighting processing operation in the rendering pipeline. On the one hand, it is possible to improve the rendering effect in scenes with dynamically changing light source environments while ensuring the realism of the rendering. On the other hand, when it is necessary to render the implicit object model and the explicit object model at the same time, the rendering data that matches the geometry buffer can be obtained through the two object models respectively, and then the rendering data in the geometry buffer can be uniformly processed through the lighting processing operation in the rendering pipeline. Therefore, the embodiment of the present application also provides an image rendering system that highly integrates the rendering process of the implicit object model and the rendering process of the explicit object model.

[0060] The system shown in FIG1 is described below by taking rendering an implicit object model and an explicit object model as an example.

[0061] FIG2 is a schematic diagram of the architecture of another image rendering system provided in an embodiment of the present application.

[0062] As shown in FIG2 , the preprocessing unit 01 can be used to migrate relevant implicit object models, such as the NeRF model, to the first implicit object model required by the embodiment of the present application, or to obtain the first implicit object model required by the embodiment of the present application through other means.

[0063] The rendering calculation unit 02 is configured to, on the one hand, infer (i.e., output) first rendering data corresponding to the rendering viewpoint pose using the first implicit object model. The first rendering data includes one or more of the normal information, color information, depth information, and material information of the first rendering object, and cache the first rendering data in a geometry buffer. The rendering calculation unit 02 is also configured to perform the rendering pipeline's geometry processing operations on the explicit object model to obtain second rendering data corresponding to the rendering viewpoint pose. The second rendering data includes one or more of the normal information, depth information, color information, and material information of the second rendering object, and cache the second rendering data in the geometry buffer.

[0064] The editing interaction unit 03 is used to obtain the light source configuration information edited by the user and send the light source configuration information to the rendering calculation unit 02. The user can directly edit the light source configuration information or adjust the light source configuration information based on the initial light source information inferred by the first implicit object model.

[0065] The rendering calculation unit 02 is further configured to perform a lighting processing operation on the rendering data of each rendering object in the geometry buffer based on the light source configuration information to obtain a final rendering result, which is a two-dimensional image including the first rendering object and the second rendering object.

[0066] The system shown in FIG1 is described below by taking rendering of two implicit object models as an example.

[0067] FIG3 is a schematic diagram of the architecture of another image rendering system provided in an embodiment of the present application.

[0068] As shown in Figure 3, the preprocessing unit 01 can be used to migrate relevant implicit object models such as the NerF model into the first implicit object model and the second implicit object model required by the embodiment of the present application, or obtain the first implicit object model and the second implicit object model required by the embodiment of the present application through other means.

[0069] The rendering calculation unit 02 is configured to, on the one hand, infer (i.e., output) first rendering data corresponding to the rendering view pose using the first implicit object model. The first rendering data includes one or more of normal information, depth information, color information, and material information of the first rendering object, and cache the first rendering data in the geometry buffer. The rendering calculation unit 02 is also configured to infer (i.e., output) third rendering data corresponding to the rendering view pose using the second implicit object model. The third rendering data includes one or more of normal information, depth information, color information, and material information of the third rendering object, and cache the third rendering data in the geometry buffer.

[0070] The editing and interaction unit 03 is used to obtain the light source configuration information edited by the user and send the light source configuration information to the rendering calculation unit 02. The user can directly edit the light source configuration information or adjust the light source configuration information based on the initial light source information inferred from the first implicit object model and / or the second implicit object model.

[0071] The rendering calculation unit 02 is further configured to perform a lighting processing operation on the rendering data of each rendering object in the geometry buffer based on the light source configuration information to obtain a final rendering result, which is a two-dimensional image including the first rendering object and the third rendering object.

[0072] It should be noted that Figures 2 and 3 are used for illustration purposes. In the embodiment of the present application, the image rendering system 00 can render a larger or smaller number of object models, which will not be illustrated one by one here.

[0073] In addition, the image rendering system 00 can be exemplified as a cloud platform. In this scenario, the image rendering system 00 can be implemented by one or more computing devices. The pre-processing unit 01, the rendering computing unit 02, and the editing and interaction unit 03 can be integrated on the same computing device or deployed on different computing devices, which is not limited in this embodiment of the present application.

[0074] The image rendering method provided in the embodiment of the present application is explained below.

[0075] Figure 4 is a flow chart of an image rendering method provided by an embodiment of the present application. As shown in Figure 4, the method includes the following steps.

[0076] Step 401: Input multiple first images of a first rendering object into a first implicit object model training network to generate a first implicit object model of the first rendering object, where the multiple first images are images of the first rendering object in multiple first poses, the first implicit object model is a neural network model, and the first rendering object is located in a target scene.

[0077] The target scene may be any scene to be rendered, for example, the target scene includes all objects on a road in an autonomous driving scene and surrounding buildings, etc. The first rendering object may be understood as one of the objects in the target scene, for example, the first rendering object is a vehicle.

[0078] In order to implement the image rendering method provided in the embodiments of the present application, it is necessary to first train the implicit object model required by the embodiments of the present application in step 401. In some embodiments, training the implicit object model may be implemented by: determining one or more of normal information, depth information, color information, and material information for each of the multiple first images; and training a first implicit object model training network based on one or more of the normal information, depth information, color information, and material information for each of the multiple first images to generate the first implicit object model.

[0079] Among them, after obtaining one or more of the normal information, depth information, color information and material information of each of the multiple first images, one or more of the normal information, depth information, color information and material information of each of the multiple first images can be used as supervision information to perform supervised training on the first implicit object model training network, so that the first implicit object model training network learns one of the normal information, depth information, color information and material information of the first rendered object under different viewing angles and poses during the training process, that is, the first implicit object model is trained.

[0080] Each first image illustratively includes a red-green-blue (RGB) image and / or a depth image corresponding to the corresponding first pose.

[0081] In this case, the multiple first images can be output, for example, via an implicit object model associated with the first rendering object, i.e., an image-output implicit object model. In this scenario, the image-output implicit object model also outputs a first pose corresponding to the first image each time it outputs a first image. Based on this, an exemplary implementation method for obtaining the multiple first images corresponding to the multiple first poses can be: obtaining a first NeRF model, where the first NeRF model is used to predict arbitrary-viewpoint images of the first rendering object; and obtaining, based on the first NeRF model, the multiple first images corresponding to the multiple first poses, respectively.

[0082] In an embodiment of the present application, the first implicit object model can be trained by the output of the first NeRF model, and this process can be called an implicit model migration process. That is, the relevant implicit object migration model, i.e., the image output type implicit object model, is migrated to an implicit object model that can output rendering data that matches the geometric buffer. Optionally, in other embodiments provided in the present application, the same input can be used as the relevant implicit object migration model, i.e., multiple images of rendered objects under different perspectives can be input to generate the implicit object model provided in the embodiment of the present application.

[0083] Optionally, multiple first images can also be acquired through an image acquisition device, etc. In this scenario, the first position corresponding to each first image can be understood as the position of the image acquisition device that acquires the first image, that is, the camera position.

[0084] In addition, determining one or more of the normal information, depth information, color information and material information of each of the multiple first images can be achieved, for example, by inverse rendering or the like. Inverse rendering is used to infer scene parameters from a two-dimensional image, and the scene parameters include, for example, the lighting conditions of the scene, the color information, material information and geometric information of the objects in the scene. In the forward rendering process, scene parameters are input to generate (i.e., output) an image. However, in the inverse rendering process, these scene parameters need to be inferred from the image. Inverse rendering involves multiple operations, including photometric estimation, material restoration and geometric reconstruction. The embodiments of the present application do not limit the specific implementation method of inverse rendering.

[0085] Optionally, in the embodiment of the present application, after acquiring multiple first images, one or more of the normal information, depth information, color information, and material information of each first image can be acquired through other methods. For example, a user can manually annotate one or more of the normal information, depth information, color information, and material information of each first image. However, this implementation method requires a lot of human resources.

[0086] It should be noted that the above-mentioned process of training to obtain the first implicit object model is used for illustration. The embodiment of the present application does not limit the method of how to train to obtain the first implicit object model. Any implementation method that can train to obtain the first implicit object model can be applied to the embodiment of the present application.

[0087] In addition, the training process of the first implicit object model can be performed offline on training resources such as a graphic processing unit (GPU). The embodiment of the present application does not limit the device for training the first implicit object model.

[0088] Figure 5 is a schematic diagram of the structure of a first implicit object model provided in an embodiment of the present application. As shown in Figure 5, the first implicit object model includes multiple network layers, and each rectangular block in Figure 5 represents a group of network layers. As shown in Figure 5, some network layers are used for geometric expression, such as normal expression and depth expression, to output geometric information including normal information and depth information. Some network layers are used for appearance expression, such as material expression and color expression, to output material information and color information of the first object. Some network layers are used for lighting expression to output initial light source information.

[0089] Each group of network layers may include a fully connected layer, etc. The embodiment of the present application does not limit the specific internal structure of the first implicit object model, and any neural network model that can output the color information, geometric information, and material information of the first object can be applied in the embodiment of the present application.

[0090] Step 402: Receive rendering pose information, where the rendering pose information is used to indicate the pose of the rendering perspective of the target scene.

[0091] The rendering perspective pose of the target scene can be understood as the pose of the virtual camera used to render the target scene. For the convenience of subsequent description, the rendering perspective pose can be referred to as the rendering perspective pose.

[0092] In some embodiments, the rendering pose information can be specified by the user. For example, the user can set the rendering pose information through the editing interaction unit in the image rendering system shown in Figure 1. Alternatively, the rendering pose information can also be obtained through other methods, and the embodiments of the present application do not limit the method for obtaining the rendering pose information.

[0093] In addition, the rendering perspective pose indicated by the rendering pose information can be one or more. When there are multiple rendering perspective poses, the embodiments of the present application can generate rendering results corresponding to each rendering perspective pose. The subsequent embodiments will be described using one rendering perspective pose as an example.

[0094] Step 403: Input the rendering pose information into the first implicit object model to obtain first rendering data of the first rendering object, where the first rendering data includes one or more of normal information, depth information, color information, and material information of the first rendering object under the rendering perspective.

[0095] The implicit object model provided in the embodiment of the present application can output normal information, depth information, color information, material information and other information of the corresponding rendering object, so as to facilitate the subsequent combination of the output of the implicit object model with the lighting processing operation in the rendering pipeline.

[0096] The color information of the first rendering object in the first rendering data can be understood as the color information of the first rendering object when the first rendering object is observed from the rendering perspective. The color information can include the color of each pixel on a two-dimensional plane corresponding to the rendering perspective.

[0097] Normal information and / or depth information may be collectively referred to as geometric information.

[0098] The geometric information of the first rendering object in the first rendering data can be understood as: the geometric information of the first rendering object observed when the first rendering object is observed from the rendering perspective pose. The geometric information may include, for example, normal information of the first rendering object and / or depth information of the first rendering object. The normal information may include a normal vector of each pixel point on the two-dimensional plane corresponding to the rendering perspective pose, and the normal vector is a normal vector at the position point of the first rendering object corresponding to each pixel point. The depth information may include the depth of each pixel point on the two-dimensional plane corresponding to the rendering perspective pose, and the depth is the distance between the position point of the first rendering object corresponding to each pixel point and the rendering camera (i.e., the observer).

[0099] The material information of the first rendering object in the first rendering data can be understood as the material information of the first rendering object observed when the first rendering object is viewed from the rendering perspective. The material information may, for example, include the material texture of each pixel on the two-dimensional plane corresponding to the rendering perspective.

[0100] Optionally, the first implicit object model may also output initial light source information, which may indicate relevant information about the light source in the target scene where the first rendered object is located. This relevant information may, for example, include the location of the light source, the emission direction of the light source, the illumination intensity of the light source, and the color of the light source. In this way, subsequent illumination processing operations can be performed based on this initial light source information during the illumination processing phase. In some embodiments, this initial light source information may be represented using spherical harmonics.

[0101] Since the first rendering data includes normal information, depth information, color information, material information, and initial light source information of the first rendering object, after obtaining the first rendering perspective in step 403, the first rendering data can be cached in a geometry buffer configured for the rendering pipeline.

[0102] Step 404: Determine light source configuration information, where the light source configuration information is used to indicate the lighting conditions of the target scene.

[0103] In some embodiments, the first rendering data also includes initial light source information. In other words, when the first implicit object model infers one or more of the normal information, depth information, color information, and material information of the first rendered object, it also infers initial light source information for the target scene in which the first rendered object resides. In this case, light source configuration information can be determined based on the initial light source information, increasing flexibility in determining light source configuration information.

[0104] For example, the implementation method of determining the light source configuration information based on the initial light source information can be: providing a light source configuration interface; displaying the initial light source information on the light source configuration interface, and in response to the user's editing operation on the light source configuration interface, obtaining the light source information after adjusting the initial light source information, and using the adjusted light source information as the light source configuration information.

[0105] In an embodiment of the present application, the user can edit the initial light source information in the target scene where the first object is located obtained by inferring the first implicit object model to adjust the initial light source information, thereby performing lighting processing operations based on the adjusted light source information, and further improving the rendering effect in scenes with dynamically changing light source environments.

[0106] Among them, the light source configuration interface can be an interface provided by the editing interaction unit of the system shown in Figure 1. Figure 6 is a schematic diagram of a light source configuration interface provided by an embodiment of the present application. As shown in Figure 6, the light source configuration interface displays a virtual light source generated based on the initial light source information. The virtual light source includes virtual light source 1 and virtual light source 2. The user can adjust the position of the virtual light source by dragging one of the virtual light sources. The lighting direction and light intensity of the virtual light source can also be adjusted through the input box on the right.

[0107] As another example, determining the light source configuration information based on the initial light source information may be accomplished by: providing a light source configuration interface; displaying the initial light source information on the light source configuration interface; and determining the initial light source information as the light source configuration information in response to a user confirmation operation on the light source configuration interface. Alternatively, the initial light source information may be directly used as the light source configuration information without requiring a user confirmation operation.

[0108] In this scenario, the initial light source information of the target scene where the first object is located can be directly inferred based on the first implicit object model to perform lighting processing operations, thereby improving the efficiency of image rendering.

[0109] Optionally, in other embodiments, before performing the lighting processing operation of the rendering pipeline on the first rendering data based on the light source configuration information, a light source configuration interface may be displayed; and the light source configuration information may be obtained in response to user operations on the light source configuration interface.

[0110] That is, in the embodiment of the present application, the user can also directly set the light source configuration information without relying on the first implicit object model to determine the light source configuration information, which further improves the flexibility of determining the light source configuration information.

[0111] Step 405: Perform lighting processing on the first rendering data based on the light source configuration information to obtain a target image, where the target image includes an image of the first rendering object under the rendering perspective and lighting conditions.

[0112] During the lighting processing operation, the first rendering data in the geometry buffer is calculated based on the light source configuration information for the target scene to be rendered to obtain a first image. Because the light source configuration information can be flexibly configured, when the first image is rendered using the first implicit object model, rendering fidelity is not only ensured, but also rendering effects are improved in scenes with dynamically changing light source environments.

[0113] Among them, the lighting processing operation can be implemented by a shader for example, and the embodiment of the present application does not limit the implementation method of the lighting processing operation.

[0114] In summary, in the image rendering method provided in the embodiments of the present application, a new implicit object model is generated from multiple images. Compared to related implicit object models, the implicit object model provided in the embodiments of the present application can output rendering data that matches the geometry buffer in the rendering pipeline, thereby combining the output of the implicit object model with the lighting processing operations of the rendering pipeline for the explicit object model. Because the light source configuration information in the lighting processing operations of the rendering pipeline can be flexibly set, the rendering effect in scenes with dynamically changing light source environments can be improved while ensuring rendering realism.

[0115] In addition, when rendering an image of a scene, the scene may include multiple rendering objects. For example, in addition to the aforementioned first rendering object, it may also include other objects such as a second rendering object and / or a third rendering object. This is illustrated below using two embodiments.

[0116] Figure 7 is a flow chart of another image rendering method provided by an embodiment of the present application. As shown in Figure 7, the method includes the following steps.

[0117] Step 701: Input multiple first images of a first rendering object into a first implicit object model training network to generate a first implicit object model of the first rendering object, where the multiple first images are images of the first rendering object in multiple first poses, the first implicit object model is a neural network model, and the first rendering object is located in a target scene.

[0118] Step 702: Receive rendering pose information, where the rendering pose information is used to indicate the pose of the rendering perspective of the target scene.

[0119] Step 703: Input the rendering pose information into the first implicit object model to obtain first rendering data of the first rendering object, where the first rendering data includes one or more of normal information, depth information, color information, and material information of the first rendering object at a rendering perspective.

[0120] The implementation of steps 701 to 703 may refer to steps 401 to 403 in the embodiment of FIG. 4 , and will not be described in detail here.

[0121] Step 704: Obtain an explicit object model of a second rendering object, where the second rendering object is located in the target scene.

[0122] The explicit object model of the second rendering object can be input directly by the user through the editing interaction unit shown in Figure 1. Alternatively, the image rendering system shown in Figure 1 can also model an explicit object model using CG technology.

[0123] Step 705 : Based on the rendering pose information, geometrically process the explicit object model to obtain second rendering data of a second rendering object, where the second rendering data includes one or more of normal information, depth information, color information, and material information of the second rendering object.

[0124] In an embodiment of the present application, the object model for the second rendering object is an explicit object model. For example, the explicit object model may be a mesh model. Therefore, the explicit object model can be directly processed by geometry processing operations in the rendering pipeline to directly obtain second rendering data that matches the geometry buffer.

[0125] The geometry processing operations may include, for example, vertex shading, projection transformation, clipping transformation, screen mapping, etc. The embodiment of the present application does not limit the implementation of the geometry processing operations.

[0126] Since the second rendering data includes one or more of normal information, depth information, color information, and material information of the second rendering object, the second rendering data may be cached in a geometry buffer configured for the rendering pipeline.

[0127] Step 706: Determine light source configuration information, where the light source configuration information is used to indicate the lighting conditions of the target scene.

[0128] The implementation of the light source configuration information may refer to step 404 in the embodiment of FIG. 4 , and will not be described in detail here.

[0129] Step 707: performing lighting processing on the first rendering data and the second rendering data based on the light source configuration information to obtain a target image, where the target image includes images of the first rendering object and the second rendering object under the rendering perspective and lighting conditions.

[0130] Through step 707 , a target image including both the first rendering object and the second rendering object can be obtained by rendering.

[0131] In addition, when performing lighting processing operations of the rendering pipeline on the first rendering data and the second rendering data based on the light source configuration information, the depth information of the first rendering object and the depth information of the second rendering object can be combined to analyze the occlusion relationship between the first rendering object and the second rendering object in the target image to render a target image with higher realism.

[0132] In summary, in an embodiment of the present application, when it is necessary to render the implicit object model and the explicit object model at the same time, the rendering data matching the geometry buffer can be obtained through the two object models respectively, and then the rendering data in the geometry buffer can be uniformly processed through the lighting processing operation in the rendering pipeline. Therefore, the embodiment of the present application also provides an image rendering method that highly integrates the rendering process of the implicit object model and the rendering process of the explicit object model.

[0133] It should be noted that the serial numbers of the steps in the embodiment of Figure 7 are only used to mark the steps and do not represent the execution order of the steps. When applying the embodiment of this application, the execution order of certain steps can be adjusted according to actual conditions, or certain steps can be executed in parallel.

[0134] FIG8 is a flow chart of another image rendering method provided by an embodiment of the present application. As shown in FIG8 , the method includes the following steps.

[0135] Step 801: Input multiple first images of a first rendering object into a first implicit object model training network to generate a first implicit object model of the first rendering object, where the multiple first images are images of the first rendering object in multiple first poses, the first implicit object model is a neural network model, and the first rendering object is located in a target scene.

[0136] Step 802: Input multiple second images of the third rendering object into a second implicit object model training network to generate a second implicit object model of the third rendering object, where the multiple second images are images of the third rendering object in multiple second poses, the second implicit object model is a neural network model, and the third rendering object is located in a target scene.

[0137] The implementation of step 801 and step 802 may refer to step 401 in the embodiment of FIG4 , and will not be described in detail here.

[0138] Step 803: Receive rendering pose information, where the rendering pose information is used to indicate the pose of the rendering perspective of the target scene.

[0139] The implementation of step 803 may refer to step 402 in the embodiment of FIG. 4 , and will not be described in detail here.

[0140] Step 804: Input the rendering pose information into the first implicit object model to obtain first rendering data of the first rendering object, where the first rendering data includes one or more of normal information, depth information, color information, and material information of the first rendering object at a rendering perspective.

[0141] Step 805: Input the rendering pose information into the second implicit object model to obtain third rendering data of the third rendering object, where the third rendering data includes one or more of normal information, depth information, color information, and material information of the third rendering object under the rendering pose information.

[0142] The implementation of step 804 and step 805 may refer to step 403 in the embodiment of FIG. 4 , and will not be described in detail here.

[0143] Step 806: Determine light source configuration information, where the light source configuration information is used to indicate the lighting conditions of the target scene.

[0144] The implementation of step 806 may refer to step 404 in the embodiment of FIG. 4 , and will not be described in detail here.

[0145] In some scenarios, when both the first rendering data and the third rendering data include initial light source information, the user can integrate this initial light source information on a light source configuration interface to obtain light source configuration information. Based on this, in some embodiments, a light source configuration interface is provided; the light source configuration interface displays first initial light source information and second initial light source information, where the first initial light source information is the initial light source information in the first rendering data, and the second initial light source information is the initial light source information in the second rendering data; and in response to a user editing operation on the light source configuration interface, the first initial light source information and the second initial light source information are integrated into the light source configuration information.

[0146] In other scenarios, when the first rendering data does not include initial light source information and the third rendering data includes initial light source information, the light source configuration information can be determined based on the third rendering data including initial light source information, with reference to the aforementioned step 403, which will not be described in detail here.

[0147] In other scenarios, when the first rendering data does not include initial light source information and the third rendering data does not include initial light source information, the user can directly input the light source configuration information through the light source configuration interface, which will not be described in detail here.

[0148] Step 807: performing lighting processing on the first rendering data and the third rendering data based on the light source configuration information to obtain a target image, where the target image includes images of the first rendering object and the third rendering object under the rendering perspective and lighting conditions.

[0149] Through step 807 , a first image including both the first object and the third object can be rendered.

[0150] The implementation of step 807 may refer to step 405 in the embodiment of FIG4 , and will not be described in detail here.

[0151] In addition, when performing lighting processing operations of the rendering pipeline on the first rendering data and the third rendering data based on the light source configuration information, the depth information of the first rendering object and the depth information of the third rendering object can be combined to analyze the occlusion relationship between the first rendering object and the third rendering object in the target image to render a target image with higher realism.

[0152] In summary, in an embodiment of the present application, when multiple implicit object models need to be rendered simultaneously, rendering data that matches the geometry buffer can be obtained through each implicit object model separately, and then the rendering data in the geometry buffer can be uniformly processed through the lighting processing operation in the rendering pipeline, thereby further improving the flexibility of image rendering.

[0153] It should be noted that the serial numbers of the steps in the embodiment of Figure 8 are only used to mark the steps and do not represent the execution order of the steps. When applying the embodiment of this application, the execution order of certain steps can be adjusted according to actual conditions, or certain steps can be executed in parallel.

[0154] The following is an explanation of the relevant devices and equipment provided in the embodiments of the present application.

[0155] The present application also provides an image rendering device, as shown in FIG9 , wherein the device 900 includes:

[0156] A first input module 901 is configured to input multiple first images of a first rendering object into a first implicit object model training network to generate a first implicit object model of the first rendering object, wherein the multiple first images are images of the first rendering object in multiple first poses, the first implicit object model is a neural network model, and the first rendering object is located in a target scene. For specific implementation, reference may be made to step 401 in the embodiment of FIG. 4 .

[0157] The receiving module 902 is used to receive rendering pose information, where the rendering pose information is used to indicate the pose of the rendering perspective of the target scene. For specific implementation, please refer to step 402 in the embodiment of FIG. 4 .

[0158] The second input module 903 inputs the rendering pose information into the first implicit object model to obtain first rendering data of the first rendering object, where the first rendering data includes one or more of normal information, depth information, color information, and material information of the first rendering object under the rendering perspective. For specific implementation methods, please refer to step 403 in the embodiment of Figure 4.

[0159] The first determining module 904 is used to determine light source configuration information, where the light source configuration information is used to indicate the lighting conditions of the target scene. For specific implementation, please refer to step 404 in the embodiment of FIG. 4 .

[0160] The lighting processing module 905 is configured to perform lighting processing on the first rendering data based on the light source configuration information to obtain a target image, which includes an image of the first rendering object under the rendering perspective and lighting conditions. For specific implementation, please refer to step 405 in the embodiment of FIG. 4 .

[0161] Optionally, the apparatus 900 further includes:

[0162] A first acquisition module is used to acquire a first neural radiation field (NeRF) model, where the first NeRF model is used to predict an arbitrary perspective image of a first rendered object;

[0163] The second acquisition module is used to acquire multiple first images in multiple first poses based on the first NeRF model.

[0164] Optionally, the first rendering data further includes initial light source information, and the apparatus 900 further includes:

[0165] Provide a module for providing a light source configuration interface;

[0166] Display module, used to display initial light source information on the light source configuration interface;

[0167] The third acquisition module is configured to acquire light source information after adjusting the initial light source information in response to an editing operation of the user on the light source configuration interface, and use the adjusted light source information as the light source configuration information.

[0168] Optionally, the first rendering data further includes initial light source information, and the apparatus 900 further includes:

[0169] Provide a module for providing a light source configuration interface;

[0170] Display module, used to display initial light source information on the light source configuration interface;

[0171] The second determining module is configured to determine the initial light source information as the light source configuration information in response to a confirmation operation of the user on the light source configuration interface.

[0172] Optionally, the apparatus 900 further includes:

[0173] a third acquisition module, configured to acquire an explicit object model of a second rendering object, the second rendering object being located in the target scene;

[0174] a geometry processing module, configured to perform geometry processing on the explicit object model based on the rendering pose information to obtain second rendering data of a second rendering object, the second rendering data comprising one or more of normal information, depth information, color information, and material information of the second rendering object;

[0175] The lighting processing module is used to:

[0176] Lighting processing is performed on the first rendering data and the second rendering data based on the light source configuration information to obtain a target image, where the target image includes images of the first rendering object and the second rendering object under a rendering perspective and lighting conditions.

[0177] Optionally, the apparatus 900 further includes:

[0178] a third input module, configured to input a plurality of second images of a third rendered object into a second implicit object model training network to generate a second implicit object model of the third rendered object, wherein the plurality of second images are images of the third rendered object in a plurality of second poses, the second implicit object model is a neural network model, and the third rendered object is located in a target scene;

[0179] a fourth input module, configured to input the rendering pose information into the second implicit object model to obtain third rendering data of a third rendering object, the third rendering data comprising one or more of normal information, depth information, color information, and material information of the third rendering object in the rendering pose information;

[0180] The lighting processing module is used to:

[0181] Lighting processing is performed on the first rendering data and the third rendering data based on the light source configuration information to obtain a target image, where the target image includes images of the first rendering object and the third rendering object under a rendering perspective and lighting conditions.

[0182] Optionally, the first input module is used to:

[0183] determining one or more of normal information, depth information, color information, and material information for each of the plurality of first images;

[0184] The first implicit object model training network is trained based on one or more of normal information, depth information, color information, and material information of each of the plurality of first images to generate a first implicit object model.

[0185] In summary, in the image rendering method provided in the embodiments of the present application, a new implicit object model is generated from multiple images. Compared to related implicit object models, the implicit object model provided in the embodiments of the present application can output rendering data that matches the geometry buffer in the rendering pipeline, thereby combining the output of the implicit object model with the lighting processing operations of the rendering pipeline for the explicit object model. Because the light source configuration information in the lighting processing operations of the rendering pipeline can be flexibly set, the rendering effect in scenes with dynamically changing light source environments can be improved while ensuring rendering realism.

[0186] The first input module 901, receiving module 902, second input module 903, first determination module 904, and illumination processing module 905 can all be implemented via software or hardware. For example, the implementation of the first determination module 904 will be described below using the first determination module 904 as an example. Similarly, the implementation of the first input module 901, receiving module 902, second input module 903, and illumination processing module 905 can refer to the implementation of the first determination module 904.

[0187] As an example of a software functional unit, the first determination module 904 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the first determination module 904 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0188] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0189] As an example of a hardware functional unit, the first determination module 904 may include at least one computing device, such as a server. Alternatively, the first determination module 904 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0190] The multiple computing devices included in the first determination module 904 can be distributed in the same region or in different regions. The multiple computing devices included in the first determination module 904 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first determination module 904 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0191] It should be noted that, in other embodiments, the first determination module 904 can be used to execute any step in the image rendering method, the first input module 901 can be used to execute any step in the image rendering method, the receiving module 902 can be used to execute any step in the image rendering method, the second input module 903 can be used to execute any step in the image rendering method, and the illumination processing module 905 can be used to execute any step in the image rendering method. The steps that the first input module 901, the receiving module 902, the second input module 903, the first determination module 904, and the illumination processing module 905 are responsible for implementing can be specified as needed. The full functionality of the image rendering device is achieved by having the first input module 901, the receiving module 902, the second input module 903, the first determination module 904, and the illumination processing module 905 respectively implement different steps in the image rendering method.

[0192] The present application also provides a computing device 1000. As shown in FIG10 , computing device 1000 includes a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. Processor 1004, memory 1006, and communication interface 1008 communicate with each other via bus 1002. Computing device 1000 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 1000.

[0193] Bus 1002 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG10 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 1004 may include a path for transmitting information between various components of computing device 1000 (e.g., memory 1006, processor 1004, and communication interface 1008).

[0194] The processor 1004 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0195] The memory 1006 may include volatile memory, such as random access memory (RAM). The processor 1004 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0196] Memory 1006 stores executable program code. Processor 1004 executes the executable program code to implement the functions of the aforementioned first input module 901, receiving module 902, second input module 903, first determination module 904, and illumination processing module 905, thereby implementing the image rendering method. In other words, memory 1006 stores instructions for executing the image rendering method.

[0197] Alternatively, the memory 1006 stores executable codes, and the processor 1004 executes the executable codes to respectively implement the functions of the aforementioned image rendering apparatus, thereby implementing the image rendering method. That is, the memory 1006 stores instructions for executing the image rendering method.

[0198] The communication interface 1003 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1000 and other devices or a communication network.

[0199] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0200] As shown in Figure 11, the computing device cluster includes at least one computing device 1000. The memory 1006 in one or more computing devices 1000 in the computing device cluster may store the same instructions for executing the image processing method.

[0201] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store some instructions for executing the image rendering method. In other words, the combination of one or more computing devices 1000 can jointly execute the instructions for executing the image rendering method.

[0202] It should be noted that the memory 1006 in different computing devices 1000 in the computing device cluster can store different instructions, each for executing a portion of the functions of the image rendering apparatus. In other words, the instructions stored in the memory 1006 in different computing devices 1000 can implement the functions of one or more of the first input module 901, the receiving module 902, the second input module 903, the first determination module 904, and the illumination processing module 905.

[0203] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network (WAN) or a local area network (LAN), among others. FIG. 12 illustrates a possible implementation. As shown in FIG. 12 , two computing devices 1000A and 1000B are connected via a network. Specifically, the connection to the network is achieved via a communication interface in each computing device. In this type of possible implementation, the memory 1006 in the computing device 1000A stores instructions for executing the functions of the first input module 901, the receiving module 902, and the second input module 903. Simultaneously, the memory 1006 in the computing device 1000B stores instructions for executing the functions of the first determination module 904 and the illumination processing module 905.

[0204] The connection method between the computing device clusters shown in Figure 12 can be that considering that the image rendering method provided in this application needs to interact with the user and a large amount of calculation is required in the background, it is considered to hand over the functions implemented by the first input module 901, the receiving module 902, and the second input module 903 to the computing device 1000A for execution, and the functions implemented by the first determination module 904 and the lighting processing module 905 to the computing device 1000B for execution.

[0205] It should be understood that the functionality of the computing device 1000A shown in FIG12 may also be accomplished by multiple computing devices 1000. Similarly, the functionality of the computing device 1000B may also be accomplished by multiple computing devices 1000.

[0206] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to perform an image processing method.

[0207] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the image processing method.

[0208] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. An image rendering method, characterized in that: The method comprises: Inputting a plurality of first images of a first rendering object into a first implicit object model training network to generate a first implicit object model of the first rendering object, wherein the plurality of first images are images of the first rendering object in a plurality of first poses, the first implicit object model is a neural network model, and the first rendering object is located in a target scene; Receiving rendering pose information, where the rendering pose information is used to indicate a pose of a rendering perspective of the target scene; Inputting the rendering pose information into the first implicit object model to obtain first rendering data of the first rendering object, wherein the first rendering data includes one or more of normal information, depth information, color information, and material information of the first rendering object at the rendering perspective; Determining light source configuration information, where the light source configuration information is used to indicate the lighting condition of the target scene; The first rendering data is subjected to illumination processing based on the light source configuration information to obtain the target image, wherein the target image includes an image of the first rendering object under the rendering viewing angle and the illumination condition.

2. The method according to claim 1, characterized in that Before inputting the plurality of first images into the first implicit object model training network, the method further includes: Acquire a first neural radiation field NeRF model, where the first NeRF model is used to predict an arbitrary viewing angle image of the first rendered object; Based on the first NeRF model, the multiple first images in the multiple first poses are acquired.

3. The method according to claim 1 or 2, characterized in that The first rendering data further includes initial light source information, and the method further includes: Provide light source configuration interface; Displaying the initial light source information on the light source configuration interface; In response to the user's editing operation on the light source configuration interface, light source information after adjusting the initial light source information is obtained, and the adjusted light source information is used as the light source configuration information.

4. The method according to claim 1 or 2, characterized in that: The first rendering data further includes initial light source information, and the method further includes: Provide light source configuration interface; Displaying the initial light source information on the light source configuration interface; In response to a confirmation operation of the user on the light source configuration interface, the initial light source information is determined as the light source configuration information.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Obtaining an explicit object model of a second rendering object, the second rendering object being located in the target scene; Based on the rendering pose information, geometrically process the explicit object model to obtain second rendering data of the second rendering object, where the second rendering data includes one or more of normal information, depth information, color information, and material information of the second rendering object; The performing illumination processing on the first rendering data based on the light source configuration information to obtain the target image includes: The first rendering data and the second rendering data are subjected to illumination processing based on the light source configuration information to obtain the target image, wherein the target image includes images of the first rendering object and the second rendering object under the rendering viewing angle and the illumination condition.

6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Inputting a plurality of second images of a third rendered object into a second implicit object model training network to generate a second implicit object model of the third rendered object, wherein the plurality of second images are images of the third rendered object in a plurality of second postures, the second implicit object model is a neural network model, and the third rendered object is located in the target scene; Inputting the rendering pose information into the second implicit object model to obtain third rendering data of the third rendering object, wherein the third rendering data includes one or more of normal information, depth information, color information, and material information of the third rendering object under the rendering pose information; The performing illumination processing on the first rendering data based on the light source configuration information to obtain the target image includes: performing illumination processing on the first rendering data and the third rendering data based on the light source configuration information to obtain the target image, The target image includes images of the first rendering object and the third rendering object under the rendering viewing angle and the lighting condition.

7. The method according to any one of claims 1 to 6, characterized in that: The step of inputting the plurality of first images into a first implicit object model training network to generate a first implicit object model of the first rendered object comprises: Determine one or more of normal information, depth information, color information, and material information of each of the plurality of first images; The first implicit object model training network is trained based on one or more of the normal information, depth information, color information, and material information of each of the plurality of first images to generate the first implicit object model.

8. An image rendering device, characterized in that: The device comprises: a first input module, configured to input a plurality of first images of a first rendering object into a first implicit object model training network to generate a first implicit object model of the first rendering object, wherein the plurality of first images are images of the first rendering object in a plurality of first poses, the first implicit object model is a neural network model, and the first rendering object is located in a target scene; A receiving module, used for receiving rendering posture information, wherein the rendering posture information is used for indicating the posture of the rendering perspective of the target scene; A second input module, inputting the rendering pose information into the first implicit object model to obtain first rendering data of the first rendering object, wherein the first rendering data includes one or more of normal information, depth information, color information and material information of the first rendering object at the rendering perspective; A first determining module, used to determine light source configuration information, where the light source configuration information is used to indicate the lighting condition of the target scene; A lighting processing module is used to perform lighting processing on the first rendering data based on the light source configuration information to obtain the target image, wherein the target image includes an image of the first rendering object under the rendering viewing angle and the lighting condition.

9. The device according to claim 8, characterized in that The device also includes: A first acquisition module, used for acquiring a first neural radiation field NeRF model, where the first NeRF model is used for predicting an arbitrary viewing angle image of the first rendered object; The second acquisition module is used to acquire the multiple first images in the multiple first poses based on the first NeRF model.

10. The device according to claim 8 or 9, characterized in that The first rendering data further includes initial light source information, and the device further includes: Providing a module for providing a light source configuration interface; A display module, used for displaying the initial light source information on the light source configuration interface; The third acquisition module is used to obtain the light source information after the initial light source information is adjusted in response to the user's editing operation on the light source configuration interface, and use the adjusted light source information as the light source configuration information.

11. The device according to claim 8 or 9, characterized in that The first rendering data further includes initial light source information, and the device further includes: Providing a module for providing a light source configuration interface; A display module, used for displaying the initial light source information on the light source configuration interface; The second determining module is configured to determine the initial light source information as the light source configuration information in response to a confirmation operation of the user on the light source configuration interface.

12. The device according to any one of claims 8 to 11, characterized in that: The device also includes: A third acquisition module, configured to acquire an explicit object model of a second rendering object, where the second rendering object is located in the target scene; a geometry processing module, configured to perform geometry processing on the explicit object model based on the rendering pose information to obtain second rendering data of the second rendering object, wherein the second rendering data includes one or more of normal information, depth information, color information, and material information of the second rendering object; The illumination processing module is used for: The first rendering data and the second rendering data are subjected to illumination processing based on the light source configuration information to obtain the target image, wherein the target image includes images of the first rendering object and the second rendering object under the rendering viewing angle and the illumination condition.

13. The device according to any one of claims 8 to 11, characterized in that: The device also includes: a third input module, configured to input a plurality of second images of a third rendering object into a second implicit object model training network to generate a second implicit object model of the third rendering object, wherein the plurality of second images are images of the third rendering object in a plurality of second postures, the second implicit object model is a neural network model, and the third rendering object is located in the target scene; a fourth input module, configured to input the rendering pose information into the second implicit object model to obtain third rendering data of the third rendering object, wherein the third rendering data includes one or more of normal information, depth information, color information and material information of the third rendering object under the rendering pose information; The illumination processing module is used for: The first rendering data and the third rendering data are subjected to illumination processing based on the light source configuration information to obtain the target image, wherein the target image includes images of the first rendering object and the third rendering object under the rendering viewing angle and the illumination condition.

14. The device according to any one of claims 8 to 13, characterized in that: The first input module is used for: Determine one or more of normal information, depth information, color information, and material information of each of the plurality of first images; The first implicit object model training network is trained based on one or more of normal information, depth information, color information, and material information of each of the plurality of first images to generate the first implicit object model.

15. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1-7.

16. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 7.

Images