Image rendering method and device and related equipment

Abstract

The invention discloses an image rendering method and device and related equipment, and belongs to the technical field of computer vision. The method comprises the following steps: inputting a plurality of first images of a first rendering object into a first implicit object model training network to generate a first implicit object model of the first rendering object; receiving rendering pose information; inputting the rendering pose information into a first implicit object model to obtain first rendering data of the first rendering object; determining light source configuration information; and performing illumination processing on the first rendering data based on the light source configuration information to obtain a target image. According to the implicit object model provided by the embodiment of the invention, the rendering data matched with the geometric buffer area in the rendering pipeline can be output, so that the output of the implicit object model is combined with the rendering pipeline for the explicit object model. And under the condition of ensuring the rendering trueness, the rendering effect under the dynamic change scene of the light source environment is improved.

Description

[0001] This application claims priority to Chinese patent application No. 202310904896.5, filed on July 21, 2023, and entitled “An image rendering method, device and system”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of computer vision technology, and in particular to an image rendering method, apparatus, and related equipment. Background Art

[0003] In computer vision technology, a three-dimensional object model for a certain rendering object can be first constructed, and then a two-dimensional image of the rendering object at a certain rendering perspective position can be rendered using the three-dimensional object model.

[0004] In the related art, the pre-built three-dimensional object model can be an implicit object model, such as a neural radiance field (NeRF) model. When a certain rendering perspective pose is input into the implicit object model, the implicit object model can output a two-dimensional image corresponding to the rendering perspective pose. Among them, the relevant implicit object model renders the two-dimensional image according to the fixed light source information learned in advance when generating the two-dimensional image, resulting in the inability to modify the light source information during the image rendering process through the implicit object model, thereby making the rendering effect of this image rendering method not very ideal in the scene where the light source environment changes dynamically, and the way of rendering the implicit object model is quite different from the way of rendering the explicit object model. Summary of the invention

[0005] The embodiments of the present application provide an image rendering method, apparatus and related equipment, which can improve the rendering effect in a scene with a dynamically changing light source environment while ensuring the rendering realism. The technical solution is as follows:

[0006] In a first aspect, an image rendering method is provided, in which a plurality of first images of a first rendering object are input into a first implicit object model training network to generate a first implicit object model of the first rendering object, the plurality of first images are images of the first rendering object in a plurality of first poses, the first implicit object model is a neural network model, and the first rendering object is located in a target scene; rendering pose information is received, the rendering pose information is used to indicate the pose of a rendering perspective of the target scene; the rendering pose information is input into the first implicit object model to obtain first rendering data of the first rendering object, the first rendering data includes one or more of normal information, depth information, color information and material information of the first rendering object at the rendering perspective; light source configuration information is determined, the light source configuration information is used to indicate the lighting condition of the target scene; and lighting processing is performed on the first rendering data based on the light source configuration information to obtain a target image, the target image includes the image of the first rendering object at the rendering perspective and the lighting condition.

[0007] For the sake of convenience, the implicit object model that directly outputs a two-dimensional image without lighting processing after inputting the rendering perspective pose is called the relevant implicit object model.

[0008] In the image rendering method provided in the embodiment of the present application, an implicit object model is generated by multiple images, and the implicit object model can output rendering data that matches the geometric buffer in the rendering pipeline, such as normal information, depth information, color information or material information, so that the rendering pipeline can perform lighting processing on the model based on the output. Since the light source configuration information in the lighting processing operation of the rendering pipeline can be flexibly set, the rendering effect in the scene with dynamically changing light source environment can be improved while ensuring the rendering realism. At the same time, the rendering process can be combined with the rendering pipeline for the explicit object model, thereby improving the flexibility of rendering.

[0009] Based on the method provided in the first aspect, in a possible implementation, before inputting multiple first images into the first implicit object model training network, a first neural radiation field NeRF model is also obtained, and the first NeRF model is used to predict arbitrary perspective images of the first rendered object; based on the first NeRF model, multiple first images in multiple first poses are obtained.

[0010] In the embodiment of the present application, the first implicit object model can be trained by the output of the first NeRF model, and this process can be called an implicit model migration process, that is, the relevant implicit object model is migrated to an implicit object model that can output rendering data matching the geometry buffer.

[0011] Based on the method provided in the first aspect, in a possible implementation, the first rendering data also includes initial light source information. In the method, a light source configuration interface is also provided; the initial light source information is displayed on the light source configuration interface; in response to the user's editing operation on the light source configuration interface, the light source information after adjusting the initial light source information is obtained, and the adjusted light source information is used as the light source configuration information.

[0012] In an embodiment of the present application, a user can edit the initial light source information in the target scene where the first object is located obtained by inferring the first implicit object model to adjust the initial light source information, thereby performing lighting processing operations based on the adjusted light source information, and further improving the rendering effect in scenes where the light source environment changes dynamically.

[0013] Based on the method provided in the first aspect, in a possible implementation, the first rendering data also includes initial light source information. In the method, a light source configuration interface is also provided; the initial light source information is displayed on the light source configuration interface; and in response to a user's confirmation operation on the light source configuration interface, the initial light source information is determined as the light source configuration information.

[0014] In the embodiment of the present application, the initial light source information in the target scene where the first object is located can be directly inferred based on the first implicit object model to perform lighting processing operations, thereby improving the efficiency of image rendering.

[0015] Based on the method provided in the first aspect, in a possible implementation manner, in the method, an explicit object model of a second rendering object is also obtained, and the second rendering object is located in the target scene; based on the rendering pose information, the explicit object model is geometrically processed to obtain second rendering data of the second rendering object, and the second rendering data includes one or more of normal information, depth information, color information, and material information of the second rendering object.

[0016] Correspondingly, the implementation method of performing lighting processing on the first rendering data based on the light source configuration information to obtain the target image can be: performing lighting processing on the first rendering data and the second rendering data based on the light source configuration information to obtain the target image, and the target image includes images of the first rendering object and the second rendering object under the rendering perspective and lighting conditions.

[0017] In an embodiment of the present application, when it is necessary to render the implicit object model and the explicit object model at the same time, the rendering data matching the geometry buffer can be obtained through the two object models respectively, and then the rendering data in the geometry buffer can be uniformly processed through the lighting processing operation in the rendering pipeline. Therefore, the embodiment of the present application also provides an image rendering method that highly integrates the rendering process of the implicit object model and the rendering process of the explicit object model.

[0018] Based on the method provided in the first aspect, in a possible implementation manner, in the method, multiple second images of the third rendering object are also input into a second implicit object model training network to generate a second implicit object model of the third rendering object, the multiple second images are images of the third rendering object in multiple second poses, the second implicit object model is a neural network model, and the third rendering object is located in the target scene; the rendering pose information is input into the second implicit object model to obtain third rendering data of the third rendering object, and the third rendering data includes one or more of normal information, depth information, color information and material information of the third rendering object in the rendering pose information.

[0019] Correspondingly, the implementation method of performing lighting processing on the first rendering data based on the light source configuration information to obtain the target image can be: performing lighting processing on the first rendering data and the third rendering data based on the light source configuration information to obtain the target image, and the target image includes images of the first rendering object and the third rendering object under the rendering perspective and lighting conditions.

[0020] In an embodiment of the present application, when it is necessary to render multiple implicit object models at the same time, rendering data matching the geometry buffer can be obtained through each implicit object model respectively, and then the rendering data in the geometry buffer can be uniformly processed through the lighting processing operation in the rendering pipeline, thereby further improving the flexibility of image rendering.

[0021] Based on the method provided in the first aspect, in a possible implementation manner, in the method, inputting multiple first images into a first implicit object model training network to generate a first implicit object model of a first rendering object can be implemented by: determining one or more of the normal information, depth information, color information, and material information of each of the multiple first images; training the first implicit object model training network based on one or more of the normal information, depth information, color information, and material information of each of the multiple first images to generate a first implicit object model.

[0022] After obtaining one or more of the normal information, depth information, color information and material information of each of the multiple first images, one or more of the normal information, depth information, color information and material information of each of the multiple first images can be used as supervision information to perform supervised training on the first implicit object model training network, so that the first implicit object model training network can learn one of the normal information, depth information, color information and material information of the first rendered object under different viewing angles and poses during the training process, that is, the first implicit object model is trained.

[0023] In a second aspect, an image rendering device is provided, wherein the image rendering device has the function of implementing the image rendering method in the first aspect. The image rendering device comprises at least one module, wherein the at least one module is used to implement the image rendering method provided in the first aspect.

[0024] In a third aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the image rendering method provided in the first aspect above.

[0025] In a fourth aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the image rendering method provided in the first aspect.

[0026] In a fifth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the image rendering method provided in the first aspect.

[0027] The technical effects obtained by the above-mentioned second, third, fourth and fifth aspects are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a schematic diagram of the architecture of an image rendering system provided by an embodiment of the present application;

[0029] Figure 2 is a schematic diagram of the architecture of another image rendering system provided in an embodiment of the present application;

[0030] Figure 3 is a schematic diagram of the architecture of another image rendering system provided in an embodiment of the present application;

[0031] Figure 4 is a flow chart of an image rendering method provided by an embodiment of the present application;

[0032] Figure 5 is a structural diagram of a first implicit object model provided in an embodiment of the present application;

[0033] Figure 6 is a schematic diagram of a light source configuration interface provided in an embodiment of the present application;

[0034] Figure 7 is a flow chart of another image rendering method provided by an embodiment of the present application;

[0035] Figure 8is a flow chart of another image rendering method provided by an embodiment of the present application;

[0036] Fig. 9 is a structural schematic diagram of an image rendering device provided in an embodiment of the present application;

[0037] Fig.10 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0038] Fig.11 It is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application;

[0039] Fig.12 It is a schematic diagram of the architecture of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the present application more clear, the implementation mode of the present application will be further described in detail below with reference to the accompanying drawings.

[0041] In computer graphics (CG) technology, three-dimensional (3D) object models are usually expressed as explicit mesh models. At present, a mature real-time rendering pipeline has gradually been formed to render mesh models to obtain a two-dimensional image under a certain rendering perspective. The rendering pipeline can usually be divided into a geometry processing stage and a lighting processing stage. In the geometry processing stage, the color information, geometry information, material information and other data of the mesh model under a certain rendering perspective are calculated, and the data obtained in the geometry processing stage is stored in a set of buffers, which are also called geometry buffers (G-Buffer). In the lighting processing stage, the data cached in the geometry buffer is combined with the light source information to calculate and generate the final rendering result, which is a two-dimensional image corresponding to the rendering perspective.

[0042] In recent years, NeRF technology has been widely studied and applied, and it can produce more realistic rendering effects. Unlike the original CG technology, the object model represented by NeRF technology is expressed by a neural network, and its object model structure is stored in the neural network parameters. Therefore, this type of model is usually called an implicit object model, while the WYSIWYG object model represented by the mesh model is usually called an explicit object model.

[0043] The rendering method of the implicit object model is inconsistent with the rendering pipeline for the explicit object model. Usually, an implicit object model is trained for a certain rendering object first, and then when a certain rendering view pose is directly input to the implicit object model, the implicit object model can output a 2D image corresponding to the rendering view pose.

[0044] Among them, the method of training an implicit object model for a certain rendering object can be: first collect sample images of different perspectives for the rendering object, and determine the camera pose of each sample image. The camera pose can be understood as the pose of the camera that collects the sample image. The sample image and the corresponding camera pose are sent to the NeRF model for training, so that the NeRF model can learn the mapping relationship between the camera pose and the sample image. The trained NeRF model can input the two-dimensional image of the rendering object under any rendering perspective pose.

[0045] For the convenience of subsequent explanation, the implicit object model that directly outputs a two-dimensional image after inputting the rendering perspective pose is called the relevant implicit object model.

[0046] Due to the characteristics of the neural network model itself, the rendering realism of the relevant implicit object model is higher than that of the explicit object model. However, after the relevant implicit object model learns the mapping relationship between the camera pose and the sample image during the training process, for example, after learning the mapping relationship between the camera pose and the lighting conditions in the sample image, the subsequent rendering of the two-dimensional image is directly rendered and output based on the mapping relationship. This will result in the inability to modify the light source information during the image rendering process through the relevant implicit object model, and thus the rendering effect of the image rendering method in the scene where the light source environment changes dynamically is not very ideal.

[0047] Based on this, an image rendering method provided by an embodiment of the present application, in which a new type of implicit object model is generated through multiple images, relative to the relevant implicit object model, the implicit object model provided by the embodiment of the present application can output rendering data that matches the geometric buffer in the rendering pipeline, so as to combine the output of the implicit object model with the lighting processing operation of the rendering pipeline for the explicit object model. Since the light source configuration information in the lighting processing operation of the rendering pipeline can be flexibly set, the rendering effect in the scene where the light source environment changes dynamically can be improved while ensuring the rendering realism.

[0048] The implicit object models in the subsequent embodiments all refer to the new implicit object models provided in the embodiments of the present application, and will not be explained one by one in the following.

[0049] In addition, the image rendering method provided in the embodiment of the present application can be applied to a cloud scene, which can be an autonomous driving scene, etc. In this scenario, the image rendering method is executed by a cloud platform, that is, the cloud platform provides cloud services to users, and the cloud service is used to implement the image rendering method provided in the embodiment of the present application. Optionally, the image rendering method provided in the embodiment of the present application can also be executed by a terminal device. The embodiment of the present application does not limit the application scenario of the image rendering method.

[0050] The image rendering system provided in the embodiment of the present application is explained below.

[0051] Figure 1 Schematic diagram of the architecture of an image rendering system provided by an embodiment of the present application. Figure 1 As shown, the image rendering system 00 includes a preprocessing unit 01 , a rendering calculation unit 02 and an editing interaction unit 03 .

[0052] Among them, the preprocessing unit 01 is used to process data to obtain the implicit object model required by the embodiment of the present application.

[0053] In an embodiment of the present application, the implicit object model can output data that matches the geometry buffer in the rendering pipeline. In other words, the implicit object model can output rendering data that can be cached in the geometry buffer, and the rendering data includes one or more of the normal information, depth information, color information, and material information of the rendering object. This allows the output of the implicit object model to be flexibly processed through the lighting processing operation in the rendering pipeline to obtain a rendering result.

[0054] Among them, the implicit object model can be obtained by training with sample images. In some embodiments, the sample image can be directly obtained by a related implicit object model such as a NeRF model. In this scenario, the process of obtaining the implicit object model provided by the embodiment of the present application can be called a model migration process. In other words, the implicit object model required by the embodiment of the present application can be obtained by migrating a related implicit object model such as a NeRF model.

[0055] Optionally, the sample image may also be acquired through an image acquisition device, which will not be described in detail here.

[0056] The rendering calculation unit 02 is used to output the rendering data of a certain rendering object through an implicit object model, and cache the rendering data of the rendering object in a geometry buffer. Optionally, the rendering calculation unit 02 can output the rendering data of each rendering object through multiple implicit object models, and cache the rendering data of each rendering object in a geometry buffer. Optionally, the rendering calculation unit 02 can also perform a rendering pipeline geometry processing operation on an explicit object model of a certain rendering object to obtain the rendering data of the rendering object, and cache the rendering data of the rendering object in a geometry buffer.

[0057] In other words, in an embodiment of the present application, when rendering a scene, regardless of whether the object model included in the scene is an implicit object model or an explicit object model, the rendering calculation unit 02 can obtain the rendering data corresponding to the rendered object, and cache the rendering data in the geometry buffer, and then render the rendering data of each rendering object in combination with the light source information to obtain the final rendering result.

[0058] The editing interaction unit 03 is used for human-computer interaction to provide the light source information edited by the user, thereby realizing the adjustment of the light source information.

[0059] based on Figure 1 The image rendering system shown can combine the output of the implicit object model with the lighting processing operation in the rendering pipeline. On the one hand, it can improve the rendering effect in the scene where the light source environment changes dynamically while ensuring the rendering realism. On the other hand, when it is necessary to render the implicit object model and the explicit object model at the same time, the rendering data matching the geometry buffer can be obtained through the two object models respectively, and then the rendering data in the geometry buffer can be uniformly processed through the lighting processing operation in the rendering pipeline. Therefore, the embodiment of the present application also provides an image rendering system that highly integrates the rendering process of the implicit object model and the rendering process of the explicit object model.

[0060] The following example takes rendering an implicit object model and an explicit object model as an example. Figure 1 The system shown is used for illustration purposes only.

[0061] Figure 2 It is a schematic diagram of the architecture of another image rendering system provided in an embodiment of the present application.

[0062] like Figure 2 As shown, the preprocessing unit 01 can be used to migrate relevant implicit object models such as the NeRF model to the first implicit object model required by the embodiment of the present application, or obtain the first implicit object model required by the embodiment of the present application through other methods.

[0063] The rendering calculation unit 02 is used, on the one hand, to infer (i.e., output) first rendering data corresponding to the rendering perspective pose through the first implicit object model, the first rendering data including one or more of the normal information, color information, depth information, and material information of the first rendering object, and cache the first rendering data in the geometry buffer. The rendering calculation unit 02 is used, on the other hand, to perform the geometry processing operation of the rendering pipeline on the explicit object model, to obtain second rendering data corresponding to the rendering perspective pose, the second rendering data including one or more of the normal information, depth information, color information, and material information of the second rendering object, and cache the second rendering data in the geometry buffer.

[0064] The editing interaction unit 03 is used to obtain the light source configuration information edited by the user and send the light source configuration information to the rendering calculation unit 02. The user can directly edit the light source configuration information or adjust the light source configuration information based on the initial light source information inferred by the first implicit object model.

[0065] The rendering calculation unit 02 is further used to perform a lighting processing operation on the rendering data of each rendering object in the geometry buffer based on the light source configuration information to obtain a final rendering result, which is a two-dimensional image including the first rendering object and the second rendering object.

[0066] The following example uses rendering of two implicit object models. Figure 1 The system shown is used for illustration purposes only.

[0067] Figure 3 It is a schematic diagram of the architecture of another image rendering system provided in an embodiment of the present application.

[0068] like Figure 3 As shown, the preprocessing unit 01 can be used to migrate relevant implicit object models such as the NerF model into the first implicit object model and the second implicit object model required by the embodiment of the present application, or obtain the first implicit object model and the second implicit object model required by the embodiment of the present application by other means.

[0069] The rendering calculation unit 02 is used to infer (i.e., output) first rendering data corresponding to the rendering perspective pose through the first implicit object model, the first rendering data including one or more of the normal information, depth information, color information, and material information of the first rendering object, and cache the first rendering data in the geometry buffer. The rendering calculation unit 02 is also used to infer (i.e., output) third rendering data corresponding to the rendering perspective pose through the second implicit object model, the third rendering data including one or more of the normal information, depth information, color information, and material information of the third rendering object, and cache the third rendering data in the geometry buffer.

[0070] The editing interaction unit 03 is used to obtain the light source configuration information edited by the user and send the light source configuration information to the rendering calculation unit 02. The user can directly edit the light source configuration information, or adjust the light source configuration information based on the initial light source information inferred from the first implicit object model and / or the second implicit object model.

[0071] The rendering calculation unit 02 is further used to perform a lighting processing operation on the rendering data of each rendering object in the geometry buffer based on the light source configuration information to obtain a final rendering result, which is a two-dimensional image including the first rendering object and the third rendering object.

[0072] It should be noted that Figure 2 and Figure 3 For illustration purposes, in the embodiment of the present application, the image rendering system 00 can render a greater number or a smaller number of object models, which will not be illustrated one by one here.

[0073] In addition, the image rendering system 00 can be exemplified as a cloud platform. In this scenario, the image rendering system 00 can be implemented by one or more computing devices. Among them, the preprocessing unit 01, the rendering computing unit 02, and the editing interaction unit 03 can be integrated on the same computing device, or deployed on different computing devices, which is not limited in the embodiments of the present application.

[0074] The image rendering method provided in the embodiment of the present application is explained below.

[0075] Figure 4 This is a flow chart of an image rendering method provided by an embodiment of the present application. Figure 4 As shown, the method includes the following steps.

[0076] Step 401: Input multiple first images of a first rendering object into a first implicit object model training network to generate a first implicit object model of the first rendering object, wherein the multiple first images are images of the first rendering object in multiple first poses, the first implicit object model is a neural network model, and the first rendering object is located in a target scene.

[0077] The target scene may be any scene to be rendered, for example, the target scene includes all objects on a road in an autonomous driving scene and surrounding buildings, etc. The first rendering object may be understood as one of the objects in the target scene, for example, the first rendering object is a vehicle.

[0078] In order to implement the image rendering method provided in the embodiment of the present application, it is necessary to first train the implicit object model required by the embodiment of the present application through step 401. In some embodiments, the training of the implicit object model can be implemented by: determining one or more of the normal information, depth information, color information, and material information of each of the multiple first images; training the first implicit object model training network based on one or more of the normal information, depth information, color information, and material information of each of the multiple first images to generate the first implicit object model.

[0079] Among them, after obtaining one or more of the normal information, depth information, color information and material information of each of the multiple first images, one or more of the normal information, depth information, color information and material information of each of the multiple first images can be used as supervision information to perform supervised training on the first implicit object model training network, so that the first implicit object model training network can learn one of the normal information, depth information, color information and material information of the first rendered object under different viewing angles and poses during the training process, that is, the first implicit object model is trained.

[0080] Each first image illustratively includes a red-green-blue (RGB) image and / or a depth image corresponding to the corresponding first pose.

[0081] Among them, multiple first images can be outputted by, for example, a related implicit object model for the first rendering object, that is, an image output implicit object model. In this scenario, the image output implicit object model also outputs a first position corresponding to the first image each time it outputs a first image. Based on this, an exemplary implementation method for obtaining multiple first images corresponding to multiple first positions can be: obtaining a first NeRF model, the first NeRF model is used to predict an arbitrary perspective image of the first rendering object; based on the first NeRF model, obtaining multiple first images corresponding to the multiple first positions.

[0082] In an embodiment of the present application, the first implicit object model can be trained by the output of the first NeRF model, and this process can be called an implicit model migration process. That is, the relevant implicit object migration model, i.e., the image output type implicit object model, is migrated to an implicit object model that can output rendering data that matches the geometric buffer. Optionally, in other embodiments provided in the present application, the same input can be used as the relevant implicit object migration model, i.e., multiple images of rendered objects under different perspectives can be input to generate the implicit object model provided in the embodiment of the present application.

[0083] Optionally, multiple first images can also be acquired through an image acquisition device, etc. In this scenario, the first position corresponding to each first image can be understood as the position of the image acquisition device that acquires the first image, that is, the camera position.

[0084] In addition, determining one or more of the normal information, depth information, color information, and material information of each of the multiple first images can be achieved, for example, by inverse rendering or the like. Inverse rendering is used to infer scene parameters from a two-dimensional image, and the scene parameters include, for example, the lighting conditions of the scene, the color information of the objects in the scene, the material information, and the geometric information. In the forward rendering process, the scene parameters are input to generate (ie, output) an image. However, in the inverse rendering process, these scene parameters need to be inferred from the image. Inverse rendering involves multiple operations, including photometric estimation, material recovery, and geometric reconstruction. The specific implementation method of inverse rendering is not limited in the embodiments of the present application.

[0085] Optionally, in the embodiment of the present application, after acquiring multiple first images, one or more of the normal information, depth information, color information, and material information of each first image may be acquired by other means. For example, a user may directly manually annotate one or more of the normal information, depth information, color information, and material information of each first image, but this implementation method requires more human resources.

[0086] It should be noted that the above-mentioned process of training the first implicit object model is used for illustrative purposes. The embodiment of the present application does not limit the method of how to train the first implicit object model. Any implementation method that can train the first implicit object model can be applied to the embodiment of the present application.

[0087] In addition, the training process of the first implicit object model can be performed offline on training resources such as a graphic processing unit (GPU). The embodiment of the present application does not limit the device for training the first implicit object model.

[0088] Figure 5 is a schematic diagram of the structure of a first implicit object model provided in an embodiment of the present application. Figure 5 As shown, the first implicit object model includes multiple network layers, Figure 5 Each rectangular block in represents a set of network layers. Figure 5 As shown, some network layers are used for geometric expression, such as normal expression and depth expression, to output geometric information including normal information and depth information. Some network layers are used for appearance expression, such as material expression and color expression, to output material information and color information of the first object. Some network layers are used for lighting expression to output initial light source information.

[0089] Each group of network layers may include a fully connected layer, etc. The embodiment of the present application does not limit the specific internal structure of the first implicit object model, and any neural network model that can output the color information, geometric information, and material information of the first object can be applied in the embodiment of the present application.

[0090] Step 402: receiving rendering pose information, where the rendering pose information is used to indicate the pose of the rendering perspective of the target scene.

[0091] The pose of the rendering perspective of the target scene can be understood as the pose of the virtual camera used to render the target scene. For the convenience of subsequent description, the pose of the rendering perspective can be referred to as the rendering perspective pose.

[0092] In some embodiments, the rendering pose information can be specified by the user. For example, the user can Figure 1 The rendering pose information is set by the editing interaction unit in the image rendering system shown in the figure. Optionally, the rendering pose information can also be obtained by other methods, and the embodiment of the present application does not limit the method for obtaining the rendering pose information.

[0093] In addition, the rendering perspective pose indicated by the rendering pose information may be one or more. When there are multiple rendering perspective poses, the present application embodiment may generate a rendering result corresponding to each rendering perspective pose. The subsequent embodiments are described by taking one rendering perspective pose as an example.

[0094] Step 403: Input the rendering pose information into the first implicit object model to obtain first rendering data of the first rendering object, where the first rendering data includes one or more of normal information, depth information, color information and material information of the first rendering object at a rendering perspective.

[0095] The implicit object model provided in the embodiment of the present application can output normal information, depth information, color information, material information and other information of the corresponding rendering object, so as to facilitate the subsequent combination of the output of the implicit object model with the lighting processing operation in the rendering pipeline.

[0096] The color information of the first rendering object in the first rendering data can be understood as: the color information of the first rendering object observed when the first rendering object is observed from the rendering perspective posture. The color information can include the color of each pixel point on the two-dimensional plane corresponding to the rendering perspective posture.

[0097] Normal information and / or depth information may be collectively referred to as geometric information.

[0098] The geometric information of the first rendering object in the first rendering data can be understood as: the geometric information of the first rendering object observed when the first rendering object is observed from the rendering perspective pose. The geometric information may include, by way of example, normal information of the first rendering object and / or depth information of the first rendering object. Among them, the normal information may include a normal vector of each pixel point on a two-dimensional plane corresponding to the rendering perspective pose, and the normal vector is a normal vector at a position point of the first rendering object corresponding to each pixel point. The depth information may include the depth of each pixel point on a two-dimensional plane corresponding to the rendering perspective pose, and the depth is the distance between the position point of the first rendering object corresponding to each pixel point and the rendering camera (i.e., the observer).

[0099] The material information of the first rendering object in the first rendering data can be understood as: the material information of the first rendering object observed when the first rendering object is observed from the rendering perspective. The material information can exemplarily include the material texture of each pixel point on the two-dimensional plane corresponding to the rendering perspective.

[0100] Optionally, the first implicit object model may also output initial light source information, which may indicate relevant information about the light source in the target scene where the first rendering object is located. The relevant information may include, for example, the position of the light source, the emission direction of the light source, the illumination intensity of the light source, and the color of the light source. In this way, in the subsequent illumination processing stage, illumination processing operations may be performed based on the initial light source information. In some embodiments, the initial light source information may be represented by spherical harmonics.

[0101] Since the first rendering data includes normal information, depth information, color information, material information, initial light source information, etc. of the first rendering object, after obtaining the first rendering perspective in step 403, the first rendering data can be cached in a geometry buffer configured for the rendering pipeline.

[0102] Step 404: Determine light source configuration information, where the light source configuration information is used to indicate the lighting condition of the target scene.

[0103] In some embodiments, the first rendering data also includes initial light source information. In other words, when the first implicit object model infers one or more of the normal information, depth information, color information, and material information of the first rendering object, it also infers the initial light source information in the target scene where the first rendering object is located. In this case, the light source configuration information can be determined based on the initial light source information, which improves the flexibility of determining the light source configuration information.

[0104] By way of example, an implementation method for determining light source configuration information based on initial light source information may be: providing a light source configuration interface; displaying the initial light source information on the light source configuration interface, and in response to a user's editing operation on the light source configuration interface, obtaining light source information after adjusting the initial light source information, and using the adjusted light source information as the light source configuration information.

[0105] In an embodiment of the present application, a user can edit the initial light source information in the target scene where the first object is located obtained by inferring the first implicit object model to adjust the initial light source information, thereby performing lighting processing operations based on the adjusted light source information, and further improving the rendering effect in scenes where the light source environment changes dynamically.

[0106] The light source configuration interface can be Figure 1 The interface provided by the editing interaction unit of the system shown. Figure 6 is a schematic diagram of a light source configuration interface provided by an embodiment of the present application. Figure 6 As shown, the light source configuration interface displays a virtual light source generated based on the initial light source information, which includes virtual light source 1 and virtual light source 2. The user can adjust the position of the virtual light source by dragging one of the virtual light sources. The lighting direction and lighting intensity of the virtual light source can also be adjusted through the input box on the right.

[0107] As another example, the implementation method of determining the light source configuration information based on the initial light source information may be: providing a light source configuration interface; displaying the initial light source information on the light source configuration interface; and determining the initial light source information as the light source configuration information in response to a user confirmation operation on the light source configuration interface. Alternatively, the initial light source information may be directly used as the light source configuration information without responding to the user's confirmation operation.

[0108] In this scenario, the initial light source information in the target scene where the first object is located can be directly inferred based on the first implicit object model to perform lighting processing operations, thereby improving the efficiency of image rendering.

[0109] Optionally, in other embodiments, before performing the lighting processing operation of the rendering pipeline on the first rendering data based on the light source configuration information, a light source configuration interface may be displayed; and the light source configuration information is obtained in response to the user's operation on the light source configuration interface.

[0110] That is, in the embodiment of the present application, the user can also directly set the light source configuration information without relying on the first implicit object model to determine the light source configuration information, which further improves the flexibility of determining the light source configuration information.

[0111] Step 405: Perform lighting processing on the first rendering data based on the light source configuration information to obtain a target image, where the target image includes an image of the first rendering object under a rendering viewing angle and lighting conditions.

[0112] When performing the lighting processing operation, the first rendering data in the geometry buffer is calculated based on the light source configuration information for the target scene to be rendered to obtain a first image. Since the light source configuration information can be flexibly configured, when the first image is rendered through the first implicit object model, not only the rendering realism can be guaranteed, but also the rendering effect in the scene where the light source environment changes dynamically can be improved.

[0113] Among them, the lighting processing operation can be implemented by a shader for example, and the embodiment of the present application does not limit the implementation method of the lighting processing operation.

[0114] In summary, in the image rendering method provided in the embodiment of the present application, a new type of implicit object model is generated through multiple images. Compared with the relevant implicit object model, the implicit object model provided in the embodiment of the present application can output rendering data that matches the geometric buffer in the rendering pipeline, so as to combine the output of the implicit object model with the lighting processing operation of the rendering pipeline for the explicit object model. Since the light source configuration information in the lighting processing operation of the rendering pipeline can be flexibly set, the rendering effect in the scene where the light source environment changes dynamically can be improved while ensuring the rendering realism.

[0115] In addition, when rendering an image of a scene, the scene may include multiple rendering objects. For example, in addition to the aforementioned first rendering object, other objects such as a second rendering object and / or a third rendering object may also be included. This is illustrated below by two embodiments.

[0116] Figure 7 FIG. 1 is another flow chart of an image rendering method provided by an embodiment of the present application. Figure 7 As shown, the method includes the following steps.

[0117] Step 701: Input multiple first images of a first rendering object into a first implicit object model training network to generate a first implicit object model of the first rendering object, wherein the multiple first images are images of the first rendering object in multiple first poses, the first implicit object model is a neural network model, and the first rendering object is located in a target scene.

[0118] Step 702: receiving rendering pose information, where the rendering pose information is used to indicate the pose of the rendering perspective of the target scene.

[0119] Step 703: Input the rendering pose information into the first implicit object model to obtain first rendering data of the first rendering object, where the first rendering data includes one or more of normal information, depth information, color information and material information of the first rendering object at a rendering perspective.

[0120] The implementation of steps 701 to 703 can refer to Figure 4 Steps 401 to 403 in the embodiment are not described in detail here.

[0121] Step 704: Obtain an explicit object model of a second rendering object, where the second rendering object is located in the target scene.

[0122] The explicit object model of the second rendering object can be directly generated by the user. Figure 1 Optionally, the editing interaction unit can also be input by Figure 1 The image rendering system shown models an explicit object model through CG technology.

[0123] Step 705: Based on the rendering pose information, geometrically process the explicit object model to obtain second rendering data of a second rendering object, where the second rendering data includes one or more of normal information, depth information, color information, and material information of the second rendering object.

[0124] In an embodiment of the present application, the object model for the second rendering object is an explicit object model. The explicit object model may be a mesh model, for example. Therefore, the explicit object model may be directly processed by a geometry processing operation in the rendering pipeline to directly obtain the second rendering data matching the geometry buffer.

[0125] The geometry processing operation may include, for example, vertex shading, projection transformation, clipping transformation, screen mapping, etc. The embodiment of the present application does not limit the implementation of the geometry processing operation.

[0126] Since the second rendering data includes one or more of the normal information, depth information, color information, and material information of the second rendering object, the second rendering data may be cached in a geometry buffer configured for the rendering pipeline.

[0127] Step 706: Determine light source configuration information, where the light source configuration information is used to indicate the lighting condition of the target scene.

[0128] The implementation of light source configuration information can refer to Figure 4 Step 404 in the embodiment will not be described in detail here.

[0129] Step 707: performing illumination processing on the first rendering data and the second rendering data based on the light source configuration information to obtain a target image, wherein the target image includes images of the first rendering object and the second rendering object under rendering viewing angle and illumination conditions.

[0130] Through step 707 , a target image including both the first rendering object and the second rendering object can be rendered.

[0131] In addition, when performing lighting processing operations of the rendering pipeline on the first rendering data and the second rendering data based on the light source configuration information, the occlusion relationship between the first rendering object and the second rendering object in the target image can be analyzed in combination with the depth information of the first rendering object and the depth information of the second rendering object to render a target image with higher realism.

[0132] In summary, in an embodiment of the present application, when it is necessary to render the implicit object model and the explicit object model at the same time, the rendering data matching the geometry buffer can be obtained through the two object models respectively, and then the rendering data in the geometry buffer can be uniformly processed through the lighting processing operation in the rendering pipeline. Therefore, the embodiment of the present application also provides an image rendering method that highly integrates the rendering process of the implicit object model and the rendering process of the explicit object model.

[0133] It should be noted that Figure 7 The serial numbers of the steps in the embodiments are only used to mark the steps and do not represent the execution order of the steps. In the application of the embodiments of the present application, the execution order of some steps can be adjusted according to the actual situation, or some steps can be executed in parallel.

[0134] Figure 8 FIG. 1 is another flow chart of an image rendering method provided by an embodiment of the present application. Figure 8 As shown, the method includes the following steps.

[0135] Step 801: Input multiple first images of a first rendering object into a first implicit object model training network to generate a first implicit object model of the first rendering object, wherein the multiple first images are images of the first rendering object in multiple first poses, the first implicit object model is a neural network model, and the first rendering object is located in a target scene.

[0136] Step 802: Input multiple second images of the third rendering object into the second implicit object model training network to generate a second implicit object model of the third rendering object, the multiple second images are images of the third rendering object in multiple second poses, the second implicit object model is a neural network model, and the third rendering object is located in the target scene.

[0137] The implementation of step 801 and step 802 can refer to Figure 4Step 401 in the embodiment will not be described in detail here.

[0138] Step 803: receiving rendering pose information, where the rendering pose information is used to indicate the pose of the rendering perspective of the target scene.

[0139] The implementation of step 803 can refer to Figure 4 Step 402 in the embodiment will not be described in detail here.

[0140] Step 804: Input the rendering pose information into the first implicit object model to obtain first rendering data of the first rendering object, where the first rendering data includes one or more of normal information, depth information, color information, and material information of the first rendering object at a rendering perspective.

[0141] Step 805: Input the rendering pose information into the second implicit object model to obtain third rendering data of the third rendering object, where the third rendering data includes one or more of normal information, depth information, color information and material information of the third rendering object under the rendering pose information.

[0142] The implementation of step 804 and step 805 can refer to Figure 4 Step 403 in the embodiment will not be described in detail here.

[0143] Step 806: Determine light source configuration information, where the light source configuration information is used to indicate the lighting condition of the target scene.

[0144] The implementation of step 806 can refer to Figure 4 Step 404 in the embodiment will not be described in detail here.

[0145] In some scenarios, when both the first rendering data and the third rendering data include initial light source information, the user can integrate the initial light source information on the light source configuration interface to obtain the light source configuration information. Based on this, in some embodiments, a light source configuration interface is provided; the first initial light source information and the second initial light source information are displayed on the light source configuration interface, the first initial light source information is the initial light source information in the first rendering data, and the second initial light source information is the initial light source information in the second rendering data; in response to the user's editing operation on the light source configuration interface, the first initial light source information and the second initial light source information are integrated into the light source configuration information.

[0146] In other scenarios, when the first rendering data does not include initial light source information and the third rendering data includes initial light source information, the light source configuration information can be determined based on the third rendering data including initial light source information with reference to the aforementioned step 403, which will not be described in detail here.

[0147] In other scenarios, when the first rendering data does not include the initial light source information and the third rendering data does not include the initial light source information, the user can directly input the light source configuration information through the light source configuration interface, which will not be described in detail here.

[0148] Step 807: performing lighting processing on the first rendering data and the third rendering data based on the light source configuration information to obtain a target image, wherein the target image includes images of the first rendering object and the third rendering object under the rendering viewing angle and lighting conditions.

[0149] Through step 807 , a first image including both the first object and the third object can be rendered.

[0150] The implementation of step 807 can refer to Figure 4 Step 405 in the embodiment will not be described in detail here.

[0151] In addition, when the lighting processing operation of the rendering pipeline is performed on the first rendering data and the third rendering data based on the light source configuration information, the occlusion relationship between the first rendering object and the third rendering object in the target image can be analyzed in combination with the depth information of the first rendering object and the depth information of the third rendering object to render a target image with higher realism.

[0152] In summary, in an embodiment of the present application, when multiple implicit object models need to be rendered simultaneously, rendering data matching the geometry buffer can be obtained through each implicit object model respectively, and then the rendering data in the geometry buffer can be uniformly processed through the lighting processing operation in the rendering pipeline, thereby further improving the flexibility of image rendering.

[0153] It should be noted that Figure 8 The serial numbers of the steps in the embodiments are only used to mark the steps and do not represent the execution order of the steps. In the application of the embodiments of the present application, the execution order of some steps can be adjusted according to the actual situation, or some steps can be executed in parallel.

[0154] The following is an explanation of the relevant devices and equipment provided in the embodiments of the present application.

[0155] The present application also provides an image rendering device, such as Fig. 9 As shown, the device 900 includes:

[0156] The first input module 901 is used to input multiple first images of the first rendering object into the first implicit object model training network to generate a first implicit object model of the first rendering object, the multiple first images are images of the first rendering object in multiple first poses, the first implicit object model is a neural network model, and the first rendering object is located in the target scene; the specific implementation method can refer to Figure 4Step 401 in the embodiment.

[0157] The receiving module 902 is used to receive rendering posture information, where the rendering posture information is used to indicate the posture of the rendering perspective of the target scene. Figure 4 Step 402 in an embodiment.

[0158] The second input module 903 inputs the rendering pose information into the first implicit object model to obtain first rendering data of the first rendering object, where the first rendering data includes one or more of normal information, depth information, color information, and material information of the first rendering object under the rendering perspective. For specific implementation methods, please refer to Figure 4 Step 403 in the embodiment.

[0159] The first determination module 904 is used to determine the light source configuration information, which is used to indicate the lighting conditions of the target scene; the specific implementation method can refer to Figure 4 Step 404 in an embodiment.

[0160] The illumination processing module 905 is used to perform illumination processing on the first rendering data based on the light source configuration information to obtain a target image, wherein the target image includes an image of the first rendering object under the rendering perspective and illumination conditions. For specific implementation methods, please refer to Figure 4 Step 405 in the embodiment.

[0161] Optionally, the device 900 further includes:

[0162] A first acquisition module is used to acquire a first neural radiation field NeRF model, where the first NeRF model is used to predict an arbitrary viewing angle image of a first rendering object;

[0163] The second acquisition module is used to acquire multiple first images in multiple first poses based on the first NeRF model.

[0164] Optionally, the first rendering data further includes initial light source information, and the apparatus 900 further includes:

[0165] Providing a module for providing a light source configuration interface;

[0166] A display module, used to display initial light source information on the light source configuration interface;

[0167] The third acquisition module is used to respond to the user's editing operation on the light source configuration interface, acquire the light source information after the initial light source information is adjusted, and use the adjusted light source information as the light source configuration information.

[0168] Optionally, the first rendering data further includes initial light source information, and the apparatus 900 further includes:

[0169] Providing a module for providing a light source configuration interface;

[0170] A display module, used to display initial light source information on the light source configuration interface;

[0171] The second determining module is used to determine the initial light source information as the light source configuration information in response to a confirmation operation of the user on the light source configuration interface.

[0172] Optionally, the device 900 further includes:

[0173] A third acquisition module is used to acquire an explicit object model of a second rendering object, where the second rendering object is located in the target scene;

[0174] A geometry processing module, configured to perform geometry processing on the explicit object model based on the rendering pose information to obtain second rendering data of a second rendering object, wherein the second rendering data includes one or more of normal information, depth information, color information, and material information of the second rendering object;

[0175] The lighting processing module is used to:

[0176] Lighting processing is performed on the first rendering data and the second rendering data based on the light source configuration information to obtain a target image, wherein the target image includes images of the first rendering object and the second rendering object under rendering viewing angles and lighting conditions.

[0177] Optionally, the device 900 further includes:

[0178] a third input module, configured to input a plurality of second images of a third rendering object into a second implicit object model training network to generate a second implicit object model of the third rendering object, wherein the plurality of second images are images of the third rendering object in a plurality of second postures, the second implicit object model is a neural network model, and the third rendering object is located in a target scene;

[0179] a fourth input module, configured to input the rendering pose information into the second implicit object model to obtain third rendering data of a third rendering object, wherein the third rendering data includes one or more of normal information, depth information, color information, and material information of the third rendering object under the rendering pose information;

[0180] The lighting processing module is used to:

[0181] The first rendering data and the third rendering data are subjected to illumination processing based on the light source configuration information to obtain a target image, wherein the target image includes images of the first rendering object and the third rendering object under a rendering viewing angle and illumination conditions.

[0182] Optionally, the first input module is used to:

[0183] determining one or more of normal information, depth information, color information, and material information of each of the plurality of first images;

[0184] The first implicit object model training network is trained based on one or more of normal information, depth information, color information, and material information of each first image in the plurality of first images to generate a first implicit object model.

[0185] In summary, in the image rendering method provided in the embodiment of the present application, a new type of implicit object model is generated through multiple images. Compared with the relevant implicit object model, the implicit object model provided in the embodiment of the present application can output rendering data that matches the geometric buffer in the rendering pipeline, so as to combine the output of the implicit object model with the lighting processing operation of the rendering pipeline for the explicit object model. Since the light source configuration information in the lighting processing operation of the rendering pipeline can be flexibly set, the rendering effect in the scene where the light source environment changes dynamically can be improved while ensuring the rendering realism.

[0186] Among them, the first input module 901, the receiving module 902, the second input module 903, the first determination module 904 and the illumination processing module 905 can all be implemented by software, or can be implemented by hardware. Exemplarily, the implementation of the first determination module 904 is introduced below by taking the first determination module 904 as an example. Similarly, the implementation of the first input module 901, the receiving module 902, the second input module 903 and the illumination processing module 905 can refer to the implementation of the first determination module 904.

[0187] As an example of a software functional unit, the first determination module 904 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the first determination module 904 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0188] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0189] As an example of a hardware functional unit, the first determination module 904 may include at least one computing device, such as a server, etc. Alternatively, the first determination module 904 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0190] The multiple computing devices included in the first determination module 904 may be distributed in the same region or in different regions. The multiple computing devices included in the first determination module 904 may be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first determination module 904 may be distributed in the same VPC or in multiple VPCs. The multiple computing devices may be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0191] It should be noted that, in other embodiments, the first determination module 904 can be used to execute any step in the image rendering method, the first input module 901 can be used to execute any step in the image rendering method, the receiving module 902 can be used to execute any step in the image rendering method, the second input module 903 can be used to execute any step in the image rendering method, and the illumination processing module 905 can be used to execute any step in the image rendering method. The steps that the first input module 901, the receiving module 902, the second input module 903, the first determination module 904, and the illumination processing module 905 are responsible for implementing can be specified as needed, and the first input module 901, the receiving module 902, the second input module 903, the first determination module 904, and the illumination processing module 905 respectively implement different steps in the image rendering method to realize the full functions of the image rendering device.

[0192] The present application embodiment also provides a computing device 1000. Fig.10 As shown, computing device 1000 includes: bus 1002, processor 1004, memory 1006 and communication interface 1008. Processor 1004, memory 1006 and communication interface 1008 communicate through bus 1002. Computing device 1000 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 1000.

[0193] The bus 1002 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 The bus 1004 may include a path for transmitting information between various components of the computing device 1000 (eg, the memory 1006, the processor 1004, and the communication interface 1008).

[0194] The processor 1004 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0195] The memory 1006 may include a volatile memory, such as a random access memory (RAM). The processor 1004 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0196] The memory 1006 stores executable program codes, and the processor 1004 executes the executable program codes to respectively implement the functions of the first input module 901, the receiving module 902, the second input module 903, the first determination module 904, and the illumination processing module 905, thereby implementing the image rendering method. That is, the memory 1006 stores instructions for executing the image rendering method.

[0197] Alternatively, the memory 1006 stores executable codes, and the processor 1004 executes the executable codes to respectively implement the functions of the aforementioned image rendering apparatus, thereby implementing the image rendering method. That is, the memory 1006 stores instructions for executing the image rendering method.

[0198] The communication interface 1003 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1000 and other devices or a communication network.

[0199] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0200] like Fig.11 As shown, the computing device cluster includes at least one computing device 1000. The memory 1006 in one or more computing devices 1000 in the computing device cluster may store the same instructions for executing the image processing method.

[0201] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster may also store partial instructions for executing the image rendering method. In other words, the combination of one or more computing devices 1000 may jointly execute instructions for executing the image rendering method.

[0202] It should be noted that the memory 1006 in different computing devices 1000 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the image rendering apparatus. That is, the instructions stored in the memory 1006 in different computing devices 1000 may implement the functions of one or more of the first input module 901, the receiving module 902, the second input module 903, the first determination module 904 and the illumination processing module 905.

[0203] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.12 A possible implementation is shown. Fig.12 As shown, two computing devices 1000A and 1000B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 1006 in the computing device 1000A stores instructions for executing the functions of the first input module 901, the receiving module 902, and the second input module 903. At the same time, the memory 1006 in the computing device 1000B stores instructions for executing the functions of the first determination module 904 and the illumination processing module 905.

[0204] Fig.12 The connection method between the computing device clusters shown can be that considering that the image rendering method provided in this application needs to interact with the user and a large amount of calculation is required in the background, it is considered that the functions implemented by the first input module 901, the receiving module 902, and the second input module 903 are executed by the computing device 1000A, and the functions implemented by the first determination module 904 and the lighting processing module 905 are executed by the computing device 1000B.

[0205] It should be understood that Fig.12 The functions of the computing device 1000A shown in FIG. 1000A may also be completed by multiple computing devices 1000. Similarly, the functions of the computing device 1000B may also be completed by multiple computing devices 1000.

[0206] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the image processing method.

[0207] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the image processing method.

[0208] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. An image rendering method, characterized in that: The method comprises: Inputting a plurality of first images of a first rendering object into a first implicit object model training network to generate a first implicit object model of the first rendering object, wherein the plurality of first images are images of the first rendering object in a plurality of first poses, the first implicit object model is a neural network model, and the first rendering object is located in a target scene; Receiving rendering pose information, where the rendering pose information is used to indicate a pose of a rendering perspective of the target scene; Inputting the rendering pose information into the first implicit object model to obtain first rendering data of the first rendering object, wherein the first rendering data includes one or more of normal information, depth information, color information, and material information of the first rendering object at the rendering perspective; Determining light source configuration information, where the light source configuration information is used to indicate the lighting condition of the target scene; The first rendering data is subjected to illumination processing based on the light source configuration information to obtain the target image, wherein the target image includes an image of the first rendering object under the rendering viewing angle and the illumination condition.

2. The method according to claim 1, characterized in that Before inputting the plurality of first images into the first implicit object model training network, the method further includes: Acquire a first neural radiation field NeRF model, where the first NeRF model is used to predict an arbitrary viewing angle image of the first rendered object; Based on the first NeRF model, the multiple first images in the multiple first poses are acquired.

3. The method according to claim 1 or 2, characterized in that The first rendering data further includes initial light source information, and the method further includes: Provide light source configuration interface; Displaying the initial light source information on the light source configuration interface; In response to the user's editing operation on the light source configuration interface, light source information after adjusting the initial light source information is obtained, and the adjusted light source information is used as the light source configuration information.

4. The method according to claim 1 or 2, characterized in that: The first rendering data further includes initial light source information, and the method further includes: Provide light source configuration interface; Displaying the initial light source information on the light source configuration interface; In response to a confirmation operation of the user on the light source configuration interface, the initial light source information is determined as the light source configuration information.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Obtaining an explicit object model of a second rendering object, the second rendering object being located in the target scene; Based on the rendering pose information, geometrically process the explicit object model to obtain second rendering data of the second rendering object, where the second rendering data includes one or more of normal information, depth information, color information, and material information of the second rendering object; The performing illumination processing on the first rendering data based on the light source configuration information to obtain the target image includes: The first rendering data and the second rendering data are subjected to illumination processing based on the light source configuration information to obtain the target image, wherein the target image includes images of the first rendering object and the second rendering object under the rendering viewing angle and the illumination condition.

6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Inputting a plurality of second images of a third rendered object into a second implicit object model training network to generate a second implicit object model of the third rendered object, wherein the plurality of second images are images of the third rendered object in a plurality of second postures, the second implicit object model is a neural network model, and the third rendered object is located in the target scene; Inputting the rendering pose information into the second implicit object model to obtain third rendering data of the third rendering object, wherein the third rendering data includes one or more of normal information, depth information, color information, and material information of the third rendering object under the rendering pose information; The performing illumination processing on the first rendering data based on the light source configuration information to obtain the target image includes: The first rendering data and the third rendering data are subjected to illumination processing based on the light source configuration information to obtain the target image, wherein the target image includes images of the first rendering object and the third rendering object under the rendering viewing angle and the illumination condition.

7. The method according to any one of claims 1 to 6, characterized in that: The step of inputting the plurality of first images into a first implicit object model training network to generate a first implicit object model of the first rendered object comprises: Determine one or more of normal information, depth information, color information, and material information of each of the plurality of first images; The first implicit object model training network is trained based on one or more of normal information, depth information, color information, and material information of each of the plurality of first images to generate the first implicit object model.

8. An image rendering device, characterized in that: The device comprises: a first input module, configured to input a plurality of first images of a first rendering object into a first implicit object model training network to generate a first implicit object model of the first rendering object, wherein the plurality of first images are images of the first rendering object in a plurality of first poses, the first implicit object model is a neural network model, and the first rendering object is located in a target scene; A receiving module, used for receiving rendering posture information, wherein the rendering posture information is used for indicating the posture of the rendering perspective of the target scene; A second input module, inputting the rendering pose information into the first implicit object model to obtain first rendering data of the first rendering object, wherein the first rendering data includes one or more of normal information, depth information, color information and material information of the first rendering object at the rendering perspective; A first determining module, used to determine light source configuration information, where the light source configuration information is used to indicate the lighting condition of the target scene; A lighting processing module is used to perform lighting processing on the first rendering data based on the light source configuration information to obtain the target image, wherein the target image includes an image of the first rendering object under the rendering viewing angle and the lighting condition.

9. The device according to claim 8, characterized in that The device also includes: A first acquisition module, used for acquiring a first neural radiation field NeRF model, where the first NeRF model is used for predicting an arbitrary viewing angle image of the first rendered object; The second acquisition module is used to acquire the multiple first images in the multiple first poses based on the first NeRF model.

10. The device according to claim 8 or 9, characterized in that The first rendering data further includes initial light source information, and the device further includes: Providing a module for providing a light source configuration interface; A display module, used for displaying the initial light source information on the light source configuration interface; The third acquisition module is used to obtain the light source information after the initial light source information is adjusted in response to the user's editing operation on the light source configuration interface, and use the adjusted light source information as the light source configuration information.

11. The device according to claim 8 or 9, characterized in that The first rendering data further includes initial light source information, and the device further includes: Providing a module for providing a light source configuration interface; A display module, used for displaying the initial light source information on the light source configuration interface; The second determining module is configured to determine the initial light source information as the light source configuration information in response to a confirmation operation of the user on the light source configuration interface.

12. The device according to any one of claims 8 to 11, characterized in that: The device also includes: A third acquisition module, configured to acquire an explicit object model of a second rendering object, where the second rendering object is located in the target scene; a geometry processing module, configured to perform geometry processing on the explicit object model based on the rendering pose information to obtain second rendering data of the second rendering object, wherein the second rendering data includes one or more of normal information, depth information, color information, and material information of the second rendering object; The illumination processing module is used for: The first rendering data and the second rendering data are subjected to illumination processing based on the light source configuration information to obtain the target image, wherein the target image includes images of the first rendering object and the second rendering object under the rendering viewing angle and the illumination condition.

13. The device according to any one of claims 8 to 11, characterized in that: The device also includes: a third input module, configured to input a plurality of second images of a third rendering object into a second implicit object model training network to generate a second implicit object model of the third rendering object, wherein the plurality of second images are images of the third rendering object in a plurality of second postures, the second implicit object model is a neural network model, and the third rendering object is located in the target scene; a fourth input module, configured to input the rendering pose information into the second implicit object model to obtain third rendering data of the third rendering object, wherein the third rendering data includes one or more of normal information, depth information, color information and material information of the third rendering object under the rendering pose information; The illumination processing module is used for: The first rendering data and the third rendering data are subjected to illumination processing based on the light source configuration information to obtain the target image, wherein the target image includes images of the first rendering object and the third rendering object under the rendering viewing angle and the illumination condition.

14. The device according to any one of claims 8 to 13, characterized in that: The first input module is used for: Determine one or more of normal information, depth information, color information, and material information of each of the plurality of first images; The first implicit object model training network is trained based on one or more of normal information, depth information, color information, and material information of each of the plurality of first images to generate the first implicit object model.

15. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1-7.

16. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 7.

17. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 7.

Images