Image rendering method, device and electronic equipment for three-dimensional object

Abstract

The disclosed embodiment provides a method, device and electronic device for rendering an image of a three-dimensional object, the method comprising: obtaining target viewing angle information for a target object; determining the position information of each spatial sampling point used when rendering the target object based on the target viewing angle information; using a pre-trained neural rendering network, based on the position information of each spatial sampling point and the feature map of each image to be used of the target object, generating the color information of each spatial sampling point; generating a rendered image of the target object based on the color information of each spatial sampling point. This solution can solve the problem of the lack of universality in the related art when using a neural rendering network for image rendering.

Description

Technical Field

[0001] The present disclosure relates to the field of image rendering technology, and in particular to a method, device and electronic device for rendering an image of a three-dimensional object. Background Art

[0002] New perspective image rendering of three-dimensional objects has always been an important research direction in the field of computer graphics. New perspective image rendering refers to rendering the image of the three-dimensional object from a new perspective based on images taken from multiple perspectives of the three-dimensional object. In related technologies, a pre-trained neural rendering network is used to perform new perspective image rendering of three-dimensional objects.

[0003] However, the related art uses a pre-trained neural rendering network to render new perspective images of three-dimensional objects, which only uses the position information and color information of the images taken from multiple perspectives of the three-dimensional object. This will result in the neural rendering network being able to only render new perspective images of a single object and not being universal. That is, a neural rendering network must be trained separately for each different three-dimensional object, and each training involves complex operations such as collecting data. This greatly increases the workload of technicians and is very unfriendly to people who are not in the professional field. Summary of the invention

[0004] The purpose of the embodiments of the present disclosure is to provide a method, device and electronic device for rendering an image of a three-dimensional object, so as to solve the problem that the image rendering using a neural rendering network in the related art is not universal. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present disclosure provides a method for rendering an image of a three-dimensional object, the method comprising:

[0006] Acquire target viewing angle information for a target object; wherein the target viewing angle information is viewing angle information of a viewing angle to be rendered;

[0007] Based on the target viewing angle information, determining position information of each spatial sampling point used when rendering the target object;

[0008] Using a pre-trained neural rendering network, based on the position information of each spatial sampling point and the feature map of each to-be-used image of the target object, the color information of each spatial sampling point is generated; wherein each to-be-used image is an image obtained by taking pictures from different perspectives of the target object; the neural rendering network is trained to obtain an artificial intelligence model based on the position information of the sample sampling points, the feature map of each sample image, and the true value of the image to be predicted; each of the sample images and the image to be predicted is an image taken from different perspectives of the sample object; the sample sampling points are spatial sampling points used when rendering the image to be predicted;

[0009] A rendered image of the target object is generated based on the color information of each of the spatial sampling points.

[0010] Optionally, the target viewing angle information includes: observation coordinates and observation angle;

[0011] The determining, based on the target viewing angle information, position information of each spatial sampling point used when rendering the target object includes:

[0012] Taking the observation coordinate as an endpoint and according to the observation angle, generating a virtual ray corresponding to each pixel point in the image to be rendered;

[0013] Each of the virtual rays is sampled to obtain position information of each spatial sampling point used when rendering the target object.

[0014] Optionally, the method for generating the color information of any of the spatial sampling points includes:

[0015] For each of the images to be used, determining a mapping position of the spatial sampling point in the image to be used, and determining feature information at the mapping position from a feature map of the image to be used as a sampling feature corresponding to the spatial position point;

[0016] Performing feature fusion processing on the sampling features to obtain fusion features corresponding to the spatial sampling points;

[0017] The position information of the spatial sampling points and the fusion features corresponding to the spatial sampling points are input into a pre-trained neural rendering network to generate color information of the spatial sampling points.

[0018] Optionally, for each image to be used, determining a mapping position of the spatial sampling point in the image to be used includes:

[0019] For each image to be used, based on camera calibration information of a shooting device that shoots the image to be used and the position information of the spatial sampling point, a mapping position of the spatial sampling point in the image to be used is determined.

[0020] Optionally, performing feature fusion processing on each of the sampling features to obtain a fusion feature corresponding to the spatial sampling point includes:

[0021] The pre-trained self-attention mechanism Transformer layer is used to perform feature fusion processing on each of the sampling features to obtain the fusion features corresponding to the spatial sampling points.

[0022] Optionally, the neural rendering network and the self-attention mechanism Transformer layer are obtained by joint training based on the position information of the sample sampling points, the feature maps of each of the sample images, and the true value of the image to be predicted.

[0023] Optionally, the joint training method of the neural rendering network and the self-attention mechanism Transformer layer includes:

[0024] Acquire each of the sample images and the image to be predicted;

[0025] Determining, based on the viewing angle information corresponding to the image to be predicted, the position information of the sample sampling point used when rendering the sample object;

[0026] For each of the sample images, determining a mapping position of the sample sampling point in the sample image, and determining feature information at the mapping position from an image feature map of the sample image as a sampling feature corresponding to the sample sampling point;

[0027] Using the self-attention mechanism Transformer layer in training, the sampling features of each of the sample sampling points are subjected to feature fusion processing to obtain the fusion features corresponding to the sample sampling points;

[0028] Inputting the position information of the sample sampling point and the fusion features corresponding to the sample sampling point into the neural rendering network in training, and outputting the predicted color information of the sample sampling point;

[0029] Based on the predicted color information of the sample sampling point, an image of the sample object when observed from a specified viewing angle is generated as an output image; wherein the specified viewing angle includes an observation position and an observation angle corresponding to the image to be predicted;

[0030] Calculating the model loss using the output image and the difference between the image to be predicted;

[0031] When it is determined by using the model loss that the neural rendering network has not converged, adjusting model parameters of the neural rendering network and the self-attention mechanism Transformer layer until the neural rendering network and the self-attention mechanism Transformer layer converge.

[0032] Optionally, for each of the images to be used, determining a mapping position of the spatial sampling point in the image to be used, and determining feature information at the mapping position from a feature map of the image to be used as a sampling feature corresponding to the spatial position point includes:

[0033] If the feature map of the image to be used is different from the size of the image to be used, based on the size of the image to be used, linear interpolation processing is performed on the feature map of the image to be used to obtain a target feature map with the same size as the image to be used;

[0034] Extract feature information at the mapping position in the target feature map to obtain sampling features corresponding to the spatial sampling points.

[0035] Optionally, the color information includes a color value and a density value; the density value is used to represent the weight of the spatial sampling point when the spatial sampling point is used to generate a pixel value;

[0036] The step of generating a rendered image of the target object based on the color information of each of the spatial sampling points comprises:

[0037] Performing weighted addition of the color values ​​of the spatial sampling points on each of the virtual rays according to the density value to obtain a pixel value of a pixel point in the image to be rendered corresponding to the virtual ray;

[0038] The rendered image is generated based on the pixel value of each pixel in the image to be rendered.

[0039] In a second aspect, an embodiment of the present disclosure provides an image rendering device for a three-dimensional object, the device comprising:

[0040] An acquisition module, used to acquire target viewing angle information for a target object; wherein the target viewing angle information is viewing angle information of a viewing angle to be rendered;

[0041] A determination module, used to determine the position information of each spatial sampling point used when rendering the target object based on the target viewing angle information;

[0042] The first generation module is used to generate color information of each spatial sampling point by using a pre-trained neural rendering network based on the position information of each spatial sampling point and the feature map of each to-be-used image of the target object; wherein each to-be-used image is an image obtained by taking pictures from different perspectives of the target object; the neural rendering network is trained to obtain an artificial intelligence model based on the position information of the sample sampling points, the feature map of each sample image, and the true value of the to-be-predicted image; each of the sample images and the to-be-predicted image is an image taken from different perspectives of the sample object; the sample sampling point is a spatial sampling point used when rendering the to-be-predicted image;

[0043] The second generating module is used to generate a rendering image of the target object based on the color information of each of the spatial sampling points.

[0044] Optionally, the target viewing angle information includes: observation coordinates and observation angle;

[0045] The determination module based on the target viewing angle information includes:

[0046] A first generating submodule, used for generating a virtual ray corresponding to each pixel point in the image to be rendered according to the observation angle, taking the observation coordinate as an endpoint;

[0047] The sampling module is used to sample each of the virtual rays to obtain the position information of each spatial sampling point used when rendering the target object.

[0048] Optionally, the first generating module includes:

[0049] A first determination submodule is used to determine, for each of the to-be-used images, a mapping position of the spatial sampling point in the to-be-used image, and determine feature information at the mapping position from a feature map of the to-be-used image as a sampling feature corresponding to the spatial position point;

[0050] A fusion submodule, used to perform feature fusion processing on the sampling features to obtain fusion features corresponding to the spatial sampling points;

[0051] The input submodule is used to input the position information of the spatial sampling point and the fusion features corresponding to the spatial sampling point into a pre-trained neural rendering network to generate color information of the spatial sampling point.

[0052] Optionally, the first determining submodule is specifically configured to:

[0053] For each image to be used, based on camera calibration information of a shooting device that shoots the image to be used and the position information of the spatial sampling point, a mapping position of the spatial sampling point in the image to be used is determined.

[0054] Optionally, the fusion submodule is specifically used for:

[0055] The pre-trained self-attention mechanism Transformer layer is used to perform feature fusion processing on each of the sampling features to obtain the fusion features corresponding to the spatial sampling points.

[0056] Optionally, the neural rendering network and the self-attention mechanism Transformer layer are obtained by joint training based on the position information of the sample sampling points, the feature maps of each of the sample images, and the true value of the image to be predicted.

[0057] Optionally, the joint training method of the neural rendering network and the self-attention mechanism Transformer layer includes:

[0058] Acquire each of the sample images and the image to be predicted;

[0059] Determining, based on the viewing angle information corresponding to the image to be predicted, the position information of the sample sampling point used when rendering the sample object;

[0060] For each of the sample images, determining a mapping position of the sample sampling point in the sample image, and determining feature information at the mapping position from an image feature map of the sample image as a sampling feature corresponding to the sample sampling point;

[0061] Using the self-attention mechanism Transformer layer in training, the sampling features of each of the sample sampling points are subjected to feature fusion processing to obtain the fusion features corresponding to the sample sampling points;

[0062] Inputting the position information of the sample sampling point and the fusion features corresponding to the sample sampling point into the neural rendering network in training, and outputting the predicted color information of the sample sampling point;

[0063] Based on the predicted color information of the sample sampling point, an image of the sample object when observed from a specified viewing angle is generated as an output image; wherein the specified viewing angle includes an observation position and an observation angle corresponding to the image to be predicted;

[0064] Calculating the model loss using the output image and the difference between the image to be predicted;

[0065] When it is determined by using the model loss that the neural rendering network has not converged, adjusting model parameters of the neural rendering network and the self-attention mechanism Transformer layer until the neural rendering network and the self-attention mechanism Transformer layer converge.

[0066] Optionally, the first determining submodule includes:

[0067] a linear interpolation unit, configured to, if the feature map of the image to be used is different from the size of the image to be used, perform linear interpolation processing on the feature map of the image to be used based on the size of the image to be used, so as to obtain a target feature map with the same size as the image to be used;

[0068] The extraction unit is used to extract the feature information at the mapping position in the target feature map to obtain the sampling feature corresponding to the spatial sampling point.

[0069] Optionally, the color information includes a color value and a density value; the density value is used to represent the weight of the spatial sampling point when the spatial sampling point is used to generate a pixel value;

[0070] The second generating module comprises:

[0071] An adding submodule, configured to perform weighted addition of the color values ​​of the spatial sampling points on each of the virtual rays according to the density value, so as to obtain the pixel value of the pixel point in the image to be rendered corresponding to the virtual ray;

[0072] The second generating submodule is used to generate the rendered image based on the pixel value of each pixel in the image to be rendered.

[0073] In a third aspect, an embodiment of the present disclosure provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0074] Memory, used to store computer programs;

[0075] The processor is used to implement the steps of the above-mentioned three-dimensional object image rendering method when executing the program stored in the memory.

[0076] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned three-dimensional object image rendering method are implemented.

[0077] The embodiment of the present disclosure also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the above-mentioned three-dimensional object image rendering method.

[0078] Beneficial effects of the embodiments of the present disclosure:

[0079] The image rendering method of a three-dimensional object provided by the embodiment of the present disclosure obtains the target viewing angle information for the target object; based on the target viewing angle information, determines the position information of each spatial sampling point used when rendering the target object; uses the pre-trained neural rendering network to generate the color information of each spatial sampling point based on the position information of each spatial sampling point and the feature map of each image to be used of the target object; generates the rendered image of the target object based on the color information of each spatial sampling point. It can be seen that in this scheme, since the feature map of each sample image is introduced during the training process of the neural rendering network, the neural rendering network can learn the feature information of different sample objects, so that the trained neural rendering network can be applied to different objects. In this way, for different target objects, the feature map of the image to be used of the target object and the position information of each spatial sampling point can be used to determine the color information of each spatial sampling point through the pre-trained neural rendering network, so as to further generate the rendered image of the target object, so as to have universality. It can be seen that through this scheme, the problem of lack of universality when using the neural rendering network for image rendering in the related art can be solved.

[0080] Of course, it is not necessary to achieve all the advantages described above at the same time to implement any product or method of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0082] Figure 1 A flowchart of a method for rendering an image of a three-dimensional object provided in an embodiment of the present disclosure;

[0083] Figure 2 Another flow chart of a method for rendering an image of a three-dimensional object provided by an embodiment of the present disclosure;

[0084] Figure 3 A flowchart of a method for rendering an image of a three-dimensional object in a feature processing stage provided by an embodiment of the present disclosure;

[0085] Figure 4 A flowchart of an image rendering method for a three-dimensional object in an input preprocessing stage provided by an embodiment of the present disclosure;

[0086] Figure 5 A flowchart of a method for rendering an image of a three-dimensional object in a feature extraction stage provided by an embodiment of the present disclosure;

[0087] Figure 6 A flowchart of a method for rendering an image of a three-dimensional object in a rendering phase provided by an embodiment of the present disclosure;

[0088] Figure 7 A schematic diagram of the structure of an image rendering device for a three-dimensional object provided in an embodiment of the present disclosure;

[0089] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0090] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field based on the present disclosure belong to the scope of protection of the present disclosure.

[0091] Three-dimensional reconstruction and new perspective image rendering have always been the core of the field of computer graphics. Three-dimensional reconstruction can be achieved through new perspective image rendering. When the image of a three-dimensional object at any perspective can be rendered, the three-dimensional reconstruction of the three-dimensional object is completed.

[0092] At present, with the introduction of new concepts such as digital twins and metaverse, the industry's demand for new perspective rendering is gradually increasing. Although the training speed of neural rendering networks is getting faster and faster, and the scale is getting larger and larger, the neural rendering networks in related technologies are not universal. A neural rendering network must be trained separately for each different three-dimensional object. The current training of a network for each three-dimensional object involves network design, data collection, etc., which is a very complicated operation and is only suitable for people in professional research fields. For people in non-professional fields, these operations are undoubtedly very unfriendly and form an invisible barrier. In order to enable all personnel to easily master and use neural rendering networks, a universal new perspective image rendering method is needed, so that only a few pictures of three-dimensional objects at multiple angles can be used to easily complete three-dimensional reconstruction without the need for additional complex operations.

[0093] In order to solve the problem of the lack of universality when using a neural rendering network for image rendering in the related art, the embodiments of the present disclosure provide a method, device and electronic device for rendering an image of a three-dimensional object.

[0094] The following first introduces the image rendering method of a three-dimensional object provided by an embodiment of the present disclosure.

[0095] The image rendering method of a three-dimensional object provided in the embodiment of the present disclosure can be applied to an electronic device. In practical applications, the electronic device can be a server or a terminal device, such as a computer, a smart phone, etc., which is reasonable.

[0096] The image rendering method of a three-dimensional object provided in the embodiment of the present disclosure may include the following steps:

[0097] Acquire target viewing angle information for a target object; wherein the target viewing angle information is viewing angle information of a viewing angle to be rendered;

[0098] Based on the target viewing angle information, determining position information of each spatial sampling point used when rendering the target object;

[0099] Using a pre-trained neural rendering network, based on the position information of each spatial sampling point and the feature map of each to-be-used image of the target object, the color information of each spatial sampling point is generated; wherein each to-be-used image is an image obtained by taking pictures from different perspectives of the target object; the neural rendering network is trained to obtain an artificial intelligence model based on the position information of the sample sampling points, the feature map of each sample image, and the true value of the image to be predicted; each of the sample images and the image to be predicted is an image taken from different perspectives of the sample object; the sample sampling points are spatial sampling points used when rendering the image to be predicted;

[0100] A rendered image of the target object is generated based on the color information of each of the spatial sampling points.

[0101] The image rendering method of a three-dimensional object provided by the embodiment of the present disclosure obtains the target viewing angle information for the target object; based on the target viewing angle information, determines the position information of each spatial sampling point used when rendering the target object; uses the pre-trained neural rendering network to generate the color information of each spatial sampling point based on the position information of each spatial sampling point and the feature map of each image to be used of the target object; generates the rendered image of the target object based on the color information of each spatial sampling point. It can be seen that in this scheme, since the feature map of each sample image is introduced during the training process of the neural rendering network, the neural rendering network can learn the feature information of different sample objects, so that the trained neural rendering network can be applied to different objects. In this way, for different target objects, the feature map of the image to be used of the target object and the position information of each spatial sampling point can be used to determine the color information of each spatial sampling point through the pre-trained neural rendering network, so as to further generate the rendered image of the target object, so as to have universality. It can be seen that through this scheme, the problem of lack of universality when using the neural rendering network for image rendering in the related art can be solved.

[0102] The following describes the image rendering method of a three-dimensional object provided by an embodiment of the present disclosure in conjunction with the accompanying drawings.

[0103] like Figure 1 As shown, the image rendering method of a three-dimensional object provided by the embodiment of the present disclosure may include the following steps:

[0104] S101, obtaining target viewing angle information for a target object; wherein the target viewing angle information is viewing angle information of a viewing angle to be used for image rendering;

[0105] The target object can be any three-dimensional object in reality, and the target viewing angle information can be used to represent the position and angle when observing the target object. The image rendering method of the three-dimensional object in the embodiment of the present disclosure is to render the image presented when the target object is observed at the position and angle represented by the target viewing angle information.

[0106] The target viewing angle information can be obtained by inputting the target viewing angle information through the human-computer interaction interface, or by selecting the observation position and angle, etc., or by rotating or translating the target object displayed in the human-computer interaction interface to generate the target viewing angle information.

[0107] S102, determining, based on the target viewing angle information, position information of each spatial sampling point used when rendering the target object;

[0108] In order to render an image of a certain perspective in a three-dimensional scene, the spatial sampling points can be determined first. Each pixel in the image to be rendered corresponds to multiple spatial sampling points, and each pixel is generated based on the spatial sampling points corresponding to the pixel. In addition, it can be understood that the multiple spatial sampling points corresponding to each pixel also need to be located on a straight line. Therefore, when the target perspective information includes: observation coordinates, and observation angles, the position information of each spatial sampling point used to render the target object based on the target perspective information can include steps A1-A2:

[0109] Step A1, taking the observation coordinate as an endpoint and generating a virtual ray corresponding to each pixel point in the image to be rendered according to the observation angle;

[0110] The observation angle can be represented by a direction vector. When the target object is observed from the observation coordinate, the observed area is a cone with the observation coordinate as the vertex, and the center line of the cone is parallel to the direction vector representing the observation angle. When the image to be rendered is a H*W two-dimensional image, H*W virtual rays can be uniformly generated in the cone with the observation coordinate as the endpoint, and each virtual ray corresponds to a pixel point in the image to be rendered.

[0111] Step A2: sampling each of the virtual rays to obtain position information of each spatial sampling point used when rendering the target object.

[0112] In one implementation, for each virtual ray, a predetermined number of spatial sampling points may be sampled at intervals of a predetermined length, thereby obtaining the position information of each spatial sampling point. Alternatively, in another implementation, for each virtual ray, a predetermined number of spatial sampling points may be sampled uniformly within the spatial range by triggering from an endpoint, thereby obtaining the position information of each spatial sampling point. Of course, the method of adopting each virtual ray is not limited to this.

[0113] It can be seen that generating spatial sampling points by sampling virtual rays conforms to the physical law that light propagates in a straight line, and can make the image rendered according to the spatial sampling points more realistic.

[0114] S103, using a pre-trained neural rendering network, based on the position information of each of the spatial sampling points and the feature graphs of each of the to-be-used images of the target object, generating color information of each of the spatial sampling points; wherein each of the to-be-used images is an image obtained by taking pictures from different perspectives of the target object; the neural rendering network is trained to obtain an artificial intelligence model based on the position information of the sample sampling points, the feature graphs of each of the sample images, and the true value of the to-be-predicted image; each of the sample images and the to-be-predicted image is an image obtained by taking pictures from different perspectives of the sample object; the sample sampling points are spatial sampling points used when rendering the to-be-predicted image;

[0115] When extracting the feature map here, any machine learning or deep learning feature extractor can be used, such as resnet (residual neural network), mobilnet (a lightweight neural network), etc., to extract the two-dimensional visual features of the image to be used.

[0116] In the related art, the input of the neural rendering network is the spatial coordinates and the viewing direction. After the training of the neural rendering network is completed, for a fixed spatial position and viewing direction, the output is a fixed value. However, in the disclosed embodiment, the two-dimensional visual features of the image to be used are used as a priori conditions. Different three-dimensional objects have different two-dimensional visual features, so the input visual features are different, and the output values ​​are naturally different. In this way, the trained neural rendering network has the ability to distinguish different three-dimensional objects, thereby divide and conquer. For three-dimensional object 1, the value corresponding to three-dimensional object 1 is output, and for three-dimensional object 2, the value corresponding to three-dimensional object 2 is output... Therefore, the feature map of the image to be used is used to determine the color information of each spatial sampling point, so as to further generate a rendered image of the target object, which has universality.

[0117] The neural rendering network is used to output the color information of the spatial sampling point based on the position information of the spatial sampling point and the feature map of each image to be used. In order to train the neural rendering network, a part of the image obtained by taking a sample object from different perspectives can be used as a sample image, and the other part can be used as an image to be predicted. In this way, the sample sampling points can be generated according to the perspective information when any image to be predicted is taken, and the true value of the color information of the spatial sampling point can be calibrated using the image to be predicted; then the neural rendering network in training is used to output the predicted value of the color information of the sample sampling point based on the position information of the sample sampling point and the feature map of the sample image; finally, the predicted value of the color information and the true value of the color information are used to calculate the loss value of the neural rendering network, so as to adjust the parameters of the neural rendering network based on the loss value until the neural rendering network converges to obtain a trained neural rendering network. Among them, the number of sample objects can be multiple, that is, when training the neural network, sample images and images to be predicted of different sample objects can be used.

[0118] The disclosed embodiment introduces two-dimensional visual features, which can mark different three-dimensional objects, so that the neural rendering network can learn prior knowledge of understanding three-dimensional objects, and a neural rendering network with good generalization performance can be obtained.

[0119] For the sake of clarity of the solution and layout, a specific implementation method for generating the color information of each of the spatial sampling points is introduced below in combination with other embodiments.

[0120] S104: Generate a rendered image of the target object based on the color information of each of the spatial sampling points.

[0121] Each spatial sampling point corresponds to a pixel value in the image to be rendered. For each pixel value, the pixel value can be determined by weighted addition of the color values ​​represented by the color information of the multiple spatial sampling points corresponding to the multiple spatial sampling points, or by taking the average value.

[0122] In one implementation, the color information includes a color value and a density value; illustratively, the color value may be a three-primary color RGB value; the density value is used to characterize the weight of the spatial sampling point when the spatial sampling point is used to generate a pixel value.

[0123] The step of generating a rendered image of the target object based on the color information of each of the spatial sampling points comprises:

[0124] Performing weighted addition of the color values ​​of the spatial sampling points on each of the virtual rays according to the density value to obtain a pixel value of a pixel point in the image to be rendered corresponding to the virtual ray;

[0125] The rendered image is generated based on the pixel value of each pixel in the image to be rendered.

[0126] The image rendering process is to obtain the pixel value corresponding to each virtual ray. In one implementation, an integration operation can be performed along the ray. The specific formula is as follows:

[0127]

[0128] The lowercase C(r) represents the pixel value corresponding to the virtual ray, c represents the color value, such as the RGB value of the three primary colors, r represents the virtual ray, d represents the direction of the virtual ray, σ represents the density value, and t f ,t n Indicates the range of the integral.

[0129] In practical applications, the integration operation can be converted into a weighted addition of the spatial sampling points based on the density value to obtain the pixel value of the pixel point in the image to be rendered corresponding to the virtual ray, so as to realize the rendering of the image using a computer. If each virtual ray has n spatial sampling points, it is necessary to query the neural rendering network n times. After the values ​​of these n points are calculated, the pixel value of a pixel point in the image to be rendered can be obtained, thereby realizing the rendering of a point. If the resolution of the image to be rendered is (H, W), the above step of obtaining the pixel value of a pixel point is repeated H*W times to generate the image to be rendered.

[0130] The image rendering method of a three-dimensional object provided by the embodiment of the present disclosure obtains the target viewing angle information for the target object; based on the target viewing angle information, determines the position information of each spatial sampling point used when rendering the target object; uses the pre-trained neural rendering network to generate the color information of each spatial sampling point based on the position information of each spatial sampling point and the feature map of each image to be used of the target object; generates the rendered image of the target object based on the color information of each spatial sampling point. It can be seen that in this scheme, since the feature map of each sample image is introduced during the training process of the neural rendering network, the neural rendering network can learn the feature information of different sample objects, so that the trained neural rendering network can be applied to different objects. In this way, for different target objects, the feature map of the image to be used of the target object and the position information of each spatial sampling point can be used to determine the color information of each spatial sampling point through the pre-trained neural rendering network, so as to further generate the rendered image of the target object, so as to have universality. It can be seen that through this scheme, the problem of lack of universality when using the neural rendering network for image rendering in the related art can be solved.

[0131] Optionally, in another embodiment, with respect to step S103, the method for generating the color information of any of the spatial sampling points may include steps B1-B4:

[0132] Step B1, for each of the images to be used, determining the mapping position of the spatial sampling point in the image to be used, and determining feature information at the mapping position from a feature map of the image to be used as a sampling feature corresponding to the spatial position point;

[0133] At this time, the extracted feature map also needs to maintain consistency with the original image position, so that the spatial sampling points can be mapped to the corresponding positions of the feature map, so that the features corresponding to the spatial sampling points can be obtained from the feature map.

[0134] The step of determining, for each image to be used, a mapping position of the spatial sampling point in the image to be used includes:

[0135] For each image to be used, based on camera calibration information of a shooting device that shoots the image to be used and the position information of the spatial sampling point, a mapping position of the spatial sampling point in the image to be used is determined.

[0136] Exemplarily, a camera calibration method, such as colmap (a three-dimensional reconstruction method), can be used to obtain the camera calibration information of the shooting device when shooting the image to be used of the target object. The camera calibration information includes camera intrinsic parameters and camera extrinsic parameters. The camera extrinsic parameters are used to convert the world coordinate system to the camera coordinate system, and the camera intrinsic parameters are used to convert the camera coordinate system to the pixel coordinate system. The camera intrinsic parameters and camera extrinsic parameters can be used to correspond any point in the space, such as a spatial sampling point, to the pixel point of the image taken by the shooting device, that is, the mapping position. Both camera intrinsic parameters and camera extrinsic parameters are prior art and will not be further introduced here.

[0137] In practical applications, for each image to be used, the coordinates represented by the position information of the spatial sampling points, as well as the camera intrinsic parameters and camera extrinsic parameters of the camera calibration information, can be used to obtain the mapping position of the spatial sampling points in the image to be used, which can be expressed as follows:

[0138] in, K1 is the camera internal parameter, K2 is the camera external parameter, both are known inputs. u, v are the two-dimensional coordinates in the pixel coordinate system, X w , Y w , Z w are the coordinates of the spatial sampling points.

[0139] By calculating the values ​​of u and v, the mapping position of the spatial sampling point in the image to be used can be obtained, thereby obtaining the sampling features corresponding to the spatial sampling point.

[0140] Step B2, performing feature fusion processing on the sampling features to obtain fusion features corresponding to the spatial sampling points;

[0141] Each sampling point can correspond to a mapping position in each image to be used to obtain multiple sampling features. Therefore, it is necessary to perform feature fusion processing on each sampling feature corresponding to each sampling point, such as splicing processing, linear addition processing, etc., to obtain the fusion feature of the sampling point.

[0142] Exemplarily, in other implementations, a pre-trained self-attention mechanism Transformer layer may be used to perform feature fusion processing on each of the sampling features to obtain fused features corresponding to the spatial sampling points.

[0143] The Transformer layer can learn the relationship between each pair of features. Since the Transformer layer can effectively fuse multiple extracted features, it can retain the correlation between different perspectives and retain the fusion features with guiding significance, ensuring that more complete three-dimensional information can be extracted, and the rendering of three-dimensional images can be guided more efficiently, thereby enhancing the effect of rendered images.

[0144] Step B3: input the position information of the spatial sampling points and the fusion features corresponding to the spatial sampling points into a pre-trained neural rendering network to generate color information of the spatial sampling points.

[0145] In the disclosed embodiment, for each image to be used, the mapping position of the spatial sampling point in the image to be used is determined, and the feature information at the mapping position is determined from the feature map of the image to be used as the sampling feature corresponding to the spatial position point; the sampling feature is subjected to feature fusion processing to obtain the fusion feature corresponding to the spatial sampling point; the position information of the spatial sampling point and the fusion feature corresponding to the spatial sampling point are input into a pre-trained neural rendering network to generate the color information of the spatial sampling point. In this solution, by performing feature fusion processing on the sampling features of the spatial sampling point at the mapping position of the image to be used, the fusion feature corresponding to each spatial sampling point is obtained, so that the color information of the spatial sampling point can be generated more accurately based on the fusion feature corresponding to the spatial sampling point.

[0146] Optionally, in another embodiment, for any of the images to be used, determining the mapping position of the spatial sampling point in the image to be used, and determining feature information at the mapping position from a feature map of the image to be used as the sampling feature corresponding to the spatial position point may include steps C1-C2:

[0147] Step C1, if the feature map of the image to be used is different from the size of the image to be used, based on the size of the image to be used, linear interpolation processing is performed on the feature map of the image to be used to obtain a target feature map with the same size as the image to be used;

[0148] Considering that the extracted feature map is often smaller than the original image, the feature map of the image to be used can be processed using a linear interpolation method. For example, the feature map of the image to be used can be processed using a nearest neighbor interpolation algorithm, a bilinear interpolation algorithm, etc., so that the processed feature map has the same size as the image to be used.

[0149] Step C2, extracting feature information at the mapping position in the target feature map to obtain sampling features corresponding to the spatial sampling points.

[0150] In this way, the mapping position in the target feature map can be directly found according to the position where the spatial sampling point is mapped to the image to be used, thereby extracting the sampling features of the spatial sampling point.

[0151] In this embodiment, when the size of the feature map of the image to be used is different from that of the image to be used, a linear interpolation process is performed on the feature map of the image to be used based on the size of the image to be used to obtain a target feature map of the same size as that of the image to be used, and then the feature information at the mapping position in the target feature map is extracted to obtain the sampling features corresponding to the spatial sampling points. In this embodiment, the problem that the size of the feature map of the image to be used is different from that of the image to be used is solved by performing a linear interpolation process on the feature map of the image to be used, so that the sampling features corresponding to the spatial sampling points can be directly obtained.

[0152] Optionally, in another embodiment, the neural rendering network and the self-attention mechanism Transformer layer are obtained by joint training based on the position information of the sample sampling points, the feature maps of each of the sample images, and the true value of the image to be predicted.

[0153] Specifically, the joint training method of the neural rendering network and the self-attention mechanism Transformer layer may include steps D1-D8:

[0154] Step D1, obtaining each of the sample images and the image to be predicted;

[0155] In one implementation, multiple images can be obtained by simultaneously photographing the sample object from different perspectives, with a portion of the images used as sample images and another portion of the images used as images to be predicted. In addition, there can be multiple sample objects.

[0156] Step D2, determining the position information of the sample sampling points used when rendering the sample object based on the viewing angle information corresponding to the image to be predicted;

[0157] The corresponding viewing angle information of the image to be predicted may include the position and angle representing the target object when being observed, and may also be represented by spatial coordinates and a direction vector. Determining the position information of the sample sampling points according to the corresponding viewing angle information of the image to be predicted is similar to the above step S102, which determines the position information of each spatial sampling point used when rendering the target object based on the target viewing angle information.

[0158] Step D3, for each of the sample images, determining a mapping position of the sample sampling point in the sample image, and determining feature information at the mapping position from an image feature map of the sample image as a sampling feature corresponding to the sample sampling point;

[0159] Acquiring the sampling features corresponding to the sample sampling points is similar to the above step B1, in which, for each of the images to be used, the mapping position of the spatial sampling points in the image to be used is determined, and the feature information at the mapping position is determined from the feature map of the image to be used as the sampling features corresponding to the spatial position points, and will not be repeated here.

[0160] Step D4, using the self-attention mechanism Transformer layer in training, performing feature fusion processing on each of the sampling features of the sample sampling point to obtain the fusion feature corresponding to the sample sampling point;

[0161] Step D5, inputting the position information of the sample sampling point and the fusion features corresponding to the sample sampling point into the neural rendering network in training, and outputting the predicted color information of the sample sampling point;

[0162] The predicted color information may also include a color value and a density value.

[0163] Step D6, based on the predicted color information of the sample sampling point, generating an image of the sample object when observed from a specified viewing angle as an output image; wherein the specified viewing angle includes an observation position and an observation angle corresponding to the image to be predicted;

[0164] The method of generating the output image is similar to the above step S104, in which the rendering image of the target object is generated based on the color information of each of the spatial sampling points, and will not be described in detail here.

[0165] Step D7, calculating the model loss using the output image and the difference between the image to be predicted;

[0166] Step D8: When it is determined by using the model loss that the neural rendering network has not converged, adjust the model parameters of the neural rendering network and the self-attention mechanism Transformer layer until the neural rendering network and the self-attention mechanism Transformer layer converge.

[0167] By taking multiple images taken from different perspectives of the sample object as sample images and images to be predicted, and finally using the images to be predicted as true values ​​and the output images obtained by the neural rendering model to calculate the loss value, the model parameters of the neural rendering network and the self-attention mechanism Transformer layer are adjusted until the neural rendering network and the self-attention mechanism Transformer layer converge. By using the images obtained by real shooting as the true values ​​of the output images, the accuracy of the trained neural rendering network and the self-attention mechanism Transformer layer in processing new perspective image rendering problems is improved.

[0168] In this embodiment, the effect of the image rendering method of the three-dimensional object provided in the embodiment of the present disclosure can be further improved by jointly training the neural rendering network and the self-attention mechanism Transformer layer.

[0169] In order to facilitate understanding of the image rendering method of the three-dimensional object provided by the present disclosure, Figure 2-Figure 6 Provide an exemplary introduction.

[0170] like Figure 2 As shown, first, it is necessary to obtain a 2D image of the target object at a known viewing angle, and use a feature extraction module to obtain feature maps of these 2D images, wherein the feature extraction module is used to extract visual features of the 2D image of the target object at a known viewing angle to obtain feature maps of these 2D images. At the same time, the target viewing angle information is input and a preprocessing operation is performed, that is, the target viewing angle information is used to generate a 3D virtual ray, and further generate a 3D space sampling point;

[0171] The coordinate x of the spatial sampling point is projected into each two-dimensional image, and the feature map of the two-dimensional image is linearly interpolated. Further, the mapping position corresponding to the spatial sampling point is obtained, so as to extract the sampling features of the spatial sampling point; the multiple sampling features corresponding to each spatial sampling point are subjected to feature fusion processing using the Transformer feature fusion module to obtain the fusion feature W(x) of the spatial sampling point;

[0172] The position information of the spatial sampling point and the target viewing angle information (x, d), where x is the coordinate of the spatial sampling point and d is the coordinate and direction represented by the target viewing angle information, are input into the neural rendering network module, and the fusion feature W(x) is also input into the neural rendering network module, and the color information of the spatial sampling point is output, (r, g, b, σ), where r, g, b represent the three primary color values ​​and σ represents the density value; wherein the neural rendering network module is used to output the color information of the spatial sampling point using the position information of the spatial sampling point and the corresponding fusion feature;

[0173] The color information of the spatial sampling points is input into the three-dimensional rendering module, and a rendered image of the target object is output; wherein the three-dimensional rendering module is used to obtain the pixel value corresponding to each virtual ray using the color information of each spatial sampling point on each virtual ray, and finally obtain the pixel value of each pixel point of the image to be rendered, and output the rendered image.

[0174] like Figure 3 As shown, the feature extraction stage in the image rendering method of a three-dimensional object provided in the embodiment of the present disclosure may include:

[0175] S1-1, obtaining a two-dimensional image of a three-dimensional object at a known viewing angle;

[0176] S1-2, extract features from these two-dimensional images.

[0177] like Figure 4 As shown, the input preprocessing stage of the image rendering method of a three-dimensional object provided in the embodiment of the present disclosure may include:

[0178] S2-1, obtain the target perspective;

[0179] S2-2, generating virtual rays in three-dimensional space;

[0180] S2-3, sampling the virtual ray.

[0181] like Figure 5 As shown, the feature extraction stage of the image rendering method of a three-dimensional object provided in the embodiment of the present disclosure may include:

[0182] S3-1, obtain the sampling spatial sampling points;

[0183] S3-2, spatial sampling points are projected onto a two-dimensional image;

[0184] S3-3, extract the features at the corresponding position of the feature map after linear interpolation;

[0185] S3-4, performing feature fusion processing on the features of the spatial sampling points;

[0186] Get the characteristic information of the spatial sampling points.

[0187] like Figure 6 As shown, the rendering phase of the image rendering method of a three-dimensional object provided in the embodiment of the present disclosure may include:

[0188] S4-1, inputting the position information and feature information into the neural rendering network;

[0189] Use neural rendering network to generate color information for each spatial sampling point

[0190] S4-2, performing volume rendering operation along the spatial sampling points on each virtual ray;

[0191] That is, according to the color information of each spatial sampling point on the virtual ray, the pixel value of the pixel point corresponding to the virtual ray is calculated.

[0192] S4-3, generate a target image.

[0193] That is, the rendered image of the target three-dimensional image.

[0194] The image rendering method of a three-dimensional object provided by the embodiment of the present disclosure obtains the target viewing angle information for the target object; based on the target viewing angle information, determines the position information of each spatial sampling point used when rendering the target object; uses the pre-trained neural rendering network to generate the color information of each spatial sampling point based on the position information of each spatial sampling point and the feature map of each image to be used of the target object; generates the rendered image of the target object based on the color information of each spatial sampling point. It can be seen that in this scheme, since the feature map of each sample image is introduced during the training process of the neural rendering network, the neural rendering network can learn the feature information of different sample objects, so that the trained neural rendering network can be applied to different objects. In this way, for different target objects, the feature map of the image to be used of the target object and the position information of each spatial sampling point can be used to determine the color information of each spatial sampling point through the pre-trained neural rendering network, so as to further generate the rendered image of the target object, so as to have universality. It can be seen that through this scheme, the problem of lack of universality when using the neural rendering network for image rendering in the related art can be solved.

[0195] The present disclosure also provides a device for rendering an image of a three-dimensional object. Figure 7 As shown, the device comprises:

[0196] An acquisition module 710 is used to acquire target viewing angle information for a target object; wherein the target viewing angle information is viewing angle information of a viewing angle to be rendered;

[0197] A determination module 720, configured to determine, based on the target viewing angle information, position information of each spatial sampling point used when rendering the target object;

[0198] The first generation module 730 is used to generate color information of each spatial sampling point based on the position information of each spatial sampling point and the feature map of each to-be-used image of the target object using a pre-trained neural rendering network; wherein each to-be-used image is an image obtained by taking pictures from different perspectives of the target object; the neural rendering network is trained to obtain an artificial intelligence model based on the position information of the sample sampling points, the feature map of each sample image, and the true value of the to-be-predicted image; each of the sample images and the to-be-predicted image is an image taken from different perspectives of the sample object; the sample sampling point is a spatial sampling point used when rendering the to-be-predicted image;

[0199] The second generating module 740 is configured to generate a rendering image of the target object based on the color information of each of the spatial sampling points.

[0200] Optionally, the target viewing angle information includes: observation coordinates and observation angle;

[0201] The determination module based on the target viewing angle information includes:

[0202] A first generating submodule, used for generating a virtual ray corresponding to each pixel point in the image to be rendered according to the observation angle, taking the observation coordinate as an endpoint;

[0203] The sampling module is used to sample each of the virtual rays to obtain the position information of each spatial sampling point used when rendering the target object.

[0204] Optionally, the first generating module includes:

[0205] A first determination submodule is used to determine, for each of the images to be used, a mapping position of the spatial sampling point in the image to be used, and determine feature information at the mapping position from a feature map of the image to be used as a sampling feature corresponding to the spatial position point;

[0206] A fusion submodule, used to perform feature fusion processing on the sampling features to obtain fusion features corresponding to the spatial sampling points;

[0207] The input submodule is used to input the position information of the spatial sampling point and the fusion features corresponding to the spatial sampling point into a pre-trained neural rendering network to generate color information of the spatial sampling point.

[0208] Optionally, specifically for:

[0209] For each image to be used, based on camera calibration information of a shooting device that shoots the image to be used and the position information of the spatial sampling point, a mapping position of the spatial sampling point in the image to be used is determined.

[0210] Optionally, the fusion submodule is specifically used to:

[0211] The pre-trained self-attention mechanism Transformer layer is used to perform feature fusion processing on each of the sampling features to obtain the fusion features corresponding to the spatial sampling points.

[0212] Optionally, the neural rendering network and the self-attention mechanism Transformer layer are obtained by joint training based on the position information of the sample sampling points, the feature maps of each of the sample images, and the true value of the image to be predicted.

[0213] Optionally, the joint training method of the neural rendering network and the self-attention mechanism Transformer layer includes:

[0214] Acquire each of the sample images and the image to be predicted;

[0215] Determining, based on the viewing angle information corresponding to the image to be predicted, position information of a sample sampling point used when rendering the sample object;

[0216] For each of the sample images, determining a mapping position of the sample sampling point in the sample image, and determining feature information at the mapping position from an image feature map of the sample image as a sampling feature corresponding to the sample sampling point;

[0217] Using the self-attention mechanism Transformer layer in training, the sampling features of each of the sample sampling points are subjected to feature fusion processing to obtain the fusion features corresponding to the sample sampling points;

[0218] Inputting the position information of the sample sampling point and the fusion features corresponding to the sample sampling point into the neural rendering network in training, and outputting the predicted color information of the sample sampling point;

[0219] Based on the predicted color information of the sample sampling point, an image of the sample object when observed from a specified viewing angle is generated as an output image; wherein the specified viewing angle includes an observation position and an observation angle corresponding to the image to be predicted;

[0220] Calculating the model loss using the output image and the difference between the image to be predicted;

[0221] When it is determined by using the model loss that the neural rendering network has not converged, adjusting model parameters of the neural rendering network and the self-attention mechanism Transformer layer until the neural rendering network and the self-attention mechanism Transformer layer converge.

[0222] Optionally, the first determining submodule includes:

[0223] a linear interpolation unit, configured to, if the feature map of the image to be used is different from the size of the image to be used, perform linear interpolation processing on the feature map of the image to be used based on the size of the image to be used, so as to obtain a target feature map with the same size as the image to be used;

[0224] The extraction unit is used to extract the feature information at the mapping position in the target feature map to obtain the sampling feature corresponding to the spatial sampling point.

[0225] Optionally, the color information includes a color value and a density value; the density value is used to represent the weight of the spatial sampling point when the spatial sampling point is used to generate a pixel value;

[0226] The second generating module comprises:

[0227] An adding submodule, configured to perform weighted addition of the color values ​​of the spatial sampling points on each of the virtual rays according to the density value, so as to obtain the pixel value of the pixel point in the image to be rendered corresponding to the virtual ray;

[0228] The second generating submodule is used to generate the rendered image based on the pixel value of each pixel in the image to be rendered.

[0229] The present disclosure also provides an electronic device, such as Figure 8 As shown, it includes a processor 801, a communication interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804.

[0230] Memory 803, used for storing computer programs;

[0231] The processor 801 is used to implement the steps of the above-mentioned three-dimensional object image rendering method when executing the program stored in the memory 803.

[0232] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0233] The communication interface is used for communication between the above electronic device and other devices.

[0234] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0235] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0236] In another embodiment provided by the present disclosure, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned three-dimensional object image rendering method are implemented.

[0237] In another embodiment provided by the present disclosure, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the above-mentioned three-dimensional object image rendering method.

[0238] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0239] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0240] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0241] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the protection scope of the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure are included in the protection scope of the present disclosure.

Claims

1. A method for rendering an image of a three-dimensional object, characterized in that: The method comprises: Acquire target viewing angle information for a target object; wherein the target viewing angle information is viewing angle information of a viewing angle to be rendered; Based on the target viewing angle information, determining position information of each spatial sampling point used when rendering the target object; Using a pre-trained neural rendering network, based on the position information of each of the spatial sampling points and the feature graphs of each to-be-used image of the target object, the color information of each of the spatial sampling points is generated; wherein each of the to-be-used images is an image obtained by taking pictures from different perspectives of the target object, and the position information of the feature graph of the to-be-used image is consistent with the position information of the to-be-used image; the neural rendering network is trained to obtain an artificial intelligence model based on the position information of the sample sampling points, the feature graphs of each sample image, and the true value of the to-be-predicted image; each of the sample images and the to-be-predicted image is an image taken from different perspectives of the sample object; the sample sampling points are spatial sampling points used when rendering the to-be-predicted image; A rendered image of the target object is generated based on the color information of each of the spatial sampling points.

2. The method according to claim 1, characterized in that The target viewing angle information includes: observation coordinates and observation angle; The determining, based on the target viewing angle information, position information of each spatial sampling point used when rendering the target object includes: Taking the observation coordinate as an endpoint and according to the observation angle, generating a virtual ray corresponding to each pixel point in the image to be rendered; Each of the virtual rays is sampled to obtain position information of each spatial sampling point used when rendering the target object.

3. The method according to claim 1 or 2, characterized in that: The method for generating the color information of any of the spatial sampling points includes: For each of the images to be used, determining a mapping position of the spatial sampling point in the image to be used, and determining feature information at the mapping position from a feature map of the image to be used as a sampling feature corresponding to the spatial position point; Performing feature fusion processing on the sampling features to obtain fusion features corresponding to the spatial sampling points; The position information of the spatial sampling points and the fusion features corresponding to the spatial sampling points are input into a pre-trained neural rendering network to generate color information of the spatial sampling points.

4. The method according to claim 3, characterized in that The step of determining, for each image to be used, a mapping position of the spatial sampling point in the image to be used includes: For each image to be used, based on camera calibration information of a shooting device that shoots the image to be used and the position information of the spatial sampling point, a mapping position of the spatial sampling point in the image to be used is determined.

5. The method according to claim 3, characterized in that: The step of performing feature fusion processing on each of the sampling features to obtain a fusion feature corresponding to the spatial sampling point includes: The pre-trained self-attention mechanism Transformer layer is used to perform feature fusion processing on each of the sampling features to obtain the fusion features corresponding to the spatial sampling points.

6. The method according to claim 5, characterized in that The neural rendering network and the self-attention mechanism Transformer layer are obtained by joint training based on the position information of the sample sampling points, the feature maps of each of the sample images, and the true value of the image to be predicted.

7. The method according to claim 6, characterized in that The joint training method of the neural rendering network and the self-attention mechanism Transformer layer includes: Acquire each of the sample images and the image to be predicted; Determining, based on the viewing angle information corresponding to the image to be predicted, the position information of the sample sampling point used when rendering the sample object; For each of the sample images, determining a mapping position of the sample sampling point in the sample image, and determining feature information at the mapping position from an image feature map of the sample image as a sampling feature corresponding to the sample sampling point; Using the self-attention mechanism Transformer layer in training, the sampling features of each of the sample sampling points are subjected to feature fusion processing to obtain the fusion features corresponding to the sample sampling points; Inputting the position information of the sample sampling point and the fusion features corresponding to the sample sampling point into the neural rendering network in training, and outputting the predicted color information of the sample sampling point; Based on the predicted color information of the sample sampling point, an image of the sample object when observed from a specified viewing angle is generated as an output image; wherein the specified viewing angle includes an observation position and an observation angle corresponding to the image to be predicted; Calculating the model loss using the output image and the difference between the image to be predicted; When it is determined by using the model loss that the neural rendering network has not converged, adjusting model parameters of the neural rendering network and the self-attention mechanism Transformer layer until the neural rendering network and the self-attention mechanism Transformer layer converge.

8. The method according to claim 3, characterized in that The step of determining, for each of the images to be used, a mapping position of the spatial sampling point in the image to be used, and determining feature information at the mapping position from a feature map of the image to be used as a sampling feature corresponding to the spatial position point includes: If the feature map of the image to be used is different from the size of the image to be used, based on the size of the image to be used, linear interpolation processing is performed on the feature map of the image to be used to obtain a target feature map with the same size as the image to be used; Extract feature information at the mapping position in the target feature map to obtain sampling features corresponding to the spatial sampling points.

9. The method according to claim 3, characterized in that: The color information includes a color value and a density value; the density value is used to represent the weight of the spatial sampling point when the pixel value is generated by using the spatial sampling point; The step of generating a rendered image of the target object based on the color information of each of the spatial sampling points comprises: Performing weighted addition of the color values ​​of the spatial sampling points on each of the virtual rays according to the density value to obtain a pixel value of a pixel point in the image to be rendered corresponding to the virtual ray; The rendered image is generated based on the pixel value of each pixel in the image to be rendered.

10. An image rendering device for a three-dimensional object, characterized in that: The device comprises: An acquisition module, used to acquire target viewing angle information for a target object; wherein the target viewing angle information is viewing angle information of a viewing angle to be rendered; A determination module, used to determine the position information of each spatial sampling point used when rendering the target object based on the target viewing angle information; The first generation module is used to generate color information of each spatial sampling point based on the position information of each spatial sampling point and the feature map of each to-be-used image of the target object using a pre-trained neural rendering network; wherein each to-be-used image is an image obtained by taking pictures from different perspectives of the target object, and the position information of the feature map of the to-be-used image is consistent with the position information of the to-be-used image; the neural rendering network is trained to obtain an artificial intelligence model based on the position information of the sample sampling points, the feature map of each sample image, and the true value of the to-be-predicted image; each of the sample images and the to-be-predicted image is an image taken from different perspectives of the sample object; the sample sampling point is a spatial sampling point used when rendering the to-be-predicted image; The second generating module is used to generate a rendering image of the target object based on the color information of each of the spatial sampling points.

11. The device according to claim 10, characterized in that The target viewing angle information includes: observation coordinates and observation angle; The determination module based on the target viewing angle information includes: A first generating submodule, used for generating a virtual ray corresponding to each pixel point in the image to be rendered according to the observation angle, taking the observation coordinate as an endpoint; The sampling module is used to sample each of the virtual rays to obtain the position information of each spatial sampling point used when rendering the target object.

12. The device according to claim 10 or 11, characterized in that The first generating module comprises: A first determination submodule is used to determine, for each of the images to be used, a mapping position of the spatial sampling point in the image to be used, and determine feature information at the mapping position from a feature map of the image to be used as a sampling feature corresponding to the spatial position point; A fusion submodule, used to perform feature fusion processing on the sampling features to obtain fusion features corresponding to the spatial sampling points; The input submodule is used to input the position information of the spatial sampling point and the fusion features corresponding to the spatial sampling point into a pre-trained neural rendering network to generate color information of the spatial sampling point.

13. The device according to claim 12, characterized in that The first determining submodule is specifically used for: For each image to be used, based on camera calibration information of a shooting device that shoots the image to be used and the position information of the spatial sampling point, a mapping position of the spatial sampling point in the image to be used is determined.

14. The device according to claim 12, characterized in that The fusion submodule is specifically used for: The pre-trained self-attention mechanism Transformer layer is used to perform feature fusion processing on each of the sampling features to obtain the fusion features corresponding to the spatial sampling points.

15. The device according to claim 14, characterized in that The neural rendering network and the self-attention mechanism Transformer layer are obtained by joint training based on the position information of the sample sampling points, the feature maps of each of the sample images, and the true value of the image to be predicted.

16. The device according to claim 15, characterized in that The joint training method of the neural rendering network and the self-attention mechanism Transformer layer includes: Acquire each of the sample images and the image to be predicted; Determining, based on the viewing angle information corresponding to the image to be predicted, the position information of the sample sampling point used when rendering the sample object; For each of the sample images, determining a mapping position of the sample sampling point in the sample image, and determining feature information at the mapping position from an image feature map of the sample image as a sampling feature corresponding to the sample sampling point; Using the self-attention mechanism Transformer layer in training, the sampling features of each of the sample sampling points are subjected to feature fusion processing to obtain the fusion features corresponding to the sample sampling points; Inputting the position information of the sample sampling point and the fusion features corresponding to the sample sampling point into the neural rendering network in training, and outputting the predicted color information of the sample sampling point; Based on the predicted color information of the sample sampling point, an image of the sample object when observed from a specified viewing angle is generated as an output image; wherein the specified viewing angle includes an observation position and an observation angle corresponding to the image to be predicted; Calculating the model loss using the output image and the difference between the image to be predicted; When it is determined by using the model loss that the neural rendering network has not converged, adjusting model parameters of the neural rendering network and the self-attention mechanism Transformer layer until the neural rendering network and the self-attention mechanism Transformer layer converge.

17. The device according to claim 12, characterized in that The first determining submodule includes: a linear interpolation unit, configured to, if the feature map of the image to be used is different from the size of the image to be used, perform linear interpolation processing on the feature map of the image to be used based on the size of the image to be used, so as to obtain a target feature map with the same size as the image to be used; The extraction unit is used to extract the feature information at the mapping position in the target feature map to obtain the sampling feature corresponding to the spatial sampling point.

18. The device according to claim 12, characterized in that The color information includes a color value and a density value; the density value is used to represent the weight of the spatial sampling point when the pixel value is generated by using the spatial sampling point; The second generating module comprises: An adding submodule, configured to perform weighted addition of the color values ​​of the spatial sampling points on each of the virtual rays according to the density value, so as to obtain the pixel value of the pixel point in the image to be rendered corresponding to the virtual ray; The second generating submodule is used to generate the rendered image based on the pixel value of each pixel in the image to be rendered.

19. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 9 when executing a program stored in a memory.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 9 are implemented.

Images