Image generation method, model training method, equipment and storage medium

Abstract

The embodiment of the invention provides an image generation method, a model training method, equipment and a storage medium. The image generation method comprises the following steps: obtaining a plurality of view angle images of a target object under different view angles, and a plurality of pieces of pose information and a plurality of depth images which are in one-to-one correspondence with the plurality of view angle images; based on the multiple view angle images and the multiple depth images, multiple mask images in one-to-one correspondence with the multiple view angle images are determined, and a target area contained in each mask image is used for indicating an area where a target object in the corresponding view angle image is located; based on the plurality of view angle images, the plurality of depth images, the plurality of mask images and the plurality of pieces of pose information, training an initial image generation model to obtain a trained image generation model, the initial image generation model being a neural radiation field model; and according to the trained image generation model, generating a target view angle image different from the view angles of the plurality of view angle images.

Description

Technical Field

[0001] This disclosure relates to, but is not limited to, the field of artificial intelligence technology, and in particular to an image generation method, a model training method, an apparatus, and a storage medium. Background Technology

[0002] In various applications such as autonomous driving, gaming, virtual reality, and augmented reality, it is often necessary to render images from new perspectives within specific scenes. However, some technologies consume significant computational resources, are inefficient, and produce poor rendering results during the generation of these new perspective images. Summary of the Invention

[0003] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0004] In a first aspect, embodiments of this disclosure provide an image generation method, comprising: obtaining multiple viewpoint images, multiple pose information corresponding one-to-one with the multiple viewpoint images, and multiple depth images corresponding one-to-one with the multiple viewpoint images, wherein the multiple viewpoint images are images of a target object under different viewpoints; determining multiple mask images corresponding one-to-one with the multiple viewpoint images and the multiple depth images based on the multiple viewpoint images and the multiple depth images, wherein each mask image contains a target region used to indicate the region where the target object is located in the corresponding viewpoint image; training an initial image generation model based on the multiple viewpoint images, the multiple depth images, the multiple mask images, and the multiple pose information to obtain a trained image generation model, wherein the initial image generation model is a neural radiation field model; and generating a target viewpoint image according to the trained image generation model, wherein the target viewpoint image has a different viewpoint than the multiple viewpoint images.

[0005] In some exemplary embodiments, determining a plurality of mask images corresponding one-to-one with the plurality of viewpoint images and the plurality of depth images includes: inputting the plurality of viewpoint images into a first feature extraction network to obtain a plurality of first feature maps corresponding one-to-one with the plurality of viewpoint images; inputting the plurality of depth images into a second feature extraction network to obtain a plurality of second feature maps corresponding one-to-one with the plurality of depth images; inputting the plurality of first feature maps and the plurality of second feature maps into a feature fusion network to obtain a plurality of fused feature maps corresponding one-to-one with the plurality of viewpoint images; inputting the plurality of fused feature maps into a prediction network to obtain a plurality of initial mask images corresponding one-to-one with the plurality of viewpoint images; inputting the plurality of initial mask images into an optimization network to obtain a plurality of optimized mask images corresponding one-to-one with the plurality of viewpoint images; and determining the plurality of mask images based on the plurality of optimized mask images.

[0006] In some exemplary embodiments, the first feature extraction network includes: a first encoding / decoding network, the first encoding / decoding network including: a first encoder and a first decoder, the input of the first encoder being the plurality of viewpoint images, the output of the first encoder being the input of the first decoder, and the output of the first decoder being the plurality of first feature maps; the second feature extraction network includes: a second encoding / decoding network, the second encoding / decoding network including: a second encoder and a second decoder, the input of the second encoder being the plurality of depth images, the output of the second encoder being the input of the second decoder, and the output of the second decoder being the plurality of second feature maps.

[0007] In some exemplary embodiments, determining the plurality of mask images based on the plurality of optimized mask images includes: performing binarization processing on the plurality of optimized mask images to obtain a plurality of binary images; and determining the plurality of binary images as the plurality of mask images.

[0008] In some exemplary embodiments, the optimized network includes a residual refinement module.

[0009] In some exemplary embodiments, training an initial image generation model based on the multiple viewpoint images, the multiple depth images, the multiple mask images, and the multiple pose information to obtain a trained image generation model includes: determining the sampling information of the target object in the multiple viewpoint images in the world coordinate system according to the multiple viewpoint images, the multiple mask images, and the multiple pose information, wherein the sampling information of the target object in the world coordinate system in each viewpoint image includes: the position information and viewpoint information of multiple sampling points on each ray corresponding to each pixel point in the region where the target object is located in the world coordinate system, wherein each ray starts from the camera center and passes through the camera center and the pixel points in the region where the target object is located; and performing position encoding on the sampling information of the target object in the world coordinate system in the multiple viewpoint images to obtain the position-encoded sampling information of the target object in the multiple viewpoint images. The location-encoded sampling information includes: location-encoded position information and location-encoded viewpoint information; the location-encoded sampling information of the target object in the multiple viewpoint images is input into the neural network of the initial image generation model to obtain the voxel information of each ray corresponding to the target object in the multiple viewpoint images, the voxel information of each ray including: the volume density and color corresponding to each of the multiple sampling points on each ray; the voxel information of each ray corresponding to the target object in the multiple viewpoint images is input into the rendering network of the initial image generation model to obtain the predicted rendering image corresponding to each viewpoint image; based on the reconstruction loss, the parameters of the neural network in the initial image generation model are adjusted until the trained image generation model is obtained, wherein the reconstruction loss is calculated using a preset loss function through each viewpoint image, the depth image corresponding to each viewpoint image, and the predicted rendering image corresponding to each viewpoint image.

[0010] In some exemplary embodiments, determining the sampling information of the target object in the multiple view images in the world coordinate system based on the multiple view images, the multiple mask images, and the multiple pose information includes: obtaining the coordinates of the pixels in the region where the target object is located in the multiple view images based on the target region contained in the multiple view images and the multiple mask images; calculating the coordinates of the pixels in the region where the target object is located in the multiple view images in the camera coordinate system based on the coordinates of the pixels in the region where the target object is located in the multiple view images and the intrinsic parameters in the multiple pose information corresponding to the multiple view images; Based on the coordinates of the pixels in the region where the target object is located in the camera coordinate system in multiple viewpoint images, the sampling information of the target object in the camera coordinate system in the multiple viewpoint images is determined. The sampling information of the target object in the camera coordinate system in each viewpoint image includes: the position information and viewpoint information of multiple sampling points on each ray corresponding to each pixel in the region where the target object is located in the camera coordinate system. Based on the extrinsic parameters in the multiple pose information corresponding to the multiple viewpoint images, and the sampling information of the target object in the camera coordinate system in the multiple viewpoint images, the sampling information of the target object in the world coordinate system in the multiple viewpoint images is determined.

[0011] In some exemplary embodiments, adjusting the parameters of the neural network in the initial image generation model based on reconstruction loss until the trained image generation model is obtained includes: determining a color loss based on the difference between each viewpoint image and the corresponding predicted rendered image; determining a depth loss based on the depth image corresponding to each viewpoint image and each ray of the target object in each viewpoint image; determining a reconstruction loss based on the color loss and the depth loss; and adjusting the parameters of the neural network in the initial image generation model based on the reconstruction loss until a preset training termination condition is met to obtain the trained image generation model, wherein the preset training termination condition includes: the reconstruction loss being less than a preset threshold or the number of adjustments reaching a preset number.

[0012] In some exemplary embodiments, determining the color loss based on the difference between each viewpoint image and the corresponding predicted rendered image includes: determining the color loss based on the difference between each viewpoint image and the corresponding predicted rendered image according to the following formula:

[0013]

[0014] in, To render the color information of an image, C g.t. For the color information of the view image, L colorIndicates color loss, ||.|| 2 This represents the L2 norm.

[0015] In some exemplary embodiments, determining the depth loss based on the depth image corresponding to each viewpoint image and each ray of the target object in each viewpoint image includes: determining the depth loss based on the termination distance of each ray of the target object in each viewpoint image and the depth image corresponding to each viewpoint image, according to the following formula:

[0016]

[0017] Where h(t) represents the termination distance of the ray. KL[.] represents the depth information of the depth image, and KL divergence represents L. depth This indicates a deep loss.

[0018] Secondly, embodiments of this disclosure provide a model training method, comprising: obtaining multiple viewpoint images, multiple pose information corresponding one-to-one with the multiple viewpoint images, and multiple depth images corresponding one-to-one with the multiple viewpoint images, wherein the multiple viewpoint images are images of a target object under different viewpoints; determining multiple mask images corresponding one-to-one with the multiple viewpoint images and the multiple depth images based on the multiple viewpoint images and the multiple depth images, wherein each mask image contains a target region used to indicate the region where the target object is located in the corresponding viewpoint image; training an initial image generation model based on the multiple viewpoint images, the multiple depth images, the multiple mask images, and the multiple pose information to obtain a trained image generation model, wherein the initial image generation model is a neural radiation field model, and the trained image generation model is a three-dimensional model of the target object.

[0019] Thirdly, embodiments of this disclosure provide a computer device, including: a processor and a memory storing a computer program executable on the processor, wherein the processor is configured to perform the steps of the image generation method described in any of the exemplary embodiments above, or the steps of the model training method described in the exemplary embodiments above.

[0020] Fourthly, embodiments of this disclosure provide a non-transient computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to perform the steps of the image generation method described in any of the exemplary embodiments above, or the steps of the model training method described in the exemplary embodiments above.

[0021] The image generation method, model training method, device, and storage medium provided in this disclosure train an image generation model by using multiple viewpoint images of a target object from different perspectives, as well as multiple pose information, multiple depth images, and multiple mask images that correspond one-to-one with the multiple viewpoint images. This can reduce computational resource consumption, save computation time, and improve rendering efficiency. Thus, when generating a target viewpoint image with a different perspective from the multiple viewpoint images using the trained image generation model, the synthesis efficiency of the new viewpoint image can be improved, and the rendering effect can also be enhanced.

[0022] Other features and advantages of this disclosure will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the disclosure. Other advantages of this disclosure may be realized and obtained by means of the methods described in the description and the accompanying drawings.

[0023] After reading and understanding the accompanying diagrams and detailed descriptions, the other aspects can be understood. Attached Figure Description

[0024] The accompanying drawings are provided to illustrate the technical solutions of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure. The shapes and sizes of the components in the drawings do not reflect actual proportions and are only intended to illustrate the content of this disclosure.

[0025] Figure 1 This is a schematic diagram of the system architecture involved in the image generation method in an exemplary embodiment of this disclosure;

[0026] Figure 2 This is a flowchart illustrating the image generation method in an exemplary embodiment of this disclosure;

[0027] Figure 3A This is a diagram illustrating the process of photographing the target object.

[0028] Figure 3B A visualized pose diagram of the target object for photographing;

[0029] Figure 4 This is a schematic diagram illustrating the determination of a mask image in an exemplary embodiment of this disclosure;

[0030] Figure 5 This is a schematic diagram of the model training process in an exemplary embodiment of this disclosure;

[0031] Figure 6 This is a flowchart illustrating the model training method in an exemplary embodiment of this disclosure;

[0032] Figure 7A This is a schematic diagram of the rendering results from one perspective in some technologies;

[0033] Figure 7B This is a schematic diagram of the rendering results from another perspective in some technologies;

[0034] Figure 7C This is a schematic diagram of the rendering results from another perspective in some technologies;

[0035] Figure 8A This is a schematic diagram of the rendering result from one perspective in an exemplary embodiment of this disclosure;

[0036] Figure 8B This is a schematic diagram of the rendering result from another perspective in an exemplary embodiment of this disclosure;

[0037] Figure 8C This is a schematic diagram of the rendering result from another perspective in an exemplary embodiment of this disclosure.

[0038] Figure 9 This is a schematic diagram of the structure of a computer device in an exemplary embodiment of this disclosure. Detailed Implementation

[0039] This document describes several embodiments, but these descriptions are exemplary and not limiting. Many more embodiments and implementations are possible within the scope of the embodiments described herein. Although many possible combinations of features are shown in the accompanying drawings and discussed in exemplary embodiments, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with, or substitute for, any feature or element of any other embodiment.

[0040] In describing representative embodiments, the specification may have presented methods and / or processes as a specific sequence of steps. However, the method or process should not be limited to the specific order of steps described herein, to the extent that it does not depend on such a specific order. Other sequences of steps are possible, as will be understood by those skilled in the art. Therefore, the specific order of steps set forth in the specification should not be construed as a limitation of the claims. Furthermore, the claims relating to the method and / or process should not be limited to the steps performed in the order written, and those skilled in the art will readily understand that these orders may be varied and still remain within the spirit and scope of the embodiments disclosed herein.

[0041] Unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure should have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar words used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Words such as "comprising" or "including" mean that the element or object preceding the word encompasses the element or object listed following the word and its equivalents, without excluding other elements or objects. Words such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Words such as "exemplary," "for example," or "e.g.," are used to indicate exemplification, illustration, or explanation. Any embodiment or design described as "exemplary," "for example," or "e.g." in the embodiments of this invention should not be construed as being more preferred or advantageous than other embodiments or designs.

[0042] It should be understood that the term "and / or" used in the embodiments of this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. In addition, unless otherwise stated, the character " / " generally indicates that the objects before and after it are in an "or" relationship. For example, A / B can represent A or B.

[0043] In this embodiment of the disclosure, descriptions such as "when," "in the case of," "if," and "if" all refer to the device taking corresponding actions under certain objective circumstances, and are not time-limited, nor do they require the device to perform a judgment action during implementation, nor do they imply any other limitations. Depending on the context, the words "if" or "if" as used herein can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrases "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0044] For ease of understanding, some of the terms or technical terms used in the embodiments of this disclosure are described below by way of example.

[0045] 1. The Radiance Field describes the propagation behavior of light. In three-dimensional (3D) space, for any ray of light (i.e., its origin and direction), the radiance of that ray at each point in the scene can be calculated. For each point, the Radiance Field can be represented by a color value and a radiance value. The color value refers to the surface color of that point, while the radiance value refers to the brightness or darkness of that point under illumination. By calculating the radiance of light rays throughout the entire 3D scene, high-quality rendered images can be generated.

[0046] 2. Neural Radiance Fields (NeRF) is a deep learning method in computer vision technology used to generate high-quality 3D reconstruction models and renders of 3D scenes (such as objects). NeRF can learn a global representation (i.e., a 3D radiance field) of a 3D object from multiple 2D images from different perspectives, and use this global representation to generate 2D images from new perspectives. NeRF technology has broad application prospects in computer graphics, virtual reality, augmented reality, and other fields. The concept of a radiance field is extended to calculate the color and density of each point in the scene along the direction of any ray in 3D space. Therefore, the radiance field of NeRF can be used to represent the color and density information of the surface of objects in a 3D scene. Using this information, highly realistic 3D models can be presented at any angle and distance.

[0047] In the embodiments of this disclosure, rays refer to the light rays described in radiation fields and nerve radiation fields.

[0048] 3. A Multilayer Perceptron (MLP) is a type of feedforward neural network in deep learning. It consists of multiple fully connected layers of neurons, with each neuron connected to all neurons in the layer above it. It maps input data to a high-dimensional space through nonlinear mappings and combines these mappings into the output through multiple layers of nonlinear transformations.

[0049] 4. Volume rendering, also known as volumetric rendering, is a technique that converts 3D data into visualized 2D images. In 3D data, each pixel contains not only color information but also various physical quantity information, such as volume density.

[0050] 5. Encoder-decoder networks are a type of neural network in deep learning. They can be implemented using Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Bidirectional Recurrent Neural Networks (BiRNNs), Long Short-Term Memory (LSTM) networks, or Gated Recurrent Units (GRUs), etc. Encoding is the process of converting an input sequence into a fixed-length vector; decoding is the process of converting the previously generated fixed-length vector back into an output sequence.

[0051] 6. KL divergence (Kullback-Leibler divergence), also known as relative entropy or information divergence, measures the difference in Shannon entropy between two probability distributions. It describes the information loss of one probability distribution P compared to another probability distribution Q. The mathematical expression for KL divergence is shown below:

[0052]

[0053] Where KL(.) represents the KL divergence, P and Q are two probability distributions, P(i) represents the probability distribution of P on random variable i, and Q(i) represents the probability distribution of Q on random variable i, where random variable i is a discrete random variable.

[0054] 7. The L2 norm, also known as the Euclidean norm, is the square root of the sum of the squares of the elements of a vector. It is frequently used in mathematics and machine learning as a regularization term, distance metric, or error metric. In machine learning, the L2 norm is often used for model regularization and feature normalization. By adding an L2 norm regularization term to the loss function, the complexity of the model can be reduced, and overfitting can be avoided.

[0055] 3D reconstruction refers to establishing a mathematical model of a 3D object suitable for computer representation and processing. It forms the basis for processing, manipulating, and analyzing the object's properties in a computer environment and is a key technology for creating virtual reality representations of the objective world within a computer. In computer vision, 3D reconstruction refers to the process of reconstructing the 3D information of an object from single-view or multi-view images. Novel perspective synthesis technology refers to the technique of generating highly realistic images of a 3D object from any viewpoint using existing images of the object.

[0056] In some technologies, deep learning methods, such as the Neural Radiance Field (NeRF) model, can be used to reconstruct 3D objects and synthesize new perspective images. The NeRF model has attracted considerable attention due to its simple structure. Typically, the NeRF model directly samples RGB images and then uses a multilayer perceptron (MLP)-based neural network to implicitly represent the object in 3D, outputting the volume density and viewpoint-related color at that spatial location. This is then combined with volume rendering techniques to project the color and volume density output by the MLP onto a 2D image, thus achieving the synthesis of a new view. However, this approach results in a large number of ray sampling points, leading to low training and 3D reconstruction efficiency, which in turn affects the efficiency of synthesizing new perspective images. This makes it unsuitable for scenarios with high synthesis efficiency requirements, such as some real-time interactive scenarios, thus limiting the application of 3D reconstruction technology. Furthermore, it can also cause artifacts in the final rendering result, resulting in poor rendering quality.

[0057] To address the issues of low 3D reconstruction efficiency due to excessive ray sampling points in current 3D object reconstruction algorithms, which consequently affects the synthesis efficiency of new perspective images and makes them unsuitable for scenarios with high synthesis efficiency requirements, as well as artifacts in the rendering results, thus impacting rendering quality, this disclosure provides an image generation method. This method trains an initial image generation model using multiple perspective images of the target object from different viewpoints, along with corresponding pose information, depth images, and mask images. The initial image generation model is a neural radiation field model. Based on the trained image generation model, a target perspective image with a different viewpoint from the multiple perspective images is generated. Thus, compared to some existing solutions, this disclosure allows the generation of rays (i.e., beams) by selecting the region of the target object in the perspective image using mask images during 3D reconstruction. This reduces the number of rays the network needs to calculate, significantly reducing the number of ray sampling points during 3D reconstruction, saving computation time, improving 3D reconstruction efficiency, and consequently improving rendering efficiency. It also enhances the synthesis efficiency of new perspective images and expands the application of neural radiation fields. Moreover, since the area where the non-target object is located (i.e. the background area) is skipped when generating light rays, artifacts can be avoided in the rendered scene, thereby improving the rendering effect.

[0058] To facilitate understanding of this disclosure, the system architecture involved in the exemplary embodiments of this disclosure will be described below.

[0059] This exemplary embodiment provides an image processing system. In practical applications, this image processing system can be applied to various application scenarios that require 3D reconstruction and the generation of images from specified perspectives, such as panoramic display, automated modeling, 3D product display, 3D cultural relic display, and new perspective image synthesis.

[0060] Figure 1 This is a schematic diagram of an exemplary system architecture involved in the image generation method of an exemplary embodiment of this disclosure. Figure 1 As shown, the image generation system may include a client device 10 and an image generation apparatus 11. For example, the image generation apparatus 11 may include a 3D reconstruction apparatus 111 and an image rendering apparatus 112. Here, Figure 1 The number of client devices 10, 3D reconstruction devices 111, image rendering devices 112, and trained image generation models shown in this diagram is merely illustrative. Depending on the actual application scenario, any number of client devices, 3D reconstruction devices, image rendering devices, and image generation models can be configured. Specifically, the 3D reconstruction device 111 is configured to train an initial image generation model based on multiple viewpoint images, multiple depth images, multiple mask images, and multiple pose information to obtain a trained image generation model, i.e., a 3D model of the target object. The initial image generation model is a neural radiation field model. The image rendering device 112 is configured to generate an image from a specified viewpoint (i.e., a target viewpoint image) using the trained 3D reconstruction model. The client device 10 is configured to interact with at least one of the 3D reconstruction device 111 and the image rendering device 112 through user operation.

[0061] In some exemplary embodiments, the 3D reconstruction device can be a computer terminal, a server, a server cluster consisting of multiple servers, or a cloud computing service system, or other computer products with strong computing capabilities. The image rendering device can also be a computer terminal, a server, a server cluster consisting of multiple servers, or a cloud computing service system, or other computer products with strong computing capabilities. For example, the 3D reconstruction device and the image rendering device can be configured as independent servers, independent server clusters, or independent cloud computing service systems. Alternatively, the 3D reconstruction device and the image rendering device can be configured on the same server, the same server cluster, or the same cloud computing service system. A cloud computing service system, also known as a cloud server, cloud host, or cloud computing server, is a host product within the cloud computing service system, designed to address the problems of high management difficulty and weak service scalability inherent in traditional physical hosts and Virtual Private Server (VPS) services.

[0062] In some exemplary embodiments, the client device can interact with the server-side 3D reconstruction device via a network. For example, a user can send a 3D reconstruction request to the server-side 3D reconstruction device through the client device. This 3D reconstruction request includes multiple viewpoint images, multiple depth images, multiple mask images, and multiple pose information of the target object to be reconstructed from different perspectives, specified by the user. In response to the user's 3D reconstruction request, the server-side 3D reconstruction device executes the model training method of this disclosure embodiment to obtain a trained image generation model. After obtaining the trained image generation model, the 3D reconstruction device can return the information of the trained image generation model to the client device. This information may include an access interface for the trained image generation model, allowing the user to subsequently utilize this access interface to generate images of the target object from a specified perspective or to perform video synthesis of the target object from a specified perspective.

[0063] In other exemplary embodiments, the client device can interact with the server-side image rendering apparatus via a network. For example, the client device can send a specified viewpoint input by the user to the image rendering apparatus via the network. The image rendering apparatus then invokes a trained image generation model to generate an image of the target object from the specified viewpoint (i.e., the target viewpoint image) and returns the target viewpoint image to the client device.

[0064] In some exemplary embodiments, the client device may include, but is not limited to, smart mobile terminals, smart home devices, wearable devices, smart medical devices, and personal computers (PCs). For example, smart mobile devices may include, but are not limited to, mobile phones, tablets, laptops, personal digital assistants (PDAs), and smart cars. For example, smart home devices may include, but are not limited to, smart home appliances with video playback capabilities, such as smart TVs and smart refrigerators. For example, wearable devices may include, but are not limited to, smartwatches, smart glasses, smart bracelets, virtual reality (VR) devices, augmented reality (AR) devices, and mixed reality devices (i.e., devices that support both virtual reality and augmented reality).

[0065] The system architecture and business scenarios described in the exemplary embodiments of this disclosure are intended to more clearly illustrate the technical solutions of the embodiments of this disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of this disclosure. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of this disclosure are also applicable to similar technical problems.

[0066] This disclosure provides an exemplary embodiment of an image generation method. The embodiments of this disclosure are applicable to scenarios involving 3D reconstruction and the generation of new perspective images based on multiple viewpoint images of a target object from different perspectives. This method can be executed by an image generation apparatus, which can be implemented in hardware and / or software and can be configured in an electronic device with 3D reconstruction capabilities.

[0067] Figure 2 This is a flowchart illustrating the image generation method in an exemplary embodiment of this disclosure, as shown below. Figure 2 As shown, the image generation method may include the following steps:

[0068] Step 21: Obtain multiple viewpoint images, multiple pose information corresponding to the multiple viewpoint images, and multiple depth images corresponding to the multiple viewpoint images, wherein the multiple viewpoint images are images of the target object from different viewpoints.

[0069] The target object can refer to the object to be reconstructed in a 3D scene. In some exemplary embodiments, the target object can be any object, such as shoes, tables, chairs, hats, wardrobes, apples, toys, etc.

[0070] In some exemplary embodiments, a viewpoint image may refer to a 2D image containing a target object captured by an image acquisition device (such as a camera). The viewpoint image may be a pixel-level RGB (Red, Green, Blue) image, that is, an RGB image contains pixel values ​​and grayscale information of multiple pixels, wherein each pixel value has a red value, a green value, and a blue value.

[0071] Step 22: Based on multiple viewpoint images and multiple depth images, determine multiple mask images that correspond one-to-one with the multiple viewpoint images, wherein the target region contained in each mask image is used to indicate the region where the target object is located in the corresponding viewpoint image.

[0072] Each mask image is the same size as its corresponding viewpoint image. Each mask image contains a target region and other regions besides the target region. The target region indicates the location of the target object in the corresponding viewpoint image, while the other regions indicate the background regions surrounding the target object. In this way, during the training of the neural radiation field model, the location of the target object and the background region in the viewpoint image can be quickly determined. During ray generation, the rays can skip the background region and focus directly on the vicinity of the target object's surface, effectively reducing the number of sampling points on the rays involved in the volume rendering process.

[0073] Step 23: Train the initial image generation model based on multiple viewpoint images, multiple depth images, multiple mask images, and multiple pose information to obtain a trained image generation model.

[0074] The initial image generation model is a Neural Radiation Field (NeRF) model. The trained image generation model is a 3D model of the target object. Thus, by using the NeRF model as the initial image generation model for 3D reconstruction, the pixels on the surface of the target object can be modeled more comprehensively and completely.

[0075] Step 24: Generate a target viewpoint image based on the trained image generation model. The target viewpoint image has a different viewpoint than the multiple viewpoint images.

[0076] Thus, the image generation method provided by the exemplary embodiments of this disclosure, since the mask image determined based on the viewpoint image and the depth image incorporates the appearance information and three-dimensional information (such as depth information) of the target object, can reduce the number of light sampling points, reduce the consumption of computing resources, save computing time, and improve rendering efficiency when training the image generation model by using multiple viewpoint images of the target object under different viewpoints, as well as multiple pose information, multiple depth images, and multiple mask images corresponding to the multiple viewpoint images. Therefore, when generating a target viewpoint image with a different viewpoint than the multiple viewpoint images through the trained image generation model, the synthesis efficiency of the new viewpoint image can be improved, and the rendering effect can also be enhanced.

[0077] In some exemplary embodiments, step 21 may include:

[0078] Step 211: Obtain images from multiple viewpoints;

[0079] Step 212: Based on multiple viewpoint images, determine multiple pose information corresponding to each of the multiple viewpoint images, and determine multiple depth images corresponding to each of the multiple viewpoint images.

[0080] In some exemplary embodiments, an image acquisition device (such as a camera) can be used to acquire images of the target object from different viewpoints, thus obtaining multiple viewpoint images. Viewpoint refers to the shooting angle at which the image acquisition device (such as a camera) acquires an image of the target object.

[0081] In some exemplary embodiments, the image acquisition device can be any device with image acquisition function, including but not limited to cameras, video cameras, mobile phones, tablets, and other devices capable of acquiring images of the target object from different perspectives.

[0082] In some exemplary embodiments, the number of image acquisition devices may be one. For example, the image acquisition device may be a single camera, which a user can use to hold and surround the target object and acquire images of the target object from different positions and angles (i.e., different viewpoints) to obtain multiple viewpoint images of the target object. As another example, the image acquisition device may be a single unmanned aerial vehicle (UAV), which a user controls to surround the object and acquire images of the target object from different positions and angles (i.e., different viewpoints) to obtain multiple viewpoint images of the target object.

[0083] In other exemplary embodiments, there may be multiple image acquisition devices. For example, a camera system comprising multiple cameras located at different viewpoints can be used to simultaneously acquire images of the target object to obtain multiple viewpoint images of the target object.

[0084] The pose information corresponding to each viewpoint image refers to the pose information of the image acquisition device (such as a camera) that acquires the image from that viewpoint. Based on the pose information of the image acquisition device (such as a camera), the position information of the pixels of viewpoint images acquired from different viewpoints can be transformed into the same world coordinate system, which facilitates subsequent 3D reconstruction.

[0085] In some exemplary embodiments, after acquiring multiple viewpoint images of the target object from different perspectives using an image acquisition device, the API of the image acquisition device can be called to obtain the pose information of the image acquisition device (such as a camera) corresponding to each viewpoint image. This allows the position information of the image pixels acquired from different perspectives to be transformed into the same coordinate system based on the pose information of the image acquisition device (such as camera intrinsic parameters and camera extrinsic parameters), which facilitates subsequent 3D reconstruction.

[0086] The depth image can be an array of pixels (e.g., a bitmap), where each pixel value represents the depth information at that pixel's location. This allows the depth image to be subsequently applied to a corresponding single-view image, enabling the single-view image to utilize the depth characteristics of the depth image to obtain a mask image of the target object.

[0087] In some exemplary embodiments, step 212 may include performing the following operations for each of the multiple viewpoint images: determining the Euler angles of the viewpoint image based on the viewpoint image, the Euler angles including: Euler angles about the X-axis, Euler angles about the Y-axis, and Euler angles about the Z-axis; determining the rotation matrix corresponding to the viewpoint image based on the Euler angles of the viewpoint image; determining the extrinsic parameters in the pose information of the image acquisition device when acquiring the viewpoint image based on the rotation matrix corresponding to the viewpoint image; obtaining the intrinsic parameters in the pose information of the image acquisition device; and converting the multiple viewpoint images into multiple depth images based on the multiple pose information. The pose information includes: the intrinsic parameters and the extrinsic parameters of the image acquisition device.

[0088] In some exemplary embodiments, step 22 may include:

[0089] Step 221: Input multiple viewpoint images into the first feature extraction network to obtain multiple first feature maps that correspond one-to-one with the multiple viewpoint images;

[0090] Step 222: Input multiple depth images into the second feature extraction network to obtain multiple second feature maps that correspond one-to-one with the multiple depth images;

[0091] Step 223: Input multiple first feature maps corresponding to multiple viewpoint images and multiple second feature maps corresponding to multiple depth images into the feature fusion network to obtain multiple fused feature maps corresponding to multiple viewpoint images.

[0092] Step 224: Input the multiple fused feature maps into the prediction network to obtain multiple initial mask images that correspond one-to-one with the multiple viewpoint images;

[0093] Step 225: Input multiple initial mask images into the optimization network to obtain multiple optimized mask images that correspond one-to-one with multiple viewpoint images;

[0094] Step 226: Based on multiple optimized mask images, determine multiple mask images that correspond one-to-one with multiple viewpoint images.

[0095] For example, the first feature map corresponding to each viewpoint image represents the appearance information (such as color information, texture information, etc.) of the target object in that viewpoint image, and the second feature map corresponding to each depth image represents the three-dimensional information (such as depth information) of the target object in that depth image. In this way, the mask image determined based on the viewpoint image and the depth image fuses the appearance information and three-dimensional information of the target object.

[0096] Thus, the mask image determined based on the viewpoint image and depth image integrates the 3D and appearance information of the target object. During the training of the image generation model using multiple viewpoint images of the target object from different perspectives, along with multiple pose information, depth images, and mask images corresponding to these viewpoint images, rays (i.e., light rays) can be generated by selecting the region where the target object is located in the viewpoint image based on the mask image. This reduces the number of rays the network needs to compute, significantly reducing the number of ray sampling points during 3D reconstruction, saving computation time, and improving 3D reconstruction efficiency. Consequently, when generating target viewpoint images based on the trained image generation model, rendering efficiency is improved, and the synthesis efficiency of new viewpoint images is increased, expanding the application of neural radiation fields. Furthermore, since non-target object regions (i.e., background regions) in the viewpoint image are skipped when selecting rays, artifacts in the rendered scene can be avoided, improving the rendering effect.

[0097] In some exemplary embodiments, the first feature extraction network may include a first encoder-decoder network, which may include a first encoder and a first decoder. The input of the first encoder is multiple viewpoint images, the output of the first encoder is the input of the first decoder, and the output of the first decoder is multiple first feature maps.

[0098] In some exemplary embodiments, the second feature extraction network may include a second encoder-decoder network, which may include a second encoder and a second decoder. The input of the second encoder is a plurality of depth images, the output of the second encoder is the input of the second decoder, and the output of the second decoder is a plurality of second feature maps.

[0099] In some exemplary embodiments, optimizing the network may include a residual refinement module.

[0100] In some exemplary embodiments, step 226 may include: performing binarization processing on multiple optimized mask images to obtain multiple binary images; and determining the multiple binary images as multiple mask images.

[0101] In some exemplary embodiments, step 23 may include:

[0102] Step 231: Based on multiple viewpoint images, multiple mask images, and multiple pose information, determine the sampling information of the target object in the world coordinate system in the multiple viewpoint images. The sampling information of the target object in the world coordinate system in each viewpoint image includes: the position information and viewpoint information of multiple sampling points on each ray corresponding to each pixel in the region where the target object is located in the world coordinate system. Each ray starts from the camera center and passes through the camera center and the pixel in the region where the target object is located.

[0103] Step 232: Perform position encoding on the sampling information of the target object in the world coordinate system in multiple viewpoint images to obtain the position-encoded sampling information of the target object in multiple viewpoint images. The position-encoded sampling information includes: position information after position encoding and view information after position encoding.

[0104] Step 233: Input the sampling information of the position encoding of the target object in multiple view images into the neural network in the initial image generation model to obtain the voxel information of each ray corresponding to the target object in multiple view images. The voxel information of each ray includes: the volume density and color corresponding to each of the multiple sampling points on each ray.

[0105] Step 234: Input the voxel information of each ray corresponding to the target object in multiple viewpoint images into the rendering network in the initial image generation model to obtain the predicted rendering image corresponding to each viewpoint image;

[0106] Step 235: Based on the reconstruction loss, adjust the parameters of the neural network in the initial image generation model until a trained image generation model is obtained. The reconstruction loss is calculated using a preset loss function through each viewpoint image, the depth image corresponding to each viewpoint image, and the predicted rendering image corresponding to each viewpoint image.

[0107] In some exemplary embodiments, step 231 may include: extracting the 2D coordinates (i,j) of the pixels in the region where the target object is located in the multiple view images based on the target region contained in the multiple view images and multiple mask images; calculating the 3D coordinates (i-cx,j-cy,f) of the pixels in the region where the target object is located in the multiple view images in the camera coordinate system based on the 2D coordinates (i,j) of the pixels in the region where the target object is located in the multiple view images and the intrinsic parameters in the multiple pose information corresponding to the multiple view images, for example, the intrinsic parameters include the optical center coordinates (cx,cy) and the focal length f; determining the position of the target object in the multiple view images based on the 3D coordinates (i-cx,j-cy,f) of the pixels in the region where the target object is located in the camera coordinate system. The sampling information in the camera coordinate system includes: the position and view information of multiple sampling points on each ray corresponding to each pixel in the target object's region in the camera coordinate system. Each ray starts from the camera center and passes through the camera center and the corresponding pixel in the target object's region. Based on the extrinsic parameters in the pose information corresponding to multiple view images, the sampling information of the target object in the camera coordinate system in multiple view images is converted into the sampling information of the target object in the world coordinate system in multiple view images. The sampling information of the target object in the world coordinate system includes: the position information p = (x, y, z) and view information of multiple sampling points on each ray corresponding to each pixel in the target object's region in the world coordinate system. For example, extrinsic parameters include a rotation matrix R used to convert coordinates in the camera coordinate system to coordinates in the world coordinate system. This rotation matrix can be a 3x3 matrix. Thus, the image generation method provided in this exemplary embodiment can simultaneously utilize 3D and appearance information when determining a mask image based on the viewpoint image and depth image. During 3D reconstruction, it can select the region where the target object is located in the viewpoint image to generate rays (i.e., light rays) based on the mask image. This reduces the number of light rays the network needs to compute, thereby significantly reducing the number of ray sampling points during 3D reconstruction, greatly saving computation time, and improving 3D reconstruction efficiency. Consequently, when generating target viewpoint images based on a trained image generation model, it can improve rendering efficiency, enhance the synthesis efficiency of new viewpoint images, and expand the application of neural radiation fields. Furthermore, since the regions in the viewpoint image where non-target objects are located (i.e., the background region) are skipped when selecting light rays, it can also avoid rendering scene artifacts and improve rendering effects.

[0108] In some exemplary embodiments, step 235 may include: determining a color loss based on the difference between each viewpoint image and the corresponding predicted rendered image; determining a depth loss based on the depth image corresponding to each viewpoint image and the termination distance of each ray of the target object in each viewpoint image; determining a reconstruction loss based on the color loss and the depth loss; and adjusting the parameters of the neural network in the initial image generation model based on the reconstruction loss until a preset training termination condition is met to obtain a trained image generation model. The preset training termination condition includes: the reconstruction loss being less than a preset threshold or the number of adjustments reaching a preset number. Thus, during the adjustment of the parameters of the neural network in the initial image generation model, since the reconstruction loss includes the color loss determined above based on the viewpoint image and the corresponding predicted rendered image, and the depth loss determined based on the depth image and the rays of the target object in the viewpoint image, the model parameters can be further optimized, thereby improving the model's prediction performance.

[0109] This disclosure provides an exemplary embodiment of an image generation method. The image generation method may include the following steps:

[0110] Step 1: Obtain multiple viewpoint images of the target object.

[0111] Among them, multiple perspective images of the target object refer to images of the target object captured in a three-dimensional scene from multiple different perspectives through an image acquisition device (such as a camera), and the perspectives of the multiple perspective images are different.

[0112] In some exemplary embodiments, taking an image acquisition device including a camera as an example, the same camera can be positioned along a rectangle, circle, ellipse, polygon, etc., surrounding the target object, so as to capture multiple perspective images of the target object from multiple shooting positions. Thus, because the positions and angles of the multiple shooting positions are different, the multiple perspective images captured by the image acquisition device can have different perspectives. For example, by shooting along a rectangle surrounding the target object, the image acquisition device can enhance the richness of the multiple perspective images captured by the same camera.

[0113] For example, you can film n layers, with m shooting positions within each layer capturing the target object. These m shooting positions are arranged coplanarly in a rectangle surrounding the target object; that is, the m shooting positions within each layer are positioned along the same plane. Shooting positions in different layers are not coplanar. n can be a positive integer greater than or equal to 2, and n*m can be a positive integer greater than 90. For instance, consider a filming layer of n = 3, with m = 32 shooting positions within each layer. Figure 3A This demonstrates how the same camera 301 captures images of the target object 302 along a rectangle surrounding the target object 302. Figure 3A The shooting location is indicated by the dashed rectangle in the middle. Figure 3BThis shows a visualized pose diagram of the same camera 301 capturing images of the target object 302. (Example) Figure 3A and 3B As shown, camera 301 surrounds the target object 302 and captures images in three layers. In each rectangular layer, camera 301 can capture images of the target object from 32 different perspectives, thus obtaining 96 perspective images of the target object. These 96 perspective images have different perspectives (different shooting positions and shooting angles of camera 301).

[0114] In other exemplary embodiments, taking an image acquisition device including multiple cameras as an example, different cameras 301 can be arranged coplanarly along a rectangle, circle, ellipse, polygon, etc., surrounding the target object to capture image data of the target object located at a preset distance from each camera, thereby obtaining multiple perspective images of the target object. For example, positioning multiple cameras along a circle surrounding the target object can make the distances between the multiple cameras and the target object approximately equal, ensuring the quality of multiple perspective images captured by different cameras.

[0115] Step 2: After obtaining multiple viewpoint images of the target object, perform pose estimation based on the multiple viewpoint images of the target object to obtain multiple pose information corresponding to the multiple viewpoint images.

[0116] The pose information corresponding to each viewpoint image refers to the pose information of the image acquisition device (such as a camera) when acquiring the image from that viewpoint. The pose information can include the intrinsic and extrinsic parameters of the image acquisition device. The position and orientation of the image acquisition device are determined by its extrinsic parameters, while the projection attributes are determined by its intrinsic parameters. The extrinsic parameters describe the position and orientation of the image acquisition device in the world coordinate system. An extrinsic parameter is a matrix that transforms points in the world coordinate system to the camera coordinate system. The inverse matrix of the extrinsic parameters transforms points in the camera coordinate system to the world coordinate system. The extrinsic parameters can be represented by a rotation matrix R. The intrinsic parameters of the image acquisition device refer to the parameters that affect the pixel coordinates during image acquisition, such as focal length, optical center, and distortion coefficients—parameters related to the characteristics of the image acquisition device itself. These parameters are determined during the manufacturing of the image acquisition device and are usually unchanged during its use. The intrinsic parameters of an image acquisition device can be a matrix, which maps the 3D coordinates in the camera coordinate system (such as the camera coordinate system) of the image acquisition device to the 2D image plane.

[0117] In some exemplary embodiments, determining multiple pose information corresponding one-to-one with the multiple viewpoint images and determining multiple depth images corresponding one-to-one with the multiple viewpoint images may include: performing the following operations for each viewpoint image among the multiple viewpoint images: determining the Euler angles of the image acquisition device (such as a camera) when acquiring the viewpoint image based on the viewpoint image, the Euler angles including: Euler angles around the X-axis, Euler angles around the Y-axis and Euler angles around the Z-axis; determining the rotation matrix corresponding to the viewpoint image based on the Euler angles of the viewpoint image; and determining the pose information of the image acquisition device when acquiring the viewpoint image based on the rotation matrix corresponding to the viewpoint image.

[0118] In some exemplary embodiments, determining the pose information of the image acquisition device when acquiring the viewpoint image based on the rotation matrix corresponding to the viewpoint image may include: determining the extrinsic parameters of the image acquisition device when acquiring the viewpoint image based on the rotation matrix corresponding to the viewpoint image; obtaining the intrinsic parameters of the image acquisition device; and obtaining the pose information of the image acquisition device when acquiring the viewpoint image based on the extrinsic parameters and the intrinsic parameters of the image acquisition device. In this way, the pose information of the image acquisition device when acquiring the viewpoint image can be obtained.

[0119] The pose information of the image acquisition device when acquiring the image from this viewpoint can include: the intrinsic parameters of the image acquisition device (i.e., the camera's intrinsic parameters) and the extrinsic parameters of the image acquisition device when acquiring the image from this viewpoint (i.e., the camera's extrinsic parameters). The camera's intrinsic parameters include: the optical center coordinates (cx, cy) and the focal length f. The camera's extrinsic parameters include: the rotation matrix R used for transforming camera coordinates to world coordinates, which can be a 3x3 matrix. For example, the intrinsic parameters of the image acquisition device can be obtained in advance by consulting the hardware model of the image acquisition device.

[0120] In this context, the Euler angles corresponding to each viewpoint image represent the pose of the image acquisition device (i.e., the camera) in three-dimensional space when acquiring that viewpoint image, that is, the change of the image acquisition device's own coordinate system (i.e., the camera coordinate system) relative to the reference coordinate system (i.e., the world coordinate system) in three-dimensional space.

[0121] In some exemplary embodiments, when obtaining Euler angles, the rotation range of the Euler angle Pitch(α) about the X-axis is defined as [-pi,pi], the rotation range of the Euler angle Yaw(β) about the Y-axis is defined as [-pi / 2,pi / 2], and the rotation range of the Euler angle Roll(γ) about the Z-axis is defined as [pi,-pi].

[0122] In some exemplary embodiments, the Euler angles can be obtained by calling a function of the image acquisition device that provides Euler angles.

[0123] In some exemplary embodiments, the Euler angles corresponding to each viewpoint image are converted into a rotation matrix corresponding to each viewpoint image according to the following formula, that is, the rotation matrix of the image acquisition device when each viewpoint image is captured;

[0124]

[0125] R = R z (γ)*R y (β)*R x (α)

[0126] Where R represents the rotation matrix corresponding to each viewpoint image, R x (α) represents the Euler angle about the X-axis, R y (β) represents the Euler angle about the Y-axis, R z (γ) represents the Euler angle about the Z-axis.

[0127] In some exemplary embodiments, the extrinsic parameters of the image acquisition device can be represented by a rotation matrix R. Therefore, after obtaining the rotation matrix of the image acquisition device when each viewpoint image is captured, the extrinsic parameters of the image acquisition device when each viewpoint image is captured are obtained, so that the coordinates in the camera coordinate system can be converted to the coordinates in the world coordinate system based on the rotation matrix.

[0128] The extrinsic parameters of the image acquisition device are set to transform the coordinates of points in the camera coordinate system of the image acquisition device to the coordinates of points in the world coordinate system.

[0129] Currently, some techniques employ SLAM algorithms or Structure from Motion (SfM) techniques (such as the traditional COLMAP algorithm) for camera pose estimation. The COLMAP algorithm calculates the camera pose through feature detection and matching of images. However, its realism and accuracy are poor, especially when the number of image samples is small. Therefore, compared to some NeRF techniques that use the traditional COLMAP algorithm to obtain pose information, the scheme disclosed in this paper, which obtains extrinsic parameters of pose information using Euler angles, is more accurate. Furthermore, some techniques obtain pose information from video sequences. However, when extracting images from video sequences, image blurring can occur due to camera shake during shooting. Therefore, compared to some techniques that obtain pose information from video sequences, the scheme disclosed in this paper, which obtains extrinsic parameters of pose information using Euler angles, is more accurate.

[0130] In some exemplary embodiments, after obtaining multiple viewpoint images of the target object, the application programming interface (API) of the camera in the image acquisition device can be used to obtain multiple pose information corresponding to the multiple viewpoint images and multiple depth images corresponding to the multiple viewpoint images.

[0131] In some exemplary embodiments, by calling the first API of the image acquisition device, the corresponding Euler angles of the target object are obtained based on multiple view images; based on the Euler angles corresponding to the multiple view images, rotation matrices corresponding to the multiple view images are obtained; and based on the rotation matrices corresponding to the multiple view images, multiple pose information is determined. For example, the first API of the image acquisition device may refer to a function that can provide Euler angles, which is usually provided by the manufacturer of the image acquisition device.

[0132] Step 3: After obtaining multiple viewpoint images of the target object, based on these images and the corresponding pose information, obtain multiple depth images. Since the depth information of the target object can be obtained from these depth images, during model training, the approximate spatial distribution of the target object can be determined, reducing the generation of invalid rays.

[0133] In some exemplary embodiments, by calling a second API of the image acquisition device, multiple depth images corresponding one-to-one with the multiple viewpoint images are obtained based on multiple viewpoint images of the target object and multiple pose information. For example, the second API in the image acquisition device is configured to calculate the depth image corresponding to each viewpoint image based on the multiple viewpoint images and the intrinsic and extrinsic parameters of the image acquisition device, according to a preset pixel conversion formula between depth images and color images. For example, the second API of the image acquisition device may refer to a function that can provide depth images, which is usually provided by the manufacturer of the image acquisition device.

[0134] Step 4: Based on multiple viewpoint images and multiple depth images that correspond one-to-one with the multiple viewpoint images, determine multiple mask images that correspond one-to-one with the multiple viewpoint images.

[0135] The viewpoint image is an image containing the target object, and the mask image can be an image of the same size as the viewpoint image. The mask image can include: tiles for indicating the area where the target object is located (i.e., the subject) and tiles for indicating the area where the three-dimensional scene (i.e., the background) is located.

[0136] In some exemplary embodiments, multiple viewpoint images and multiple depth images corresponding one-to-one with the multiple viewpoint images are input into a preset image segmentation model to obtain multiple mask images corresponding to the multiple viewpoint images. The preset image segmentation model is configured to perform image segmentation on the input viewpoint images under the guidance of the depth images, determine the position of the target object region contained in the viewpoint image and the position of the background region other than the target object region, and output the mask image based on the determined position of the target object region contained in the viewpoint image and the position of the background region other than the target object region.

[0137] In some exemplary embodiments, such as Figure 4As shown, the preset image segmentation model may include: a first feature extraction network, a second feature extraction network, a fusion network, a prediction network, and an optimization network. Then, based on multiple viewpoint images and multiple depth images corresponding to each viewpoint image, using the preset image segmentation model to determine multiple mask images corresponding to each viewpoint image may include: inputting the multiple viewpoint images into the first feature extraction network to obtain multiple first feature maps corresponding to each viewpoint image; inputting the multiple depth images into the second feature extraction network to obtain multiple second feature maps corresponding to each depth image; inputting the multiple first feature maps and the multiple second feature maps corresponding to each depth image into the feature fusion network to obtain multiple fused feature maps corresponding to each viewpoint image; inputting the multiple fused feature maps into the prediction network to obtain multiple initial mask images corresponding to each viewpoint image; inputting the multiple initial mask images into the optimization network to obtain multiple optimized mask images; and determining multiple mask images corresponding to each viewpoint image based on the multiple optimized mask images. Thus, the first feature extraction network can obtain the feature representation of the viewpoint image (i.e., the first feature map), and the second feature extraction network can obtain the feature representation of the depth image (i.e., the second feature map). The fusion network then fuses the feature representations of the viewpoint image and the depth image to obtain a fused feature representation (the fused feature map). Using this fused feature representation (the fused feature map), the location of the target object region and the background region (excluding the target object region) within the viewpoint image can be predicted relatively accurately, resulting in a high accuracy of the initial mask image predicted by the prediction network. Therefore, during the mask image determination process, the first feature extraction network, the second feature extraction network, and the fusion network can simultaneously learn 3D information (such as depth information) and appearance information (such as color information, texture information, etc.). Consequently, the prediction network can accurately predict the location of the target object region and the background region (excluding the target object region) within the viewpoint image, resulting in a relatively accurate target segmentation result, i.e., multiple initial mask images. Furthermore, by optimizing the coarse initial mask image through the optimization network, a more optimized mask image can be obtained, further improving the accuracy of the mask image.

[0138] In some exemplary embodiments, the first feature extraction network and the second feature extraction network can be implemented using an encoder-decoder (En-De) network. For example, the encoder-decoder (En-De) network can include an encoder and a decoder, wherein the encoder can be implemented using a residual network (ResNet), and the decoder can include a deconvolution layer, with the output of the encoder serving as the input of the decoder.

[0139] In some exemplary embodiments, the first feature extraction network may include a first encoding and decoding network, which may include a first encoder and a first decoder. The input of the first encoder is multiple viewpoint images, the output of the first encoder is the input of the first decoder, and the output of the first decoder is multiple first feature maps that correspond one-to-one with the multiple viewpoint images.

[0140] In some exemplary embodiments, the second feature extraction network may include a second encoder-decoder network, which may include a second encoder and a second decoder. The input of the second encoder is a plurality of depth images, the output of the second encoder is the input of the second decoder, and the output of the second decoder is a plurality of second feature maps corresponding one-to-one with the plurality of depth images.

[0141] In some exemplary embodiments, the fusion network is configured to fuse multiple first feature maps corresponding one-to-one with multiple viewpoint images and multiple second feature maps corresponding one-to-one with multiple depth images to obtain multiple fused feature maps corresponding one-to-one with multiple viewpoint images. For example, the fusion network can be implemented using a U-Net network. The U-Net network is a type of CNN (Convolutional Neural Network).

[0142] In some exemplary embodiments, the prediction network is configured to obtain multiple initial mask images corresponding to the multiple viewpoint images based on multiple fused feature maps that correspond one-to-one with the multiple viewpoint images. For example, the prediction network can be implemented using a U-Net network.

[0143] In some exemplary embodiments, the optimized network includes a residual refinement module (RRM).

[0144] In some exemplary embodiments, determining multiple mask images based on multiple optimized mask images includes: binarizing the multiple optimized mask images to obtain multiple binary images (i.e., multiple binarized mask images); and determining the multiple binary images as multiple mask images. For example, the binary image here, i.e., the binarized mask image, refers to a black and white image obtained by marking the pixels of the target object region in the corresponding optimized mask image as 1 and marking the pixels of the region other than the target object region (i.e., the background region) as 0. The black pixel region in the black and white image corresponds to the region that can be ignored when calculating rays in the subsequent calculation, and the white pixel region in the black and white image (i.e., the region where the target object is located) corresponds to the region that can be retained when calculating rays in the subsequent calculation.

[0145] Step 5: Based on multiple viewpoint images, multiple depth images corresponding to the multiple viewpoint images, multiple mask images corresponding to the multiple viewpoint images, and multiple pose information corresponding to the multiple viewpoint images, train the initial image generation model to obtain the trained image generation model.

[0146] In some exemplary embodiments, such as Figure 5 As shown, step 5 may include:

[0147] Step 501: Based on the target region contained in multiple viewpoint images and multiple mask images, extract the 2D coordinates (i,j) of the pixel points in the region where the target object is located in the multiple viewpoint images;

[0148] Step 502: Based on the 2D coordinates (i,j) of the pixels in the region where the target object is located in the multiple view images and the intrinsic parameters in the multiple pose information corresponding to the multiple view images, calculate the 3D coordinates (i-cx,j-cy,f) of the pixels in the region where the target object is located in the multiple view images in the camera coordinate system. The intrinsic parameters include the optical center coordinates (cx,cy) and the focal length f.

[0149] Step 503: Based on the 3D coordinates (i-cx, j-cy, f) of the pixels in the target object region in the camera coordinate system in multiple viewpoint images, determine the sampling information of the target object in the camera coordinate system in multiple viewpoint images. The sampling information of the target object in the camera coordinate system in each viewpoint image includes: the position information and viewpoint information of multiple sampling points on each ray corresponding to each pixel in the target object region in the camera coordinate system. Each ray starts from the camera center and passes through the camera center and the corresponding pixel in the target object region.

[0150] Step 504: Based on the extrinsic parameters in the pose information corresponding to multiple viewpoint images, convert the sampling information of the target object in the camera coordinate system in the multiple viewpoint images into the sampling information of the target object in the world coordinate system in the multiple viewpoint images. The sampling information of the target object in the world coordinate system in each viewpoint image may include: the position information p = (x, y, z) of multiple sampling points on each ray corresponding to each pixel in the region where the target object is located in the world coordinate system and the viewpoint information d = The extrinsic parameters include the rotation matrix R;

[0151] Step 505: Perform position encoding on the sampling information of the target object in the world coordinate system in multiple viewpoint images to obtain the position-encoded sampling information of the target object in multiple viewpoint images. The position-encoded sampling information may include: position information after position encoding and view information after position encoding.

[0152] In some exemplary embodiments, positional encoding processing can be implemented using positional encoding functions. For example, positional encoding functions can be implemented using sine and cosine functions.

[0153] For example, taking the use of a position encoding function to encode position information as an example, the position encoding function can be represented by the following formula:

[0154] γ(p)=(sin(2 0 πp), cos(2 0 πp), …, sin(2 L-1 πp), cos(2 L-1 πp));

[0155] Where p represents location information, γ(p) represents location information after location encoding, and L-1 represents dimension, which is a positive integer.

[0156] Step 506: Input the sampling information of the target object's position encoded in multiple view images into the neural network in the initial image generation model to obtain the voxel information of each ray corresponding to the target object in multiple view images. The voxel information of each ray may include: the volume density and color of each sampling point on each ray.

[0157] In some exemplary embodiments, the neural network in the initial image generation model can be a multilayer perceptron (MLP) to represent color and volume density in 3D space. Given positional information p = (x, y, z) (i.e., 3D coordinates) and viewpoint information... (i.e., direction), the neural network will output the color c and volume density σ at the position information p and view information d.

[0158] For example, the function of the neural network in the initial image generation model can be represented by the following formula:

[0159] c(p, d), σ(p) = f θ (p, d);

[0160] Among them, f θ σ represents a neural network, θ is the parameter of the neural network, c represents color, and σ represents volume density.

[0161] Step 507: Input the voxel information of each ray corresponding to the target object in multiple viewpoint images into the rendering network in the initial image generation model to obtain the predicted rendering image corresponding to each viewpoint image.

[0162] In some techniques, light ray sampling employs a coarse-to-fine sampling method. First, the network is coarsely sampled using a smaller number of sampling points. Then, based on the density values ​​of the coarsely sampled locations, different numbers of sampling points are generated in the corresponding regions. For high-density locations, more sampling points are used for fine sampling. This disclosure utilizes information from the depth image and mask image when generating light rays (i.e., rays), which reduces the generation of invalid rays, thereby reducing the number of sampling points.

[0163] In some exemplary embodiments, the rendering network is configured to employ volume rendering techniques to obtain a predicted rendered image for each viewpoint image based on voxel information of each ray corresponding to the target object in multiple viewpoint images. Specifically, it calculates a weighted sum of color and volume density along the direction of each pixel ray to generate the final 2D image. For a given ray r(t), the contribution of each point on this ray to the final pixel color is calculated. The pixel color C(r) is calculated as follows:

[0164]

[0165] Where r(t) represents the spatial coordinates of ray r at point t, σ(r(t)) represents the volume density at r(t), c(r(t), d) represents the color at r(t), and T(t) represents the projection function from the ray origin to point r(t). For example, the ray origin can be the center of the image acquisition device (such as the center of the camera or webcam), set as (0,0,0).

[0166] Here, the formula for calculating pixel color C(r) represents the integral of color along a ray, that is, along the ray direction, the color of each point on the ray is weighted according to its volume density and transmission function, and then these colors are accumulated.

[0167] Step 508: Based on the reconstruction loss, adjust the parameters of the neural network in the initial image generation model to obtain a trained image generation model. The reconstruction loss is calculated using a preset loss function based on each viewpoint image, the corresponding depth image, and the predicted rendered image.

[0168] In some exemplary embodiments, step 508 may include:

[0169] Step 581: Determine the color loss based on the difference between each viewpoint image and the corresponding predicted rendered image;

[0170] Step 582: Determine the depth loss based on the depth image corresponding to each viewpoint image and the termination distance of each ray of the target object in each viewpoint image;

[0171] Step 583: Determine the reconstruction loss based on color loss and depth loss;

[0172] Step 584: Based on the reconstruction loss, adjust the parameters of the neural network in the initial image generation model until the preset training termination condition is met to obtain the trained image generation model. The preset training termination condition may include: the reconstruction loss is less than a preset threshold or the number of adjustments reaches a preset number.

[0173] In some exemplary embodiments, the color loss is determined based on the difference between each viewpoint image and the corresponding predicted rendered image, according to the following formula:

[0174]

[0175] in, To render the color information of an image, C g.t. For the color information of the view image, L color represents color loss, and ||.||2 represents the L2 norm.

[0176] In some exemplary embodiments, the depth loss is determined based on the depth image corresponding to each viewpoint image and the termination distance of each ray in each viewpoint image, according to the following formula:

[0177]

[0178] Where h(t) represents the termination distance of the ray. KL[.] represents the depth information of the depth image, and KL divergence represents L. depth This indicates a deep loss.

[0179] Thus, deep supervision can reduce the KL divergence of rendering weights and noise-adding depth, thereby improving the model rendering effect.

[0180] In some exemplary embodiments, the reconstruction loss is determined based on color loss and depth loss, according to the following preset loss function:

[0181] L = aL color +bL depth ;

[0182] Where L is the preset loss function, L color L represents the color loss, 'a' represents the weight value of the color loss, and L represents the weight value of the color loss. depth denoted as depth loss, and b represents the weight value of the depth loss.

[0183] In some exemplary embodiments, the weight value a for color loss can be 0.7, and the weight value b for depth loss can be 0.3.

[0184] Step 6: Generate a target viewpoint image based on the trained image generation model. The target viewpoint image has a different viewpoint than the multiple viewpoint images.

[0185] The descriptions of the above application scenario embodiments are similar to those of the above image generation method embodiments, and have similar beneficial effects. For technical details not disclosed in the application scenario embodiments of this disclosure, those skilled in the art should refer to the descriptions in the method embodiments of this disclosure for understanding, and will not repeat them here.

[0186] This disclosure provides an exemplary embodiment of an image generation method. Figure 6 This is a flowchart illustrating the model training method in an exemplary embodiment of this disclosure, as shown below. Figure 6 As shown, the model training method may include the following steps:

[0187] Step 61: Obtain multiple viewpoint images, multiple pose information corresponding to the multiple viewpoint images, and multiple depth images corresponding to the multiple viewpoint images, wherein the multiple viewpoint images are images of the target object from different viewpoints.

[0188] Step 62: Based on multiple viewpoint images and multiple depth images, determine multiple mask images that correspond one-to-one with the multiple viewpoint images, wherein the target region contained in each mask image is used to indicate the region where the target object is located in the corresponding viewpoint image;

[0189] Step 63: Based on multiple viewpoint images, multiple depth images corresponding to the multiple viewpoint images, multiple mask images corresponding to the multiple viewpoint images, and multiple pose information corresponding to the multiple viewpoint images, train the initial image generation model to obtain a trained image generation model. The initial image generation model can be a neural radiation field model, and the trained image generation model can be a three-dimensional model of the target object.

[0190] The inventors of this disclosure obtained the following results through testing experiments:

[0191] Figure 7A This is a schematic diagram of the rendering results from one perspective in some technologies. Figure 7B This is a schematic diagram illustrating the rendering results from another perspective in some technologies. Figure 7C This is a schematic diagram illustrating the rendering results from another perspective in some technologies. Figures 7A to 7C This is a schematic diagram of the rendering results in some technologies.

[0192] Figure 8A This is a schematic diagram of the rendering result from one perspective in an exemplary embodiment of this disclosure. Figure 8B This is a schematic diagram of the rendering result from another perspective in an exemplary embodiment of this disclosure. Figure 8C This is a schematic diagram of the rendering result from another perspective in an exemplary embodiment of this disclosure. Figures 8A to 8C This is a schematic diagram of the rendering result in an exemplary embodiment of this disclosure.

[0193] Through the Figures 7A to 8C Analysis and comparative rendering results show that the image generation method provided in this exemplary embodiment can simultaneously utilize 3D and appearance information when determining the mask image based on the viewpoint image and depth image. During 3D reconstruction, it can select the region where the target object is located in the viewpoint image to generate rays based on the mask image. This reduces the number of rays the network needs to compute, significantly reducing the number of ray sampling points during 3D reconstruction, saving computation time, and improving 3D reconstruction efficiency. Consequently, when generating target viewpoint images based on the trained image generation model, it improves rendering efficiency and the synthesis efficiency of new viewpoint images, expanding the application of neural radiation fields. Furthermore, by skipping non-target object regions (i.e., background regions) in the viewpoint image when selecting rays, it avoids rendering artifacts and improves rendering effects. Moreover, optimization using depth information during model training allows the trained model to achieve better rendering results and demonstrate superior performance.

[0194] In the various embodiments of this disclosure, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0195] Based on the same inventive concept, this disclosure also provides a computer device, including: a processor and a memory storing a computer program that can run on the processor, wherein the processor is configured to implement the steps of the image generation method as described in any of the above embodiments when executing the program.

[0196] In one exemplary embodiment, Figure 9 This is a schematic diagram of the structure of a computer device in an exemplary embodiment of this disclosure, such as... Figure 9 As shown, the computer device 90 may include: at least one processor 91; and at least one memory 92 and a bus 93 connected to the processor 91; wherein the processor 91 and the memory 92 communicate with each other via the bus 93; the processor 91 is configured to call program instructions in the memory 92 to execute the steps of the image generation method in one or more exemplary embodiments described above. It is understood that... Figure 9 The structure of the computer device 90 shown does not constitute a limitation on the computer device. Actual computer devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0197] In one exemplary embodiment, the processor may be, for example, a Central Processing Unit (CPU), other general-purpose processors, a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, application-specific integrated circuits (ASICs), or other chips with computing functions. The general-purpose processor may be a Microprocessor Unit (MPU), or it may be any conventional processor. This disclosure does not limit the scope of the embodiment.

[0198] In one exemplary embodiment, the memory may include non-permanent memory and / or non-volatile memory in a computer-readable storage medium. For example, the memory may be random access memory (RAM), read-only memory (ROM), or flash memory, or any combination thereof, configured to store programs executed by a processor. The memory may include at least one memory chip. Here, the embodiments of this disclosure do not limit this.

[0199] In one exemplary embodiment, the bus may include, in addition to a data bus, a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, in... Figure 9 In this embodiment, various buses are labeled as bus 93. This disclosure does not limit the scope of the embodiments.

[0200] In implementation, the processing performed by the computer device can be accomplished through integrated logic circuits in the processor's hardware or through software instructions. That is, the method steps of this disclosure can be executed by a hardware processor, or by a combination of hardware and software modules within the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other storage media. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0201] Based on the same inventive concept, this disclosure also provides a non-transient computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as steps of the image generation method in one or more of the above exemplary embodiments.

[0202] In one exemplary embodiment, the computer-readable storage medium described above may include, but is not limited to, ROM / RAM, magnetic disks, or optical disks. This disclosure does not limit the scope of the embodiment.

[0203] The descriptions of the computer device or computer-readable storage medium embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the computer device or computer-readable storage medium embodiments of this disclosure, those skilled in the art should refer to the descriptions of the method embodiments of this disclosure for understanding. Further details will not be repeated here.

[0204] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / modules, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / modules mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0205] While the embodiments disclosed herein are as described above, the above content is merely for the purpose of facilitating understanding of this disclosure and is not intended to limit this disclosure. Any person skilled in the art to which this disclosure pertains may make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed herein, but the scope of patent protection of this disclosure shall still be determined by the scope defined in the appended claims.

Claims

1. An image generation method, characterized in that, include: Multiple viewpoint images, multiple pose information corresponding to the multiple viewpoint images, and multiple depth images corresponding to the multiple viewpoint images are images of the target object from different viewpoints. Based on the multiple viewpoint images and the multiple depth images, multiple mask images are determined that correspond one-to-one with the multiple viewpoint images, wherein the target region contained in each mask image is used to indicate the region where the target object is located in the corresponding viewpoint image; Based on the multiple viewpoint images, the multiple depth images, the multiple mask images, and the multiple pose information, an initial image generation model is trained to obtain a trained image generation model, wherein the initial image generation model is a neural radiation field model; Based on the trained image generation model, a target viewpoint image is generated, wherein the target viewpoint image has a different viewpoint than the plurality of viewpoint images.

2. The method according to claim 1, characterized in that, The step of determining multiple mask images corresponding one-to-one with the multiple viewpoint images and the multiple depth images includes: The multiple viewpoint images are input into a first feature extraction network to obtain multiple first feature maps that correspond one-to-one with the multiple viewpoint images; The multiple depth images are input into a second feature extraction network to obtain multiple second feature maps that correspond one-to-one with the multiple depth images; The plurality of first feature maps and the plurality of second feature maps are input into a feature fusion network to obtain a plurality of fused feature maps that correspond one-to-one with the plurality of viewpoint images; The multiple fused feature maps are input into the prediction network to obtain multiple initial mask images that correspond one-to-one with the multiple viewpoint images; The multiple initial mask images are input into the optimization network to obtain multiple optimized mask images that correspond one-to-one with the multiple viewpoint images; Based on the multiple optimized mask images, the multiple mask images are determined.

3. The method according to claim 2, characterized in that, The first feature extraction network includes a first encoder-decoder network, which includes a first encoder and a first decoder. The input of the first encoder is the plurality of viewpoint images, the output of the first encoder is the input of the first decoder, and the output of the first decoder is the plurality of first feature maps. The second feature extraction network includes a second encoder-decoder network, which includes a second encoder and a second decoder. The input of the second encoder is the plurality of depth images, the output of the second encoder is the input of the second decoder, and the output of the second decoder is the plurality of second feature maps.

4. The method according to claim 2 or 3, characterized in that, The step of determining the plurality of optimized mask images includes: The multiple optimized mask images are binarized to obtain multiple binary images; The plurality of binary images are determined as the plurality of mask images.

5. The method according to claim 2 or 3, characterized in that, The optimized network includes a residual refinement module.

6. The method according to claim 1, characterized in that, The step of training an initial image generation model based on the multiple viewpoint images, the multiple depth images, the multiple mask images, and the multiple pose information to obtain a trained image generation model includes: Based on the multiple viewpoint images, the multiple mask images, and the multiple pose information, the sampling information of the target object in the multiple viewpoint images in the world coordinate system is determined. The sampling information of the target object in the world coordinate system in each viewpoint image includes: the position information and viewpoint information of multiple sampling points on each ray corresponding to each pixel in the region where the target object is located in the world coordinate system. Each ray starts from the camera center and passes through the camera center and the pixels in the region where the target object is located. The sampling information of the target object in the world coordinate system in the multiple viewpoint images is position-encoded to obtain the position-encoded sampling information of the target object in the multiple viewpoint images. The position-encoded sampling information includes: position-encoded position information and position-encoded viewpoint information. The sampling information of the position encoding of the target object in the multiple view images is input into the neural network in the initial image generation model to obtain the voxel information of each ray corresponding to the target object in the multiple view images. The voxel information of each ray includes: the volume density and color corresponding to each of the multiple sampling points on each ray. The voxel information of each ray corresponding to the target object in multiple viewpoint images is input into the rendering network in the initial image generation model to obtain the predicted rendering image corresponding to each viewpoint image. Based on the reconstruction loss, the parameters of the neural network in the initial image generation model are adjusted until the trained image generation model is obtained. The reconstruction loss is calculated using a preset loss function through each viewpoint image, the depth image corresponding to each viewpoint image, and the predicted rendering image corresponding to each viewpoint image.

7. The method according to claim 6, characterized in that, The step of determining the sampling information of the target object in the world coordinate system in the multiple viewpoint images based on the multiple viewpoint images, the multiple mask images, and the multiple pose information includes: Based on the target region contained in the multiple viewpoint images and the multiple mask images, obtain the coordinates of the pixel points in the region where the target object is located in the multiple viewpoint images; Based on the coordinates of the pixels in the region where the target object is located in the multiple viewpoint images, and the intrinsic parameters in the multiple pose information corresponding to the multiple viewpoint images, the coordinates of the pixels in the region where the target object is located in the multiple viewpoint images in the camera coordinate system are calculated. Based on the coordinates of the pixels in the region where the target object is located in the camera coordinate system in multiple viewpoint images, the sampling information of the target object in the camera coordinate system in multiple viewpoint images is determined. The sampling information of the target object in the camera coordinate system in each viewpoint image includes: the position information and viewpoint information of multiple sampling points on each ray corresponding to each pixel in the region where the target object is located in the camera coordinate system. Based on the extrinsic parameters in the pose information corresponding to the multiple viewpoint images, and the sampling information of the target object in the camera coordinate system in the multiple viewpoint images, the sampling information of the target object in the world coordinate system in the multiple viewpoint images is determined.

8. The method according to claim 6 or 7, characterized in that, The step of adjusting the parameters of the neural network in the initial image generation model based on the reconstruction loss until the trained image generation model is obtained includes: The color loss is determined based on the difference between each viewpoint image and the corresponding predicted rendered image; Determine the depth loss based on the depth image corresponding to each viewpoint image and each ray of the target object in each viewpoint image; The reconstruction loss is determined based on the color loss and the depth loss; Based on the reconstruction loss, the parameters of the neural network in the initial image generation model are adjusted until a preset training termination condition is met to obtain the trained image generation model. The preset training termination condition includes: the reconstruction loss is less than a preset threshold or the number of adjustments reaches a preset number.

9. The method according to claim 8, characterized in that, The step of determining the color loss based on the difference between each viewpoint image and the corresponding predicted rendered image includes: Based on the difference between each viewpoint image and the corresponding predicted rendered image, the color loss is determined according to the following formula: in, To render the color information of an image, C g.t. For the color information of the view image, L color Indicates color loss, ||.|| 2 This represents the L2 norm.

10. The method according to claim 8, characterized in that, The process of determining the depth loss based on the depth image corresponding to each viewpoint image and each ray of the target object in each viewpoint image includes: Based on the depth image corresponding to each viewpoint image and the termination distance of each ray of the target object in each viewpoint image, the depth loss is determined according to the following formula: Where h(t) represents the termination distance of the ray. KL[.] represents the depth information of the depth image, and KL divergence represents L. depth This indicates a deep loss.

11. A model training method, characterized in that, include: Multiple viewpoint images, multiple pose information corresponding to the multiple viewpoint images, and multiple depth images corresponding to the multiple viewpoint images are images of the target object from different viewpoints. Based on the multiple viewpoint images and the multiple depth images, multiple mask images are determined that correspond one-to-one with the multiple viewpoint images, wherein the target region contained in each mask image is used to indicate the region where the target object is located in the corresponding viewpoint image; Based on the multiple viewpoint images, the multiple depth images, the multiple mask images, and the multiple pose information, an initial image generation model is trained to obtain a trained image generation model. The initial image generation model is a neural radiation field model, and the trained image generation model is a three-dimensional model of the target object.

12. A computer device, comprising: A processor and a memory storing a computer program executable on the processor, wherein the processor is configured to perform the steps of the image generation method as claimed in any one of claims 1 to 10, or the steps of the model training method as claimed in claim 11.

13. A non-transient computer-readable storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are configured to perform the steps of the image generation method as described in any one of claims 1 to 10, or the steps of the model training method as described in claim 11.

Images