Image generation method, apparatus, device, and medium

Abstract

The application provides an image generation method, device, equipment and medium, wherein the method comprises: acquiring multi-view images and camera parameters corresponding to each view image in the multi-view images; determining point cloud data corresponding to the multi-view images and high-dimensional feature vectors corresponding to each view image according to the multi-view images and the camera parameters corresponding to each view image in the multi-view images; determining three-dimensional scene representation parameters according to the multi-view images, the point cloud data and each high-dimensional feature vector; and rendering the three-dimensional scene representation parameters to obtain a new view image. The application can generate a new view image with high realism and fineness without any supervision of geometric information such as a depth map, thereby improving image generation efficiency and quality.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image generation method, apparatus, device, and medium. Background Technology

[0002] With the continuous development of computer vision technology, new perspective images can be generated based on multi-view images, and the generated new perspective images can be applied to different fields, such as autonomous driving, film and television production, medical imaging, and extended reality (XR).

[0003] In related technologies, generating new perspective images involves inputting multi-view images into a feedforward 3D Gaussian scatter model. This model extracts 3D information about the scene from the multi-view images and generates new perspective images based on this 3D information. The feedforward 3D Gaussian scatter model can be a multi-view splatting (MVSplat) model or a depth splatting (DepthSplat) model, etc.

[0004] However, the aforementioned feedforward 3D Gaussian scatter model requires supervised training using geometric information such as depth maps to enable the model to learn three-dimensional geometric information. However, obtaining high-quality geometric information is costly and difficult. Summary of the Invention

[0005] This application provides an image generation method, apparatus, device, and medium that can generate new perspective images with high realism and detail without the need for supervision of any geometric information such as depth maps, thereby improving image generation efficiency and quality.

[0006] In a first aspect, embodiments of this application provide an image generation method, including:

[0007] Acquire multi-view images and the camera parameters corresponding to each view image in the multi-view images;

[0008] Based on the multi-view images and the camera parameters corresponding to each view image in the multi-view images, determine the point cloud data corresponding to the multi-view images and the high-dimensional feature vector corresponding to each view image.

[0009] The three-dimensional scene representation parameters are determined based on the multi-view images, the point cloud data, and each of the high-dimensional feature vectors.

[0010] The three-dimensional scene representation parameters are rendered to obtain a new perspective image.

[0011] Secondly, embodiments of this application provide an image generation apparatus, comprising:

[0012] An image acquisition module is used to acquire multi-view images and camera parameters corresponding to each view image in the multi-view images;

[0013] The first determining module is used to determine the point cloud data corresponding to the multi-view image and the high-dimensional feature vector corresponding to each view image based on the multi-view image and the camera parameters corresponding to each view image in the multi-view image.

[0014] The second determining module is used to determine the three-dimensional scene representation parameters based on the multi-view images, the point cloud data, and each of the high-dimensional feature vectors;

[0015] The generation module is used to render the 3D scene representation parameters to obtain a new perspective image.

[0016] Thirdly, embodiments of this application provide an electronic device, including:

[0017] A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the image generation method as described in the foregoing first aspect embodiments and various implementations.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to perform the image generation method as described in the foregoing first aspect embodiments and various implementations.

[0019] Fifthly, embodiments of this application provide a computer program product containing program instructions that, when executed on an electronic device, cause the electronic device to perform the image generation method as described in the first aspect embodiment and its various implementations.

[0020] The technical solution disclosed in this application obtains multi-view images and camera parameters corresponding to each view image in the multi-view images. Based on the multi-view images and camera parameters corresponding to each view image, the point cloud data corresponding to the multi-view images and the high-dimensional feature vector corresponding to each view image are determined. Then, based on the multi-view images, point cloud data and each high-dimensional feature vector, the three-dimensional scene representation parameters are determined. Finally, the three-dimensional scene representation parameters are rendered to obtain a new view image. In this way, a new view image with high realism and detail can be generated without the supervision of any geometric information such as depth maps, thereby improving the efficiency and quality of image generation. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating an image generation method provided in this application embodiment;

[0023] Figure 2 A schematic diagram of a training scenario for a scene coordinate regression network provided in an embodiment of this application;

[0024] Figure 3 A training flowchart for a scene coordinate regression network provided in an embodiment of this application;

[0025] Figure 4 This is a schematic diagram of the structure of a scene coordinate regression network provided in an embodiment of this application;

[0026] Figure 5 A flowchart illustrating the training of a scene coordinate regression network is provided in this application embodiment;

[0027] Figure 6 A schematic diagram of a training scenario for a prediction head network provided in an embodiment of this application;

[0028] Figure 7 A flowchart illustrating the training process of a prediction head network provided in an embodiment of this application;

[0029] Figure 8 This is a schematic diagram illustrating how a new perspective image is generated based on a scene coordinate regression network and a prediction head network, as provided in an embodiment of this application.

[0030] Figure 9 A flowchart illustrating another image generation method provided in an embodiment of this application;

[0031] Figure 10 A flowchart for acquiring point cloud data and high-dimensional feature vectors is provided in an embodiment of this application;

[0032] Figure 11 A schematic diagram illustrating the acquisition of upsampled point cloud data and upsampled high-dimensional feature vectors based on an upsampled network, as provided in this application embodiment;

[0033] Figure 12 This is a schematic diagram of the structure of an upsampling network provided in an embodiment of this application;

[0034] Figure 13A schematic block diagram of an image generation apparatus provided in an embodiment of this application;

[0035] Figure 14 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0037] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0038] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or solution described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0039] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more, that is, at least two. "At least one" means one or more. "Any" means any one or any several.

[0040] As mentioned earlier, the generation of new perspective images in related technologies involves inputting multi-view images into a feedforward 3D Gaussian scatter model. This model extracts 3D scene information from the multi-view images and generates new perspective images based on this 3D information. However, the aforementioned feedforward 3D Gaussian scatter model requires supervised training using geometric information such as depth maps during training to enable the model to learn 3D geometric information. Obtaining high-quality geometric information is costly and difficult.

[0041] To address the aforementioned technical problems, this application provides an image generation method, apparatus, device, and medium to solve the problem that in the related art, feedforward 3D Gaussian scatter model needs to rely on geometric information such as depth maps for supervised training during the training process so that the model can learn three-dimensional geometric information, but obtaining high-quality geometric information is costly and difficult.

[0042] The technical solutions provided by the embodiments of this application will be described in detail below through some examples. The embodiments described below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0043] Figure 1 This is a flowchart illustrating an image generation method provided in an embodiment of this application. In this embodiment, the image generation method can be executed by an image generation device. This image generation device can consist of hardware and / or software and can be integrated into an electronic device. In this application, the electronic device can be, but is not limited to, wearable devices, tablet computers, desktop computers, and mobile devices (such as smartphones). Figure 1 As shown, the image generation method may include the following steps:

[0044] S101, acquire multi-view images and the camera parameters corresponding to each view image in the multi-view images.

[0045] In this application, a multi-view image refers to multiple images of any scene taken from different perspectives. For example, multiple scene images taken from different angles of a real scene. In this application, the multi-view image can be understood as a target multi-view image.

[0046] The aforementioned scene may include one or more objects and their surrounding environment. That is, the scene may include at least one of foreground information and background information, where the foreground information may be an object, and the background information may be the object's surrounding environment. Furthermore, the aforementioned object may be a person, an animal, or other object, etc.

[0047] The camera parameters corresponding to each viewpoint image mentioned above can refer to the camera's intrinsic and extrinsic parameters for that viewpoint image. Furthermore, based on these camera parameters, the shooting angle for each viewpoint image can be obtained.

[0048] In some optional embodiments, a camera or other device with shooting capabilities can be used to capture images of any target scene from different perspectives to obtain multi-view images. Furthermore, while capturing multi-view images, this application can also obtain the camera parameters corresponding to each perspective image captured by the camera or other device.

[0049] S102, based on the multi-view images and the camera parameters corresponding to each view image in the multi-view images, determine the point cloud data corresponding to the multi-view images and the high-dimensional feature vector corresponding to each view image.

[0050] Point cloud data can be understood as scene point cloud, and each point cloud in the scene point cloud can represent a three-dimensional coordinate.

[0051] The aforementioned high-dimensional feature vectors specifically refer to high-dimensional semantic features, which are image features extracted from each viewpoint image that have semantic meaning and are represented in the form of high-dimensional vectors.

[0052] In some alternative embodiments, this application can use a machine learning model that supports the simultaneous generation of point cloud data and high-dimensional feature vectors to process multi-view images and camera parameters corresponding to each view image in the multi-view images, so as to obtain point cloud data corresponding to the multi-view images and high-dimensional feature vectors corresponding to each view image.

[0053] The aforementioned machine learning model that supports the simultaneous generation of point cloud data and high-dimensional feature vectors can be selected as a scene coordinate regression network, etc. Optionally, when the machine learning model is a scene coordinate regression network, the scene coordinate regression network can adopt techniques for accelerating and optimizing coordinate representation, such as Accelerated Coordinate Encoding (ACE) technology, so as to leverage the fast and accurate point cloud generation capability of ACE technology to provide accurate and reliable geometric prior support for the scene coordinate regression network.

[0054] In some alternative embodiments, this application may also use the Structure-from-Motion (SfM) algorithm to process the multi-view images and the camera parameters corresponding to each view image in the multi-view images to obtain the point cloud data corresponding to the multi-view images and the high-dimensional feature vector corresponding to each view image.

[0055] S103 determines the 3D scene representation parameters based on multi-view images, point cloud data, and each high-dimensional feature vector.

[0056] The aforementioned 3D scene representation parameters refer to parameters generated based on known viewpoint images and used to generate new viewpoint images. Specifically, the 3D scene representation parameters refer to the 3D scene representation parameters corresponding to each viewpoint image among multiple viewpoint images. That is, in this application, the 3D scene representation parameters are a set of parameters, and this set consists of the 3D scene representation parameters corresponding to each viewpoint image among multiple viewpoint images.

[0057] Furthermore, the aforementioned 3D scene representation parameters may include: coordinate information, opacity information, covariance matrix, and spherical harmonic function coefficients. Specifically, coordinate information refers to position information.

[0058] In some embodiments, considering the 3D Gaussian Splatting (3DGS) algorithm, which has real-time rendering speed and high-fidelity visual rendering effect, this application can process multi-view images, point cloud data and each high-dimensional feature vector based on the 3DGS algorithm to obtain three-dimensional scene representation parameters.

[0059] In some embodiments, this application may also use a machine learning model that supports the generation of 3D scene representation parameters to process multi-view images, point cloud data and high-dimensional feature vectors to obtain 3D scene representation parameters.

[0060] The aforementioned machine learning model that supports generating 3D scene representation parameters can be, for example, a prediction head network. It should be understood that the prediction head network is used to perform predictions for a specific task. Optionally, when the machine learning model is a prediction head network, this prediction head network specifically refers to a Gaussian regression head network, used to perform regression tasks with Gaussian distribution parameters.

[0061] S104 renders the 3D scene representation parameters to obtain a new perspective image.

[0062] Among them, a new perspective image refers to a new image that has a different perspective from a multi-perspective image.

[0063] After obtaining the 3D scene representation parameters, this application can render the 3D scene representation parameters from any viewpoint and use the rendering result as a new viewpoint image. Note that the viewpoint corresponding to the arbitrary viewpoint is different from that corresponding to the multi-viewpoint image.

[0064] The technical solution disclosed in this application obtains multi-view images and camera parameters corresponding to each view image in the multi-view images. Based on the multi-view images and camera parameters corresponding to each view image, the point cloud data corresponding to the multi-view images and the high-dimensional feature vector corresponding to each view image are determined. Then, based on the multi-view images, point cloud data and each high-dimensional feature vector, the three-dimensional scene representation parameters are determined. Finally, the three-dimensional scene representation parameters are rendered to obtain a new view image. In this way, a new view image with high realism and detail can be generated without the supervision of any geometric information such as depth maps, thereby improving the efficiency and quality of image generation.

[0065] In some alternative embodiments, considering that the point cloud data and high-dimensional feature vectors are determined in this application, they can be achieved through machine learning models such as scene coordinate regression networks; and the determination of the 3D scene representation parameters can also be achieved through corresponding machine learning models such as prediction head networks.

[0066] Therefore, this application may optionally provide a detailed description of the training process of the machine learning model involved in the embodiments of this application, as well as a detailed description of the specific process of using the trained machine learning model to determine point cloud data, high-dimensional feature vectors, and 3D scene representation parameters.

[0067] In other words, the technical solution provided in this application can include two parts: a first part is a model training part and a second part is a model application part. Furthermore, the model involved in this application can include a scene coordinate regression network and a prediction head network.

[0068] First, combine Figure 2 and Figure 3 The training process of the scene coordinate regression network involved in the embodiments of this application will be described. Figure 2 A schematic diagram of a training scenario for a scene coordinate regression network provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the training process of a scene coordinate regression network provided in an embodiment of this application. Figure 3 As shown, the method may include the following steps:

[0069] S201, Obtain the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image.

[0070] The first training multi-view image refers to the training data used to train the scene coordinate regression network.

[0071] Optionally, this application obtains the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image, which can be achieved in various ways, specifically at least one of the following:

[0072] The first method involves using a camera or other device with shooting capabilities to capture images of any scene from different angles, thereby obtaining multi-view images, which are then used as the first training multi-view images.

[0073] Furthermore, while capturing images from multiple perspectives, it can also acquire the camera parameters corresponding to each perspective image captured by the camera or other devices.

[0074] The second approach involves obtaining images of any scene from different perspectives from an open-source image database to create a first training multi-view image, and then obtaining the camera parameters corresponding to each perspective image in the first training multi-view image.

[0075] In addition to the two methods mentioned above, this application may also use other methods to obtain the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image, and this application does not impose any restrictions on this.

[0076] S202, based on the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image, train the scene coordinate regression network to be trained to obtain the scene coordinate regression network.

[0077] In this application, the scene coordinate regression network to be trained refers to the scene coordinate regression network in the network model whose network parameters have not been adjusted. That is, the scene coordinate regression network in its initial state.

[0078] It should be understood that the aforementioned scene coordinate regression network is a deep learning model that outputs three-dimensional scene coordinates based on the input image, used to map two-dimensional pixel coordinates in an image to three-dimensional scene coordinates.

[0079] Furthermore, considering that ACE technology can fundamentally revolutionize the technical paradigm that requires supervised training based on geometric priors, enabling the model to simultaneously optimize coordinate predictions for multiple scene views in a single iteration under the supervision of the reprojection loss function, and directly generate dense and accurate point cloud data through end-to-end learning. Crucially, ACE technology can rapidly complete new scene mapping, achieving a significant efficiency breakthrough compared to the lengthy mapping time of traditional optimal scene coordinate regression systems. Therefore, the scene coordinate regression network in this application can employ techniques for accelerating and optimizing coordinate representation, such as ACE technology, to leverage ACE's rapid and accurate point cloud generation capabilities, providing precise and reliable geometric prior support for the scene coordinate regression network.

[0080] In some optional embodiments, training the scene coordinate regression network to obtain the scene coordinate regression network may include the following steps:

[0081] The first step is to input the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image into the scene coordinate regression network to be trained, so that the scene coordinate regression network to be trained processes the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image to obtain the point cloud data corresponding to the first training multi-view image output by the scene coordinate regression network to be trained.

[0082] The second step is to determine the first loss value for the scene coordinate regression network to be trained based on the point cloud data corresponding to the first training multi-view images and the pixel coordinates in each view image.

[0083] The third step is to train the scene coordinate regression network based on the first loss value to obtain the scene coordinate regression network.

[0084] Optionally, the first loss value corresponding to the scene coordinate regression network to be trained can be determined by the following formula (1):

[0085]

[0086] Where Loss1 is the first loss value corresponding to the scene coordinate regression network to be trained; argmin is the value of the independent variable that minimizes the loss function; w is the learnable parameter of the scene coordinate regression network to be trained; i is the i-th image in the first training multi-view images; N is the N-th image in the first training multi-view images, where N is a positive integer greater than or equal to 2; X j Let I be the coordinates of the j-th pixel in the i-th image; i For the i-th image; l π Y is the reprojection loss function; j T represents the predicted 3D coordinates of the j-th pixel in the i-th image; i Let X be the camera parameters corresponding to the i-th image; where X j With Y j Related.

[0087] In some optional embodiments, after determining the first loss value corresponding to the scene coordinate regression network to be trained, this application can determine whether the first loss value is less than a first threshold. If the first loss value is greater than or equal to the first threshold, it indicates that the prediction effect of the scene coordinate regression network to be trained is poor. In this case, the scene coordinate regression network to be trained can be trained based on the first loss value to obtain the scene coordinate regression network. If the first loss value is less than the first threshold, it indicates that the prediction effect of the scene coordinate regression network to be trained is good. In this case, the training operation of the scene coordinate regression network to be trained can be ended, and the scene coordinate regression network to be trained can be used as the final scene coordinate regression network.

[0088] The first threshold mentioned above is an adjustable parameter, which can be set according to the training accuracy requirements of the scene coordinate regression network model. There are no restrictions on it here.

[0089] In some alternative embodiments, the scene coordinate regression network to be trained is trained based on the first loss value. Specifically, the network parameters in the scene coordinate regression network to be trained are adjusted, such as adjusting the weights and other parameters.

[0090] This application embodiment obtains a first training multi-view image and the camera parameters corresponding to each view image, and trains the scene coordinate regression network to be trained based on the first training multi-view image and the camera parameters corresponding to each view image to obtain a scene coordinate regression network, so as to realize the rapid acquisition of scene cross-view images by training the scene coordinate regression network. Figure 1 The method provides the required size information and geometric priors without the need for any depth map or other geometric information for supervision. Furthermore, by training the scene coordinate regression network using the aforementioned method, this application significantly shortens the training time and improves the training efficiency of the network model compared to related technologies that involve long-term (hours to days) training of feedforward 3D Gaussian scatter models using large-scale datasets, which results in high training costs.

[0091] In some alternative embodiments, such as Figure 4 As shown, the scene coordinate regression network of this application may include a feature extraction module and a point cloud generation module. The output of the feature extraction module is connected to the input of the point cloud generation module.

[0092] So, in Figure 4 Based on the network model structure shown, such as Figure 5 As shown, step S202 above may include the following steps S202-1 to S202-3:

[0093] S202-1, Input the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image into the feature extraction module to obtain the high-dimensional feature vector corresponding to each view image output by the feature extraction module.

[0094] S202-2, input the high-dimensional feature vector corresponding to each viewpoint image into the point cloud generation module to obtain the point cloud data corresponding to the first training multi-viewpoint image output by the point cloud generation module.

[0095] S202-3, based on the point cloud data corresponding to the first training multi-view image and the coordinates of each pixel in each view image, train the scene coordinate regression network to be trained to obtain the scene coordinate regression network.

[0096] The aforementioned feature extraction module can be a feature extraction network used to extract high-dimensional semantic features from each image in the first training multi-view image input.

[0097] The aforementioned point cloud generation module can be a point cloud generation network used to generate point cloud data based on the high-dimensional semantic features output by the feature extraction module.

[0098] In this application, the point cloud generation module can be selected as a multi-layer perceptron (MLP). Then, after the MLP receives the high-dimensional semantic features output by the feature extraction module, it can construct a nonlinear mapping model from the feature space to three-dimensional coordinates to obtain point cloud data corresponding to the first training multi-view image.

[0099] In some alternative embodiments, the first loss value corresponding to the scene coordinate regression network to be trained can be determined by using formula (1) in the foregoing embodiments, based on the point cloud data corresponding to the first training multi-view image and the coordinates of each pixel in each view image.

[0100] Subsequently, this application can determine whether the first loss value is less than a first threshold. If the first loss value is greater than or equal to the first threshold, it indicates that the prediction effect of the scene coordinate regression network to be trained is poor. In this case, the scene coordinate regression network to be trained can be trained based on the first loss value to obtain the scene coordinate regression network. If the first loss value is less than the first threshold, it indicates that the prediction effect of the scene coordinate regression network to be trained is good. In this case, the training operation of the scene coordinate regression network to be trained can be ended, and the scene coordinate regression network to be trained can be used as the final scene coordinate regression network.

[0101] In some optional embodiments, the above-mentioned training of the scene coordinate regression network to be trained based on the first loss value to obtain the scene coordinate regression network specifically involves back-training the scene coordinate regression network to be trained based on the first loss value to adjust the network parameters in the scene coordinate regression network to be trained. That is, updating the weights and other parameters in the scene coordinate regression network to be trained.

[0102] Optionally, the above-mentioned scene coordinate regression network to be trained is back-trained based on the first loss value to adjust the network parameters in the scene coordinate regression network to be trained, as follows:

[0103] Based on the first loss value, the scene coordinate regression network to be trained is back-trained, and the new first loss value is determined again to be less than the first threshold. If the new first loss value is still greater than the first threshold, this step is repeated until the training stopping condition is met. If the latest first loss value is less than the first threshold, it means that the scene coordinate regression network after the latest training can predict dense and accurate point cloud data. At this time, the scene coordinate regression network after the latest training is taken as the final scene coordinate regression network.

[0104] In this application, the training stopping condition can be that the first loss value is less than a first threshold, or that the number of training iterations is greater than a preset number of iterations.

[0105] The first threshold and the preset number of times are both adjustable parameters, which can be flexibly set according to the prediction accuracy of the scene coordinate regression network. No restrictions are imposed on them here.

[0106] In other words, when a scene coordinate regression network with high prediction accuracy is required, this application can set the first threshold to be relatively small or the preset number of times to be relatively large; conversely, the first threshold can be set to be relatively large or the preset number of times can be set to be relatively small. The specific setting method can be flexibly set according to the training requirements of the scene coordinate regression network.

[0107] This application trains a scene coordinate regression network based on a first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image, so that the scene coordinate regression network can learn geometric information efficiently and accurately, thereby quickly and accurately generating point cloud data corresponding to the input multiple view images and high-dimensional feature vectors corresponding to each view image, providing favorable conditions for generating new view images.

[0108] Next, the training process of the prediction head network involved in the embodiments of this application will be described. It should be noted that this application requires the use of the previously trained scene coordinate regression network when training the prediction head network. That is, the network parameters in the scene coordinate regression network remain unchanged during the training of the prediction head network. In other words, the network parameters in the prediction head network can be modified during the training process, while the network parameters in the scene coordinate regression network remain frozen and will not be modified.

[0109] Figure 6 A schematic diagram of a training scenario for a prediction head network provided in an embodiment of this application; Figure 7 This is a flowchart illustrating the training process of a prediction head network, as provided in an embodiment of this application. Figure 7 As shown, the method may include the following steps:

[0110] S301, Obtain the second training multi-view image and the camera parameters corresponding to each view image in the second training multi-view image.

[0111] S302, input the second training multi-view image and the camera parameters corresponding to each view image in the second training multi-view image into the scene coordinate regression network to obtain the point cloud data output by the scene coordinate regression network and the high-dimensional feature vector corresponding to each view image in the second training multi-view image.

[0112] S303, based on the second training multi-view image, point cloud data and each high-dimensional feature vector, train the prediction head network to be trained to obtain the prediction head network.

[0113] In this application, the prediction head network to be trained refers to the prediction head network in the network model whose network parameters have not been adjusted. That is, the prediction head network in its initial state.

[0114] It should be understood that the aforementioned prediction head network refers to a deep learning model that generates 3D scene representation parameters for generating new perspective images based on point cloud data and high-dimensional feature vectors output by a scene coordinate regression network, as well as multi-view images.

[0115] In some embodiments, this application may obtain a training dataset for training the prediction head network to be trained. The training dataset may include multiple viewpoint images and camera parameters corresponding to each viewpoint image. The training dataset may be obtained using the same method as the first training multi-view image and each viewpoint image within the first training multi-view image; alternatively, the first training multi-view image may be used as the training dataset to achieve data reuse, improve data utilization, and reduce data acquisition costs.

[0116] Before training the prediction head network to be trained based on the training dataset, an arbitrary viewpoint image can be optionally obtained from the training dataset as a real new viewpoint image. At the same time, the camera parameters corresponding to the viewpoint image are removed from the training dataset to obtain a second training multi-viewpoint image for training the prediction head network to be trained, and the camera parameters corresponding to each viewpoint image in the second training multi-viewpoint image.

[0117] For example, suppose the training dataset includes 10 images from different viewpoints and the camera parameters corresponding to each of the 10 images from different viewpoints. Then, the third viewpoint image among these 10 images from different viewpoints can be used as the real new viewpoint image. At the same time, the camera parameters corresponding to the third viewpoint image are removed. Then, the other 9 images from different viewpoints, excluding the third viewpoint image and its corresponding camera parameters, are used as the second training multi-view images. And the camera parameters corresponding to each viewpoint image in these 9 images from different viewpoints are used as the camera parameters corresponding to each viewpoint image in the second training multi-view images.

[0118] In some alternative embodiments, considering that the scene coordinate regression network is a pre-trained model when the prediction head network to be trained is trained, this application can input the second training multi-view image and the camera parameters corresponding to each view image in the second training multi-view image into the pre-trained scene coordinate regression network, so that the scene coordinate regression network can output point cloud data and a high-dimensional feature vector corresponding to each view image based on the second training multi-view image and the camera parameters corresponding to each view image.

[0119] Next, the second training multi-view images, point cloud data, and the high-dimensional feature vectors corresponding to each view image are concatenated. The concatenated result is then input into the prediction head network to be trained. The prediction head network processes the concatenated result to output predicted 3D scene representation parameters. The number of predicted 3D scene representations is the same as the number of view images in the second training multi-view images. That is, each view image in the second training multi-view images corresponds to one predicted 3D scene representation parameter.

[0120] Next, the predicted 3D scene representation parameters output by the training prediction head network are rendered, and the rendering result is used as the predicted new viewpoint image. Then, the difference between the predicted new viewpoint image and the real new viewpoint image is determined to obtain a second loss value, and the training prediction head network is trained based on this second loss value to obtain the prediction head network.

[0121] Specifically, the aforementioned real new perspective images refer to any real new perspective images selected from the training dataset.

[0122] To further illustrate with the example above, in a training dataset containing 10 different viewpoint images and the corresponding camera parameters for each viewpoint image, the third viewpoint image is taken as the true new viewpoint image. Therefore, the true new viewpoint image is the third viewpoint image, and the predicted new viewpoint image corresponding to this true new viewpoint image is a new image with the same viewpoint as the third viewpoint image.

[0123] In some alternative embodiments, the second loss value obtained by determining the difference between the predicted new viewpoint image and the real new viewpoint image can be achieved by the following formula (2):

[0124] Loss2 = MSE(I render ,I gt )........................(2)

[0125] Where Loss2 is the second loss value between the predicted new perspective image and the real new perspective image; MSE() is the mean squared error loss function; I render To predict images from new perspectives; and I gt Images from a truly new perspective.

[0126] In some optional embodiments, after determining a second loss value between the predicted new viewpoint image and the real new viewpoint image, this application can determine whether the second loss value is less than a second threshold. If the second loss value is greater than or equal to the second threshold, it indicates that the prediction head network to be trained has a poor prediction effect. In this case, the prediction head network to be trained can be trained based on the second loss value to obtain the prediction head network. If the second loss value is less than the second threshold, it indicates that the prediction head network to be trained has a good prediction effect. In this case, the training operation of the prediction head network to be trained can be terminated, and the prediction head network to be trained can be used as the final prediction head network.

[0127] The second threshold mentioned above is an adjustable parameter, which can be set according to the training accuracy requirements of the prediction head network model. There are no restrictions on it here.

[0128] In some optional embodiments, the training prediction head network is performed based on the second loss value. Specifically, the network parameters in the training prediction head network are adjusted in reverse according to the second loss value, such as adjusting the weights and other parameters. That is, the weights and other parameters in the training prediction head network are updated.

[0129] Optionally, the network parameters in the prediction head network to be trained are adjusted in reverse according to the second loss value, as follows:

[0130] Based on the second loss value, the prediction head network to be trained is back-trained, and the new second loss value is determined again to be less than the second threshold. If the new second loss value is still greater than the second threshold, this step is repeated until the training stopping condition is met. If the latest second loss value is less than the second threshold, it means that the prediction head network trained in the latest iteration can predict a new viewpoint image that is pixel-level aligned with the real new viewpoint image. At this point, the prediction head network trained in the latest iteration is taken as the final prediction head network.

[0131] In this application, the training stopping condition can be that the second loss value is less than the second threshold, or that the number of training iterations is greater than the preset number of iterations.

[0132] The second threshold and the preset number of times are both adjustable parameters, which can be flexibly set according to the prediction accuracy of the prediction head network. No restrictions are imposed on them here.

[0133] In other words, when a prediction head network with high prediction accuracy is required, this application can set the second threshold to be relatively small or the preset number of times to be relatively large; conversely, the second threshold can be set to be relatively large or the preset number of times to be relatively small. The specific setting method can be flexibly set according to the training requirements of the prediction head network.

[0134] This application's embodiments train a prediction head network based on a second training multi-view image and the camera parameters corresponding to each view image in the second training multi-view image. This enables the prediction head network to output parameters for generating high-quality 3D scene representations for generating new view images. Furthermore, by using the above training method, compared to related technologies that involve long-term (hours to days) training of feedforward 3D Gaussian scatter models using large-scale datasets, this application can significantly shorten the training time of the prediction head network and improve the training efficiency of the network model.

[0135] After a detailed explanation of the training process of the scene coordinate regression network and prediction head network involved in the embodiments of this application, the following will be combined with... Figure 8 and Figure 9 The process of generating new perspective images through scene coordinate regression network and prediction head network is explained in detail.

[0136] Figure 8 This is a schematic diagram illustrating how a scene coordinate regression network and a prediction head network generate a new perspective image, as provided in an embodiment of this application. Figure 9 This is a flowchart illustrating another image generation method provided in this application embodiment. That is, this embodiment mainly introduces the process of generating new perspective images based on multi-view images using the scene coordinate regression network and prediction head network trained above.

[0137] S401, acquire multi-view images and the camera parameters corresponding to each view image in the multi-view images.

[0138] S402, input the multi-view images and the camera parameters corresponding to each view image into the scene coordinate regression network to obtain the point cloud data corresponding to the multi-view images output by the scene coordinate regression network, as well as the high-dimensional feature vector corresponding to each view image; wherein the scene coordinate regression network is trained based on multiple images of the same scene under different views.

[0139] It should be understood that the above-mentioned scene coordinate regression network is the network model obtained after training the scene coordinate regression network to be trained in the aforementioned embodiments.

[0140] In some alternative embodiments, the acquired multi-view images and the camera parameters corresponding to each view image in the multi-view images can be input into the scene coordinate regression network. The scene coordinate regression network can then quickly and accurately output the point cloud data corresponding to the multi-view images and the high-dimensional feature vector corresponding to each view image in the multi-view images based on the multi-view images and the camera parameters corresponding to each view image in the multi-view images.

[0141] S403 inputs multi-view images, point cloud data, and each high-dimensional feature vector into the prediction head network to obtain the 3D scene representation parameters output by the prediction head network; wherein the prediction head network is trained based on the scene coordinate regression network.

[0142] S404 renders the 3D scene representation parameters to obtain a new perspective image.

[0143] It should be understood that the aforementioned prediction head network is the network model obtained after training the prediction head network to be trained in the foregoing embodiments.

[0144] Furthermore, the prediction head network can include multiple layers of convolutional layers, enabling the prediction head network to acquire rich information at different levels to achieve pixel-level prediction tasks.

[0145] In some embodiments, point cloud data output by the scene coordinate regression network, multiple high-dimensional feature vectors, and multiple viewpoint images can be stitched together, and the stitched result can be used as input data to the prediction head network. This allows the prediction head network to predict other Gaussian parameters besides coordinate information based on the stitched result, specifically opacity information, covariance matrix, and spherical harmonic function coefficients, based on multiple layers of convolutional layers.

[0146] Then, the 3D scene representation parameters output by the prediction head network can be rendered to obtain a new perspective image. In this way, the generated new perspective image can achieve pixel-level alignment with the corresponding real image, resulting in extremely high rendering quality of the new perspective image.

[0147] The technical solution disclosed in this application obtains multi-view images and camera parameters corresponding to each view image in the multi-view images. Based on the multi-view images and camera parameters corresponding to each view image, the point cloud data corresponding to the multi-view images and the high-dimensional feature vector corresponding to each view image are determined. Then, based on the multi-view images, point cloud data and each high-dimensional feature vector, the three-dimensional scene representation parameters are determined. Finally, the three-dimensional scene representation parameters are rendered to obtain a new view image. In this way, a new view image with high realism and detail can be generated without the supervision of any geometric information such as depth maps, thereby improving the efficiency and quality of image generation.

[0148] In some embodiments, such as Figure 4 As shown, the scene coordinate regression network of this application may include: a feature extraction module and a point cloud generation module. See below for details. Figure 10 ,right Figure 9 Step S402 will be further explained. For example... Figure 10 As shown, step S402 above may include the following steps S402-1 to S402-2:

[0149] S402-1, input the multi-view images and the camera parameters corresponding to each view image into the feature extraction module to obtain the high-dimensional feature vector corresponding to each view image output by the feature extraction module.

[0150] S402-2, input the high-dimensional feature vector corresponding to each viewpoint image into the point cloud generation module to obtain the point cloud data corresponding to the multi-viewpoint image output by the point cloud generation module.

[0151] In some optional embodiments, considering that determining the 3D scene representation parameters based on multi-view images and point cloud data and high-dimensional feature vectors output by the scene coordinate regression network may result in checkerboard artifacts, i.e., visual distortion, problems may exist. Therefore, after obtaining the point cloud data corresponding to the multi-view images and the high-dimensional feature vectors corresponding to each view image, this application may optionally further include: determining the upsampled point cloud data corresponding to the point cloud data and the upsampled high-dimensional feature vectors corresponding to each high-dimensional feature vector. Subsequently, the 3D scene representation parameters can be determined based on the multi-view images, the upsampled point cloud data, and each upsampled high-dimensional feature vector, thus avoiding visual distortion while maintaining computational efficiency, thereby improving the generation quality of new view images.

[0152] In some embodiments, the determination of the upsampled point cloud data corresponding to the point cloud data and the upsampled high-dimensional feature vector corresponding to each high-dimensional feature vector in this application can be optionally performed as follows: inputting the point cloud data into an upsampling network to obtain the upsampled point cloud data output by the upsampling network, and inputting each high-dimensional feature vector into the upsampling network to obtain the upsampled high-dimensional feature vector corresponding to each high-dimensional feature vector output by the upsampling network, such as... Figure 11 As shown.

[0153] In this application, the upsampling network can be pre-trained or trained synchronously during the training of the prediction head network; no specific restrictions are imposed here. Furthermore, the training process for the upsampling network can be found in existing upsampling network training procedures, which will not be elaborated upon here.

[0154] This application employs an upsampling network to upsample the point cloud data and high-dimensional feature vectors output by the scene coordinate regression network. This process progressively upsamples the lower-resolution point cloud data and the corresponding images of the high-dimensional feature vectors to the same size as the higher-resolution multi-view images, thereby achieving dimensionality unification for different types of images. This enables pixel-level dense prediction tasks based on dimensionally unified images. Specifically, the different types of images refer to the point cloud images corresponding to the point cloud data, the high-dimensional feature images corresponding to the high-dimensional feature vectors, and the multi-view images.

[0155] In some embodiments, such as Figure 12As shown, the upsampling network of this application may include an interpolation module and an optimization module. The input of the interpolation module is connected to the output vector of the scene coordinate regression network, and the output of the interpolation module is connected to the input of the optimization module.

[0156] So, in Figure 12 Based on the network model structure shown, the aforementioned steps of inputting point cloud data into the upsampling network to obtain upsampled point cloud data output by the upsampling network, and inputting each high-dimensional feature vector into the upsampling network to obtain the upsampled high-dimensional feature vector corresponding to each high-dimensional feature vector output by the upsampling network, can include the following:

[0157] Step S1: Input the point cloud data into the interpolation module to obtain the processed point cloud data output by the interpolation module. Input the processed point cloud data into the optimization module to obtain the upsampled point cloud data output by the optimization module.

[0158] Step S2: Input each high-dimensional feature vector into the interpolation module to obtain the processed high-dimensional feature vector output by the interpolation module. Input each processed high-dimensional feature vector into the optimization module to obtain the upsampled high-dimensional feature vector corresponding to each high-dimensional feature vector output by the optimization module.

[0159] The interpolation module mentioned above can be a difference module that implements the interpolation function, and can be selected as a bilinear difference module or a bilinear difference layer.

[0160] The above optimization module can be a convolutional layer.

[0161] In other words, the upsampling network in this application employs a combination of bilinear interpolation and convolutional layers to implement an upsampling strategy. Compared to the traditional transposed convolution method, this upsampling strategy effectively avoids the checkerboard artifact problem while maintaining computational efficiency. Furthermore, it can effectively fill sparse regions in point cloud data and high-dimensional feature vectors, while preserving the continuity of the geometric structure.

[0162] The following is a reference to the appendix. Figure 13 This application describes an image generation apparatus based on an embodiment. Figure 13 As shown, the image generation device 500 includes: an image acquisition module 510, a first determination module 520, a second determination module 530, and a generation module 540.

[0163] The image acquisition module 510 is used to acquire multi-view images and camera parameters corresponding to each view image in the multi-view images;

[0164] The first determining module 520 is used to determine the point cloud data corresponding to the multi-view image and the high-dimensional feature vector corresponding to each view image based on the multi-view image and the camera parameters corresponding to each view image in the multi-view image.

[0165] The second determining module 530 is used to determine the three-dimensional scene representation parameters based on the multi-view image, the point cloud data, and each of the high-dimensional feature vectors;

[0166] The generation module 540 is used to render the three-dimensional scene representation parameters to obtain a new perspective image.

[0167] In one optional implementation of this application embodiment, the first determining module 520 is specifically used for:

[0168] The multi-view images and the camera parameters corresponding to each view image are input into the scene coordinate regression network to obtain the point cloud data corresponding to the multi-view images output by the scene coordinate regression network, and the high-dimensional feature vector corresponding to each view image; wherein the scene coordinate regression network is trained based on multiple images of the same scene under different views.

[0169] In one optional implementation of this application embodiment, the scene coordinate regression network includes: a feature extraction module and a point cloud generation module, wherein the first determining module 520 is further configured to:

[0170] The multi-view images and the camera parameters corresponding to each view image in the multi-view images are input into the feature extraction module to obtain the high-dimensional feature vector corresponding to each view image output by the feature extraction module;

[0171] The high-dimensional feature vector corresponding to each of the aforementioned viewpoint images is input into the point cloud generation module to obtain the point cloud data corresponding to the multi-viewpoint images output by the point cloud generation module.

[0172] In one optional implementation of this application embodiment, the second determining module 530 is specifically used for:

[0173] The multi-view images, the point cloud data, and each of the high-dimensional feature vectors are input into the prediction head network to obtain the three-dimensional scene representation parameters output by the prediction head network; wherein the prediction head network is trained based on the scene coordinate regression network.

[0174] In one optional implementation of this application, the prediction head network includes multiple layers of convolutional layers.

[0175] In one optional implementation of this application embodiment, the image generating apparatus 500 further includes:

[0176] The sampling module is used to determine the upsampled point cloud data corresponding to the point cloud data, and the upsampled high-dimensional feature vector corresponding to each of the high-dimensional feature vectors.

[0177] In one optional implementation of this application embodiment, the sampling module is specifically used for:

[0178] The point cloud data is input into the upsampling network to obtain the upsampled point cloud data output by the upsampling network, and each of the high-dimensional feature vectors is input into the upsampling network to obtain the upsampled high-dimensional feature vector corresponding to each of the high-dimensional feature vectors output by the upsampling network.

[0179] In one optional implementation of this application embodiment, the upsampling network includes: an interpolation module and an optimization module, wherein the sampling module is further configured to:

[0180] The point cloud data is input into the interpolation module to obtain the processed point cloud data output by the interpolation module. The processed point cloud data is then input into the optimization module to obtain the upsampled point cloud data output by the optimization module.

[0181] Each of the high-dimensional feature vectors is input into the interpolation module to obtain the processed high-dimensional feature vectors output by the interpolation module. Each of the processed high-dimensional feature vectors is then input into the optimization module to obtain the upsampled high-dimensional feature vectors corresponding to each high-dimensional feature vector output by the optimization module.

[0182] In an optional implementation of this application embodiment, the second determining module 530 is further configured to:

[0183] The three-dimensional scene representation parameters are determined based on the multi-view images, the upsampled point cloud data, and each upsampled high-dimensional feature vector.

[0184] In one optional implementation of this application embodiment, the image generating apparatus 500 further includes:

[0185] The first acquisition module is used to acquire the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image.

[0186] The first training module is used to train the scene coordinate regression network to be trained based on the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image.

[0187] In one optional implementation of this application embodiment, the image generating apparatus 500 further includes:

[0188] The second acquisition module is used to acquire the second training multi-view image and the camera parameters corresponding to each view image in the second training multi-view image;

[0189] The processing module is used to input the second training multi-view image and the camera parameters corresponding to each view image in the second training multi-view image into the scene coordinate regression network to obtain the point cloud data output by the scene coordinate regression network and the high-dimensional feature vector corresponding to each view image in the second training multi-view image.

[0190] The second training module is used to train the prediction head network to be trained based on the second training multi-view image, the point cloud data, and each of the high-dimensional feature vectors.

[0191] It should be understood that the device embodiments and the foregoing method embodiments can correspond to each other, and similar descriptions can be found in the first aspect of the method embodiments. To avoid repetition, further details are omitted here. Specifically, Figure 13 The device 500 shown can perform Figure 1 The corresponding method embodiments, and the foregoing and other operations and / or functions of each module in device 500 are respectively implemented to achieve Figure 1 For the sake of brevity, the corresponding processes in each method are not described in detail here.

[0192] The apparatus 500 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the first aspect method embodiment in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the first aspect method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the first aspect method embodiment described above.

[0193] Figure 14 This is a schematic block diagram of an electronic device provided as an embodiment of this application. Figure 14As shown, the electronic device 600 may include a memory 610 and a processor 620. The memory 610 stores computer programs and transmits the program code to the processor 620. In other words, the processor 620 can call and run the computer program from the memory 610 to implement the image generation method described in the first aspect of the embodiments of this application.

[0194] For example, the processor 620 can be used to execute the image generation method described in the first aspect above according to instructions in the computer program.

[0195] In some embodiments of this application, the processor 620 may include, but is not limited to:

[0196] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0197] In some embodiments of this application, the memory 610 includes, but is not limited to:

[0198] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0199] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 610 and executed by the processor 620 to complete the image generation method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device.

[0200] like Figure 14 As shown, the electronic device 600 may further include:

[0201] Transceiver 630, which can be connected to processor 620 or memory 610.

[0202] The processor 620 can control the transceiver 630 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 630 may include a transmitter and a receiver. The transceiver 630 may further include antennas, and the number of antennas may be one or more.

[0203] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0204] This application also provides a computer storage medium having a computer program stored thereon, which, when executed by a computer, enables the computer to perform the image generation method described in the first aspect above.

[0205] This application also provides a computer program product containing program instructions that, when executed on an electronic device, cause the electronic device to perform the image generation method described in the first aspect above.

[0206] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0207] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0208] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0209] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0210] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0211] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image generation method, characterized in that, include: Acquire multi-view images and the camera parameters corresponding to each view image in the multi-view images; The multi-view images and the camera parameters corresponding to each view image are input into the scene coordinate regression network to obtain the point cloud data corresponding to the multi-view images output by the scene coordinate regression network, and the high-dimensional feature vector corresponding to each view image; wherein the scene coordinate regression network is trained based on multiple images of the same scene under different views. The multi-view images, the point cloud data, and each of the high-dimensional feature vectors are input into the prediction head network to obtain the three-dimensional scene representation parameters output by the prediction head network; wherein the prediction head network is trained based on the scene coordinate regression network. The three-dimensional scene representation parameters are rendered to obtain a new perspective image; The scene coordinate regression network includes a feature extraction module and a point cloud generation module. The process involves inputting the multi-view images and the camera parameters corresponding to each view image into the scene coordinate regression network to obtain the point cloud data corresponding to the multi-view images output by the scene coordinate regression network, and a high-dimensional feature vector corresponding to each view image, including: The multi-view images and the camera parameters corresponding to each view image in the multi-view images are input into the feature extraction module to obtain the high-dimensional feature vector corresponding to each view image output by the feature extraction module; The high-dimensional feature vector corresponding to each of the aforementioned viewpoint images is input into the point cloud generation module to obtain the point cloud data corresponding to the multi-viewpoint images output by the point cloud generation module. The high-dimensional feature vector refers to high-dimensional semantic features, which are image features extracted from each viewpoint image that have semantic meaning and are represented in the form of high-dimensional vectors.

2. The method according to claim 1, characterized in that, The prediction head network includes multiple layers of convolutional layers.

3. The method according to claim 1, characterized in that, The method further includes: Determine the upsampled point cloud data corresponding to the point cloud data, and the upsampled high-dimensional feature vector corresponding to each of the high-dimensional feature vectors.

4. The method according to claim 3, characterized in that, The step of determining the upsampled point cloud data corresponding to the point cloud data, and the upsampled high-dimensional feature vector corresponding to each high-dimensional feature vector, includes: The point cloud data is input into the upsampling network to obtain the upsampled point cloud data output by the upsampling network, and each of the high-dimensional feature vectors is input into the upsampling network to obtain the upsampled high-dimensional feature vector corresponding to each of the high-dimensional feature vectors output by the upsampling network.

5. The method according to claim 4, characterized in that, The upsampling network includes an interpolation module and an optimization module. The steps of inputting the point cloud data into the upsampling network to obtain the upsampled point cloud data output by the upsampling network, and inputting each high-dimensional feature vector into the upsampling network to obtain the upsampled high-dimensional feature vector corresponding to each high-dimensional feature vector output by the upsampling network, include: The point cloud data is input into the interpolation module to obtain the processed point cloud data output by the interpolation module. The processed point cloud data is then input into the optimization module to obtain the upsampled point cloud data output by the optimization module. Each of the high-dimensional feature vectors is input into the interpolation module to obtain the processed high-dimensional feature vectors output by the interpolation module. Each of the processed high-dimensional feature vectors is then input into the optimization module to obtain the upsampled high-dimensional feature vectors corresponding to each high-dimensional feature vector output by the optimization module.

6. The method according to any one of claims 3 to 5, characterized in that, Determining the parameters for representing a 3D scene also includes: The three-dimensional scene representation parameters are determined based on the multi-view images, the upsampled point cloud data, and each upsampled high-dimensional feature vector.

7. The method according to claim 1, characterized in that, The method further includes: Obtain the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image; Based on the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image, the scene coordinate regression network is trained to obtain the scene coordinate regression network.

8. The method according to claim 1, characterized in that, The method further includes: Obtain the second training multi-view image and the camera parameters corresponding to each view image in the second training multi-view image; The second training multi-view image and the camera parameters corresponding to each view image in the second training multi-view image are input into the scene coordinate regression network to obtain the point cloud data output by the scene coordinate regression network and the high-dimensional feature vector corresponding to each view image in the second training multi-view image. The prediction head network is obtained by training the prediction head network based on the second training multi-view image, the point cloud data, and each of the high-dimensional feature vectors.

9. An image generation apparatus, characterized in that, include: An image acquisition module is used to acquire multi-view images and camera parameters corresponding to each view image in the multi-view images; The first determining module inputs the multi-view images and the camera parameters corresponding to each view image into the scene coordinate regression network to obtain the point cloud data corresponding to the multi-view images output by the scene coordinate regression network, and the high-dimensional feature vector corresponding to each view image; wherein the scene coordinate regression network is trained based on multiple images of the same scene under different views. The second determining module is used to input the multi-view image, the point cloud data, and each of the high-dimensional feature vectors into the prediction head network to obtain the three-dimensional scene representation parameters output by the prediction head network; wherein the prediction head network is trained based on the scene coordinate regression network. The generation module is used to render the 3D scene representation parameters to obtain a new perspective image; The scene coordinate regression network includes a feature extraction module and a point cloud generation module. The process involves inputting the multi-view images and the camera parameters corresponding to each view image into the scene coordinate regression network to obtain the point cloud data corresponding to the multi-view images output by the scene coordinate regression network, and a high-dimensional feature vector corresponding to each view image, including: The multi-view images and the camera parameters corresponding to each view image in the multi-view images are input into the feature extraction module to obtain the high-dimensional feature vector corresponding to each view image output by the feature extraction module; The high-dimensional feature vector corresponding to each of the aforementioned viewpoint images is input into the point cloud generation module to obtain the point cloud data corresponding to the multi-viewpoint images output by the point cloud generation module. The high-dimensional feature vector refers to high-dimensional semantic features, which are image features extracted from each viewpoint image that have semantic meaning and are represented in the form of high-dimensional vectors.

10. An electronic device, characterized in that, include: A processor and a memory, the memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 8.

12. A computer program product containing program instructions, characterized in that, When the program instructions are executed on the electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1 to 8.

Images