Image generation method and device, equipment and medium

Abstract

The invention provides an image generation method and device, equipment and a medium. The method comprises the steps that a first target view angle image and camera parameters corresponding to the first target view angle image are acquired; determining point cloud data and high-dimensional feature vectors corresponding to the first target view angle image according to the first target view angle image and camera parameters corresponding to the first target view angle image; determining a three-dimensional scene representation parameter according to the first target view angle image, the point cloud data and the high-dimensional feature vector; and rendering the three-dimensional scene representation parameter to obtain a second target view angle image. According to the method and the device, the generated virtual image can be accurately anchored on the fixed coordinate in the virtual scene, and the scale information of scene cross-view consistency can be obtained without any geometric priori.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image generation method, apparatus, device, and medium. Background Technology

[0002] With the rapid development of Extended Reality (XR) technology, electronic devices that support XR technology can provide users with a variety of virtual scenes, allowing users to interact with virtual scenes in an immersive way.

[0003] To provide users with a more realistic and immersive experience, images from a specific perspective in a real scene can be captured and then reproduced in a virtual scene. In related technologies, when reproducing the image from that specific perspective in a virtual scene, the image from the real scene is input into a prediction model to predict the corresponding 3D scene representation parameters. Then, based on the perspective of the image from that specific perspective, the 3D scene representation parameters are rendered to obtain a virtual image from the same perspective.

[0004] However, the aforementioned prediction model is trained based on a single-view image, which means that the model can only learn to accurately map the input 2D image back to the same output image, but cannot truly understand the 3D geometric information in the single-view image. As a result, the prediction model performs well in the training viewpoint, but because the 3D scene representation parameters learned by the prediction model itself lack effective modeling of real 3D geometric information, in the virtual scene, the 3D scene representation parameters output by the prediction model based on the specific viewpoint image lack correct geometric information. Therefore, the virtual image obtained based on the 3D scene representation parameters cannot be accurately anchored to fixed coordinates in the virtual scene, and the physical size of the virtual image cannot be accurately recovered, resulting in a scale blur problem. Summary of the Invention

[0005] This application provides an image generation method, apparatus, device, and medium that can accurately anchor the generated virtual image to fixed coordinates in a virtual scene without any geometric priors, thus achieving scene cross-view accuracy. Figure 1 Scale information.

[0006] In a first aspect, embodiments of this application provide an image generation method, including:

[0007] Acquire the first target view image and the camera parameters corresponding to the first target view image;

[0008] Based on the first target view image and the camera parameters corresponding to the first target view image, determine the point cloud data and high-dimensional feature vector corresponding to the first target view image;

[0009] The three-dimensional scene representation parameters are determined based on the first target view image, the point cloud data, and the high-dimensional feature vector;

[0010] The three-dimensional scene representation parameters are rendered to obtain a second target view image.

[0011] Secondly, embodiments of this application provide an image generation apparatus, comprising:

[0012] An image acquisition module is used to acquire a first target view image and the camera parameters corresponding to the first target view image;

[0013] The first determining module is used to determine the point cloud data and high-dimensional feature vector corresponding to the first target view image based on the first target view image and the camera parameters corresponding to the first target view image.

[0014] The second determining module is used to determine the three-dimensional scene representation parameters based on the first target view image, the point cloud data, and the high-dimensional feature vector;

[0015] The generation module is used to render the three-dimensional scene representation parameters to obtain a second target view image.

[0016] Thirdly, embodiments of this application provide an electronic device, including:

[0017] A processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the image generation method as described in the foregoing first aspect embodiments and various implementations.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to perform the image generation method as described in the foregoing first aspect embodiments and various implementations.

[0019] Fifthly, embodiments of this application provide a computer program product containing program instructions that, when executed on an electronic device, cause the electronic device to perform the image generation method as described in the first aspect embodiment and its various implementations.

[0020] The technical solution disclosed in this application involves acquiring a first target view image and the corresponding camera parameters. Based on these parameters, point cloud data and a high-dimensional feature vector corresponding to the first target view image are determined. Then, based on the first target view image, point cloud data, and high-dimensional feature vector, 3D scene representation parameters are determined. Finally, the 3D scene representation parameters are rendered to obtain a second target view image. This allows the generated virtual image to be precisely anchored to fixed coordinates within the virtual scene without requiring any geometric priors, thus achieving cross-view scene representation. Figure 1 Scale information. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating an image generation method provided in this application embodiment;

[0023] Figure 2 A schematic diagram of a training scenario for a scene coordinate regression network provided in an embodiment of this application;

[0024] Figure 3 A training flowchart for a scene coordinate regression network provided in an embodiment of this application;

[0025] Figure 4 This is a schematic diagram of the structure of a scene coordinate regression network provided in an embodiment of this application;

[0026] Figure 5 A flowchart illustrating the training of a scene coordinate regression network is provided in this application embodiment;

[0027] Figure 6 A schematic diagram of a training scenario for a prediction head network provided in an embodiment of this application;

[0028] Figure 7 A flowchart illustrating the training process of a prediction head network provided in an embodiment of this application;

[0029] Figure 8 This is a schematic diagram illustrating how a second target view image is generated based on a scene coordinate regression network and a prediction head network, as provided in an embodiment of this application.

[0030] Figure 9 A flowchart illustrating another image generation method provided in an embodiment of this application;

[0031] Figure 10 A flowchart for acquiring point cloud data and high-dimensional feature vectors is provided in an embodiment of this application;

[0032] Figure 11 A schematic diagram illustrating the acquisition of upsampled point cloud data and upsampled high-dimensional feature vectors based on an upsampled network, as provided in this application embodiment;

[0033] Figure 12 This is a schematic diagram of the structure of an upsampling network provided in an embodiment of this application;

[0034] Figure 13 A schematic block diagram of an image generation apparatus provided in an embodiment of this application;

[0035] Figure 14 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0037] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0038] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or solution described as "exemplary" or "for example" in this application should not be construed as being better or more advantageous than other embodiments or solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0039] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more, that is, at least two. "At least one" means one or more. "Any" means any one or any several.

[0040] As mentioned earlier, in related technologies, when reproducing a specific viewpoint image from a real scene in a virtual scene, the specific viewpoint image is input into a prediction model to predict the corresponding 3D scene representation parameters. Then, based on the viewpoint of the specific viewpoint image, the 3D scene representation parameters are rendered to obtain a virtual image under the same viewpoint. The prediction model is a deep learning model.

[0041] However, the aforementioned prediction model is trained based on single-view images. Specifically, this model receives a single-view image as input, predicts the corresponding 3D scene representation parameters, renders the predicted 3D scene representation parameters into an image from the perspective of that single-view image, and then optimizes the predicted 3D scene representation parameters to achieve pixel-level alignment with the input single-view image. This training method means that the prediction model can only learn to accurately map the input 2D image back to the same output image, but cannot truly understand the 3D geometric information in the single-view image. As a result, the prediction model performs well on the training viewpoint, but because the 3D scene representation parameters learned by the prediction model itself lack effective modeling of real 3D geometric information, in virtual scenes, the 3D scene representation parameters output by the prediction model based on a specific viewpoint image lack correct geometric information. Therefore, the virtual image obtained based on these 3D scene representation parameters cannot be accurately anchored to fixed coordinates in the virtual scene, and the physical size of the virtual image cannot be accurately recovered, resulting in scale blurring.

[0042] To address the aforementioned technical problems, this application provides an image generation method, apparatus, device, and medium to solve the problem of scale blurring in related technologies, where the prediction model outputs 3D scene representation parameters based on images from a specific viewpoint, which lacks correct geometric information. This results in virtual images obtained based on these 3D scene representation parameters being unable to be accurately anchored to fixed coordinates in the virtual scene, and also unable to accurately recover the physical size of the virtual image.

[0043] In this application, the aforementioned virtual scene refers to a virtual scene displayed when an application runs on an electronic device. This virtual scene can be a simulation of the real world, a semi-simulated / semi-fictional virtual scene, or a purely fictional virtual scene. The virtual scene can be any one of a two-dimensional, 2.5-dimensional, or three-dimensional virtual scene; this application does not limit the dimension of the virtual scene. For example, a virtual scene may include the sky, land, ocean, etc., and the land may include environmental elements such as deserts and cities.

[0044] The technical solutions provided by the embodiments of this application will be described in detail below through some examples. The embodiments described below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0045] Figure 1 This is a flowchart illustrating an image generation method provided in an embodiment of this application. In this embodiment, the image generation method can be executed by an image generation device. This image generation device can consist of hardware and / or software and can be integrated into an electronic device. In this application, the electronic device can be, but is not limited to, wearable devices, tablet computers, desktop computers, and mobile devices (such as smartphones). Figure 1 As shown, the image generation method may include the following steps:

[0046] S101, acquire the first target view image and the corresponding camera parameters.

[0047] In this application, the first target viewpoint image refers to a scene image captured from any fixed viewpoint of any real scene. It should be understood that the first target viewpoint image specifically refers to a real target viewpoint image, that is, a specific viewpoint image in a real scene.

[0048] The aforementioned real-world scene may include one or more objects and their surrounding environment. That is, the real-world scene may include at least one of foreground information and background information, where the foreground information may be an object, and the background information may be the object's surrounding environment. Furthermore, the aforementioned object may be a person, an animal, or other object.

[0049] The camera parameters corresponding to the aforementioned first target viewpoint image can refer to the camera's intrinsic and extrinsic parameters. Furthermore, based on these camera parameters, the shooting angle of the first target viewpoint image can be obtained.

[0050] In some optional embodiments, any real scene can be photographed from any fixed angle using a camera or other device with shooting capabilities to obtain a first target viewpoint image. Furthermore, while photographing the first target viewpoint image, this application can also obtain the camera parameters corresponding to when the camera or other device photographed the first target viewpoint image.

[0051] S102, based on the first target view image and the camera parameters corresponding to the first target view image, determine the point cloud data and high-dimensional feature vector corresponding to the first target view image.

[0052] Point cloud data can be understood as scene point cloud, and each point cloud in the scene point cloud can represent a three-dimensional coordinate.

[0053] The aforementioned high-dimensional feature vector specifically refers to high-dimensional semantic features, which are image features extracted from the first target viewpoint image that have semantic meaning and are represented in the form of high-dimensional vectors.

[0054] In some optional embodiments, this application can use a machine learning model that supports the simultaneous generation of point cloud data and high-dimensional feature vectors to process the first target view image and the camera parameters corresponding to the first target view image, so as to obtain the point cloud data and high-dimensional feature vectors corresponding to the first target view image.

[0055] The aforementioned machine learning model that supports the simultaneous generation of point cloud data and high-dimensional feature vectors can be selected as a scene coordinate regression network, etc. Optionally, when the machine learning model is a scene coordinate regression network, the scene coordinate regression network can adopt techniques for accelerating and optimizing coordinate representation, such as Accelerated Coordinate Encoding (ACE) technology, so as to leverage the fast and accurate point cloud generation capability of ACE technology to provide accurate and reliable geometric prior support for the scene coordinate regression network.

[0056] In some alternative embodiments, this application may also use the Structure-from-Motion (SfM) algorithm to process the first target view image and the camera parameters corresponding to the first target view image to obtain point cloud data and high-dimensional feature vectors corresponding to the first target view image.

[0057] S103, determine the 3D scene representation parameters based on the first target view image, point cloud data, and high-dimensional feature vector.

[0058] The aforementioned three-dimensional scene representation parameters refer to parameter information generated based on the first target viewpoint image and used to reproduce the first target viewpoint image in a virtual scene.

[0059] Furthermore, the aforementioned 3D scene representation parameters may include: coordinate information, opacity information, covariance matrix, and spherical harmonic function coefficients. Specifically, coordinate information refers to position information.

[0060] In some embodiments, considering the 3D Gaussian Splatting (3DGS) algorithm, which has real-time rendering speed and high-fidelity visual rendering effect, this application can process the first target view image, point cloud data and high-dimensional feature vector based on the 3DGS algorithm to obtain the three-dimensional scene representation parameters.

[0061] In some embodiments, this application may also process the first target view image, point cloud data, and high-dimensional feature vectors through a machine learning model that supports the generation of three-dimensional scene representation parameters to obtain three-dimensional scene representation parameters.

[0062] The aforementioned machine learning model that supports generating 3D scene representation parameters can be, for example, a prediction head network. It should be understood that the prediction head network is used to perform predictions for a specific task. Optionally, when the machine learning model is a prediction head network, this prediction head network specifically refers to a Gaussian regression head network, used to perform regression tasks with Gaussian distribution parameters.

[0063] S104 renders the 3D scene representation parameters to obtain the second target view image.

[0064] The second target viewpoint image refers to a virtual image that has the same viewpoint as the first target viewpoint image.

[0065] It should be understood that the second target perspective image specifically refers to the virtual target perspective image, that is, the first target perspective image in the real scene can be reproduced in the virtual scene based on the second target perspective image.

[0066] After obtaining the three-dimensional scene representation parameters, this application can render the three-dimensional scene representation parameters based on the shooting angle of the first target view image to obtain the second target view image.

[0067] Understandably, this application determines point cloud data and high-dimensional feature vectors based on a first target view image and the corresponding camera parameters. Then, based on the first target view image, point cloud data, and high-dimensional feature vectors, it obtains 3D scene representation parameters with correct geometric information. This allows the static view virtual image rendered based on these 3D scene representation parameters to be accurately bound to fixed coordinates in the virtual scene without any external geometric priors, thus presenting a stable and high-fidelity static scene rendering effect. Furthermore, it can effectively solve the problem of size blurring caused by the inability to accurately recover the physical size of the virtual image.

[0068] The technical solution disclosed in this application involves acquiring a first target view image and the corresponding camera parameters. Based on these parameters, point cloud data and a high-dimensional feature vector corresponding to the first target view image are determined. Then, based on the first target view image, point cloud data, and high-dimensional feature vector, 3D scene representation parameters are determined. Finally, the 3D scene representation parameters are rendered to obtain a second target view image. This allows the generated virtual image to be precisely anchored to fixed coordinates within the virtual scene without requiring any geometric priors, thus achieving cross-view scene representation. Figure 1Scale information.

[0069] In some alternative embodiments, considering that the point cloud data and high-dimensional feature vectors are determined in this application, they can be achieved through machine learning models such as scene coordinate regression networks; and the determination of the 3D scene representation parameters can also be achieved through corresponding machine learning models such as prediction head networks.

[0070] Therefore, this application may optionally provide a detailed description of the training process of the machine learning model involved in the embodiments of this application, as well as a detailed description of the specific process of using the trained machine learning model to determine point cloud data, high-dimensional feature vectors, and 3D scene representation parameters.

[0071] In other words, the technical solution provided in this application can include two parts: a first part is a model training part and a second part is a model application part. Furthermore, the model involved in this application can include a scene coordinate regression network and a prediction head network.

[0072] First, combine Figure 2 and Figure 3 The training process of the scene coordinate regression network involved in the embodiments of this application will be described. Figure 2 A schematic diagram of a training scenario for a scene coordinate regression network provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the training process of a scene coordinate regression network provided in an embodiment of this application. Figure 3 As shown, the method may include the following steps:

[0073] S201, Obtain the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image.

[0074] The first training multi-view image refers to the training data used to train the scene coordinate regression network.

[0075] Optionally, this application obtains the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image, which can be achieved in various ways, specifically at least one of the following:

[0076] The first method involves using a camera or other device with shooting capabilities to capture images of any real-world scene from different perspectives, thereby obtaining multi-view images, which are then used as the first training multi-view images.

[0077] Furthermore, while capturing images from multiple perspectives, it can also acquire the camera parameters corresponding to each perspective image captured by the camera or other devices.

[0078] The second approach involves obtaining images from different perspectives corresponding to any real-world scene from an open-source image database to create a first training multi-view image, and then obtaining the camera parameters corresponding to each perspective image in the first training multi-view image.

[0079] In addition to the two methods mentioned above, this application may also use other methods to obtain the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image, and this application does not impose any restrictions on this.

[0080] S202, based on the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image, train the scene coordinate regression network to be trained to obtain the scene coordinate regression network.

[0081] In this application, the scene coordinate regression network to be trained refers to the scene coordinate regression network in the network model whose network parameters have not been adjusted. That is, the scene coordinate regression network in its initial state.

[0082] It should be understood that the aforementioned scene coordinate regression network is a deep learning model that outputs three-dimensional scene coordinates based on the input image, used to map two-dimensional pixel coordinates in an image to three-dimensional scene coordinates.

[0083] Furthermore, considering that ACE technology can fundamentally revolutionize the technical paradigm that requires supervised training based on geometric priors, enabling the model to simultaneously optimize coordinate predictions for multiple scene views in a single iteration under the supervision of the reprojection loss function, and directly generate dense and accurate point cloud data through end-to-end learning. Crucially, ACE technology can rapidly complete new scene mapping, achieving a significant efficiency breakthrough compared to the lengthy mapping time of traditional optimal scene coordinate regression systems. Therefore, the scene coordinate regression network in this application can employ techniques for accelerating and optimizing coordinate representation, such as ACE technology, to leverage ACE's rapid and accurate point cloud generation capabilities, providing precise and reliable geometric prior support for the scene coordinate regression network.

[0084] In some optional embodiments, training the scene coordinate regression network to obtain the scene coordinate regression network may include the following steps:

[0085] The first step is to input the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image into the scene coordinate regression network to be trained, so that the scene coordinate regression network to be trained processes the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image to obtain the point cloud data corresponding to the first training multi-view image output by the scene coordinate regression network to be trained.

[0086] The second step is to determine the first loss value for the scene coordinate regression network to be trained based on the point cloud data corresponding to the first training multi-view images and the pixel coordinates in each view image.

[0087] The third step is to train the scene coordinate regression network based on the first loss value to obtain the scene coordinate regression network.

[0088] Optionally, the first loss value corresponding to the scene coordinate regression network to be trained can be determined by the following formula (1):

[0089]

[0090] Where Loss1 is the first loss value corresponding to the scene coordinate regression network to be trained; argmin is the value of the independent variable that minimizes the loss function; w is the learnable parameter of the scene coordinate regression network to be trained; i is the i-th image in the first training multi-view images; N is the N-th image in the first training multi-view images, where N is a positive integer greater than or equal to 2; X j Let I be the coordinates of the j-th pixel in the i-th image; i For the i-th image; Y is the reprojection loss function; j T represents the predicted 3D coordinates of the j-th pixel in the i-th image; i Let X be the camera parameters corresponding to the i-th image; where X j With Y j Related.

[0091] In some optional embodiments, after determining the first loss value corresponding to the scene coordinate regression network to be trained, this application can determine whether the first loss value is less than a first threshold. If the first loss value is greater than or equal to the first threshold, it indicates that the prediction effect of the scene coordinate regression network to be trained is poor. In this case, the scene coordinate regression network to be trained can be trained based on the first loss value to obtain the scene coordinate regression network. If the first loss value is less than the first threshold, it indicates that the prediction effect of the scene coordinate regression network to be trained is good. In this case, the training operation of the scene coordinate regression network to be trained can be ended, and the scene coordinate regression network to be trained can be used as the final scene coordinate regression network.

[0092] The first threshold mentioned above is an adjustable parameter, which can be set according to the training accuracy requirements of the scene coordinate regression network model. There are no restrictions on it here.

[0093] In some alternative embodiments, the scene coordinate regression network to be trained is trained based on the first loss value. Specifically, the network parameters in the scene coordinate regression network to be trained are adjusted, such as adjusting the weights and other parameters.

[0094] This application embodiment obtains a first training multi-view image and the camera parameters corresponding to each view image, and trains the scene coordinate regression network to be trained based on the first training multi-view image and the camera parameters corresponding to each view image to obtain a scene coordinate regression network, so as to realize the rapid acquisition of scene cross-view images by training the scene coordinate regression network. Figure 1 The method provides consistent size information and geometric priors without requiring any external geometric information such as depth maps for supervision. Furthermore, by employing the aforementioned training method to train the scene coordinate regression network, this application can significantly shorten the training time of the scene coordinate regression network and improve the training efficiency of the network model.

[0095] In some alternative embodiments, such as Figure 4 As shown, the scene coordinate regression network of this application may include a feature extraction module and a point cloud generation module. The output of the feature extraction module is connected to the input of the point cloud generation module.

[0096] So, in Figure 4 Based on the network model structure shown, such as Figure 5 As shown, step S202 above may include the following steps S202-1 to S202-3:

[0097] S202-1, Input the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image into the feature extraction module to obtain the high-dimensional feature vector corresponding to each view image output by the feature extraction module.

[0098] S202-2, input the high-dimensional feature vector corresponding to each viewpoint image into the point cloud generation module to obtain the point cloud data corresponding to the first training multi-viewpoint image output by the point cloud generation module.

[0099] S202-3, based on the point cloud data corresponding to the first training multi-view image and the coordinates of each pixel in each view image, train the scene coordinate regression network to be trained to obtain the scene coordinate regression network.

[0100] The aforementioned feature extraction module can be a feature extraction network used to extract high-dimensional semantic features from each image in the first training multi-view image input.

[0101] The aforementioned point cloud generation module can be a point cloud generation network used to generate point cloud data based on the high-dimensional semantic features output by the feature extraction module.

[0102] In this application, the point cloud generation module can be selected as a multi-layer perceptron (MLP). Then, after the MLP receives the high-dimensional semantic features output by the feature extraction module, it can construct a nonlinear mapping model from the feature space to three-dimensional coordinates to obtain point cloud data corresponding to the first training multi-view image.

[0103] In some alternative embodiments, the first loss value corresponding to the scene coordinate regression network to be trained can be determined by using formula (1) in the foregoing embodiments, based on the point cloud data corresponding to the first training multi-view image and the coordinates of each pixel in each view image.

[0104] Subsequently, this application can determine whether the first loss value is less than a first threshold. If the first loss value is greater than or equal to the first threshold, it indicates that the prediction effect of the scene coordinate regression network to be trained is poor. In this case, the scene coordinate regression network to be trained can be trained based on the first loss value to obtain the scene coordinate regression network. If the first loss value is less than the first threshold, it indicates that the prediction effect of the scene coordinate regression network to be trained is good. In this case, the training operation of the scene coordinate regression network to be trained can be ended, and the scene coordinate regression network to be trained can be used as the final scene coordinate regression network.

[0105] In some optional embodiments, the above-mentioned training of the scene coordinate regression network to be trained based on the first loss value to obtain the scene coordinate regression network specifically involves back-training the scene coordinate regression network to be trained based on the first loss value to adjust the network parameters in the scene coordinate regression network to be trained. That is, updating the weights and other parameters in the scene coordinate regression network to be trained.

[0106] Optionally, the above-mentioned scene coordinate regression network to be trained is back-trained based on the first loss value to adjust the network parameters in the scene coordinate regression network to be trained, as follows:

[0107] Based on the first loss value, the scene coordinate regression network to be trained is back-trained, and the new first loss value is determined again to be less than the first threshold. If the new first loss value is still greater than the first threshold, this step is repeated until the training stopping condition is met. If the latest first loss value is less than the first threshold, it means that the scene coordinate regression network after the latest training can predict dense and accurate point cloud data. At this time, the scene coordinate regression network after the latest training is taken as the final scene coordinate regression network.

[0108] In this application, the training stopping condition can be that the first loss value is less than a first threshold, or the number of training iterations is greater than a preset number of iterations.

[0109] The first threshold and the preset number of times are both adjustable parameters, which can be flexibly set according to the prediction accuracy of the scene coordinate regression network. No restrictions are imposed on them here.

[0110] In other words, when a scene coordinate regression network with high prediction accuracy is required, this application can set the first threshold to be relatively small or the preset number of times to be relatively large; conversely, the first threshold can be set to be relatively large or the preset number of times can be set to be relatively small. The specific setting method can be flexibly set according to the training requirements of the scene coordinate regression network.

[0111] This application trains a scene coordinate regression network based on a first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image, so that the scene coordinate regression network can efficiently and accurately learn geometric information, thereby quickly and accurately generating point cloud data corresponding to the input multiple view images and high-dimensional feature vectors corresponding to each view image, providing favorable conditions for generating a second target view image corresponding to the first target view image.

[0112] Next, the training process of the prediction head network involved in the embodiments of this application will be described. It should be noted that this application requires the use of the previously trained scene coordinate regression network when training the prediction head network. That is, the network parameters in the scene coordinate regression network remain unchanged during the training of the prediction head network. In other words, the network parameters in the prediction head network can be modified during the training process, while the network parameters in the scene coordinate regression network remain frozen and will not be modified.

[0113] Figure 6 A schematic diagram of a training scenario for a prediction head network provided in an embodiment of this application; Figure 7 This is a flowchart illustrating the training process of a prediction head network, as provided in an embodiment of this application. Figure 7 As shown, the method may include the following steps:

[0114] S301, Obtain the second training multi-view image and the camera parameters corresponding to each view image in the second training multi-view image.

[0115] S302, input the second training multi-view image and the camera parameters corresponding to each view image in the second training multi-view image into the scene coordinate regression network to obtain the point cloud data output by the scene coordinate regression network and the high-dimensional feature vector corresponding to each view image in the second training multi-view image.

[0116] S303, based on the second training multi-view image, point cloud data and each high-dimensional feature vector, train the prediction head network to be trained to obtain the prediction head network.

[0117] In this application, the prediction head network to be trained refers to the prediction head network in the network model whose network parameters have not been adjusted. That is, the prediction head network in its initial state.

[0118] It should be understood that the aforementioned prediction head network refers to a deep learning model that uses point cloud data and high-dimensional feature vectors output by a scene coordinate regression network, as well as multi-view images, to obtain the three-dimensional scene representation parameters used to generate a second target view image.

[0119] In some embodiments, this application may obtain a training dataset for training the prediction head network to be trained. The training dataset may include multiple viewpoint images and camera parameters corresponding to each viewpoint image. The training dataset may be obtained using the same method as the first training multi-view image and each viewpoint image within the first training multi-view image; alternatively, the first training multi-view image may be used as the training dataset to achieve data reuse, improve data utilization, and reduce data acquisition costs.

[0120] In some alternative embodiments, considering that the scene coordinate regression network is a pre-trained model when the prediction head network to be trained is trained, this application can input the second training multi-view image and the camera parameters corresponding to each view image in the second training multi-view image into the pre-trained scene coordinate regression network, so that the scene coordinate regression network can output point cloud data and a high-dimensional feature vector corresponding to each view image based on the second training multi-view image and the camera parameters corresponding to each view image.

[0121] Next, the second training multi-view images, point cloud data, and the high-dimensional feature vectors corresponding to each view image are concatenated. The concatenated result is then input into the prediction head network to be trained. The prediction head network processes the concatenated result to output predicted 3D scene representation parameters. The number of predicted 3D scene representations is the same as the number of view images in the second training multi-view images. That is, each view image in the second training multi-view images corresponds to one predicted 3D scene representation parameter.

[0122] Next, each predicted 3D scene representation parameter output by the training prediction head network is rendered, and each rendering result is used as a predicted second target view image. Then, the difference between each predicted second target view image and the corresponding real first target view image is determined, and a second loss value is determined based on each difference. The training prediction head network is then trained based on the second loss value to obtain the prediction head network.

[0123] Here, predicting the second target viewpoint image refers to predicting the virtual target viewpoint image; the real first target viewpoint image refers to the different viewpoint images located in the second training multi-viewpoint image.

[0124] In some alternative embodiments, the determination of the second loss value based on the difference between each predicted second target view image and the corresponding real first target view image can be achieved by the following formula (2):

[0125]

[0126] Where Loss2 is the second loss value; i is the i-th image in the second training multi-view image; M is the M-th image in the second training multi-view image, where M is a positive integer greater than or equal to 2; MSE() is the mean squared error loss function; For the i-th predicted second target view image; and Let be the i-th real first-view image of the target.

[0127] In some optional embodiments, after determining the second loss value, this application can determine whether the second loss value is less than a second threshold. If the second loss value is greater than or equal to the second threshold, it indicates that the prediction effect of the prediction head network to be trained is poor. In this case, the prediction head network to be trained can be trained based on the second loss value to obtain the prediction head network. If the second loss value is less than the second threshold, it indicates that the prediction effect of the prediction head network to be trained is good. In this case, the training operation of the prediction head network to be trained can be terminated, and the prediction head network to be trained can be used as the final prediction head network.

[0128] The second threshold mentioned above is an adjustable parameter, which can be set according to the training accuracy requirements of the prediction head network model. There are no restrictions on it here.

[0129] In some optional embodiments, the training prediction head network is performed based on the second loss value. Specifically, the network parameters in the training prediction head network are adjusted in reverse according to the second loss value, such as adjusting the weights and other parameters. That is, the weights and other parameters in the training prediction head network are updated.

[0130] Optionally, the network parameters in the prediction head network to be trained are adjusted in reverse according to the second loss value, as follows:

[0131] Based on the second loss value, the prediction head network to be trained is back-trained, and the new second loss value is determined again to be less than the second threshold. If the new second loss value is still greater than the second threshold, this step is repeated until the training stopping condition is met. If the latest second loss value is less than the second threshold, it means that the prediction head network after the latest training can predict a predicted second target view image that is pixel-level aligned with the real first target view image. At this time, the prediction head network after the latest training is taken as the final prediction head network.

[0132] In this application, the training stopping condition can be that the second loss value is less than the second threshold, or that the number of training iterations is greater than the preset number of iterations.

[0133] The second threshold and the preset number of times are both adjustable parameters, which can be flexibly set according to the prediction accuracy of the prediction head network. No restrictions are imposed on them here.

[0134] In other words, when a prediction head network with high prediction accuracy is required, this application can set the second threshold to be relatively small or the preset number of times to be relatively large; conversely, the second threshold can be set to be relatively large or the preset number of times to be relatively small. The specific setting method can be flexibly set according to the training requirements of the prediction head network.

[0135] This application embodiment trains a prediction head network based on a second training multi-view image and the camera parameters corresponding to each view image in the second training multi-view image. This enables the prediction head network to output parameters for generating a high-quality 3D scene representation, used to generate the second target view image. Furthermore, by employing the above training method, this application can significantly shorten the training time of the prediction head network and improve the training efficiency of the network model.

[0136] After a detailed explanation of the training process of the scene coordinate regression network and prediction head network involved in the embodiments of this application, the following will be combined with... Figure 8 and Figure 9 The process of generating a second target view image corresponding to the first target view image through a scene coordinate regression network and a prediction head network is explained in detail.

[0137] Figure 8 This is a schematic diagram illustrating how a second target view image is generated based on a scene coordinate regression network and a prediction head network, as provided in an embodiment of this application. Figure 9 This is a flowchart illustrating another image generation method provided in this application embodiment. Specifically, this embodiment mainly introduces the process of generating a second target view image based on a first target view image using the scene coordinate regression network and prediction head network trained above.

[0138] S401, acquire the first target view image and the camera parameters corresponding to the first target view image.

[0139] S402, the first target view image and the camera parameters corresponding to the first target view image are input into the scene coordinate regression network to obtain the point cloud data and high-dimensional feature vector corresponding to the first target view image output by the scene coordinate regression network; wherein the scene coordinate regression network is trained based on multiple images of the same scene under different viewpoints.

[0140] It should be understood that the above-mentioned scene coordinate regression network is the network model obtained after training the scene coordinate regression network to be trained in the aforementioned embodiments.

[0141] In some alternative embodiments, the acquired first target view image and the camera parameters corresponding to the first target view image can be input into the scene coordinate regression network, so that the scene coordinate regression network can quickly and accurately output the point cloud data and high-dimensional feature vector corresponding to the first target view image based on the first target view image and the camera parameters corresponding to the first target view image.

[0142] S403, the first target view image, point cloud data and high-dimensional feature vector are input into the prediction head network to obtain the three-dimensional scene representation parameters output by the prediction head network; wherein the prediction head network is trained based on the scene coordinate regression network.

[0143] S404 renders the 3D scene representation parameters to obtain a second target view image.

[0144] It should be understood that the aforementioned prediction head network is the network model obtained after training the prediction head network to be trained in the foregoing embodiments.

[0145] Furthermore, the prediction head network can include multiple layers of convolutional layers, enabling the prediction head network to acquire rich information at different levels to achieve pixel-level prediction tasks.

[0146] In some embodiments, the point cloud data and high-dimensional feature vector output by the scene coordinate regression network, along with the obtained first target view image, can be stitched together, and the stitched result can be used as input data to the prediction head network. This allows the prediction head network to predict other Gaussian parameters besides coordinate information based on the stitched result, specifically opacity information, covariance matrix, and spherical harmonic function coefficients, using multiple layers of convolutional layers.

[0147] Subsequently, based on the shooting perspective of the first target view image, the 3D scene representation parameters output by the prediction head network can be rendered to obtain the second target view image. In this way, the generated second target view image can achieve pixel-level alignment with the corresponding first target view image, resulting in extremely high rendering quality of the second target view image.

[0148] The technical solution disclosed in this application involves acquiring a first target view image and the corresponding camera parameters. Based on these parameters, point cloud data and a high-dimensional feature vector corresponding to the first target view image are determined. Then, based on the first target view image, point cloud data, and high-dimensional feature vector, 3D scene representation parameters are determined. Finally, the 3D scene representation parameters are rendered to obtain a second target view image. This allows the generated virtual image to be precisely anchored to fixed coordinates within the virtual scene without requiring any geometric priors, thus achieving cross-view scene representation. Figure 1 Scale information.

[0149] In some embodiments, such as Figure 4 As shown, the scene coordinate regression network of this application may include: a feature extraction module and a point cloud generation module. See below for details. Figure 10 ,right Figure 9 Step S402 will be further explained. For example... Figure 10 As shown, step S402 above may include the following steps S402-1 to S402-2:

[0150] S402-1, Input the first target view image and the camera parameters corresponding to the first target view image into the feature extraction module to obtain the high-dimensional feature vector corresponding to the first target view image output by the feature extraction module.

[0151] S402-2, input the high-dimensional feature vector corresponding to the first target view image into the point cloud generation module to obtain the point cloud data corresponding to the first target view image output by the point cloud generation module.

[0152] In some optional embodiments, considering that determining the 3D scene representation parameters based on the first target viewpoint image and the point cloud data and high-dimensional feature vector output by the scene coordinate regression network may result in problems such as checkerboard artifacts, i.e., visual distortion, this application, after obtaining the point cloud data and high-dimensional feature vector corresponding to the first target viewpoint image, may optionally further include: determining the upsampled point cloud data corresponding to the point cloud data, and the upsampled high-dimensional feature vector corresponding to the high-dimensional feature vector. Then, the 3D scene representation parameters can be determined based on the first target viewpoint image, the upsampled point cloud data, and the upsampled high-dimensional feature vector, thus avoiding visual distortion while maintaining computational efficiency, thereby improving the generation quality of the second target viewpoint image.

[0153] In some embodiments, the determination of the upsampled point cloud data corresponding to the point cloud data and the upsampled high-dimensional feature vector corresponding to the high-dimensional feature vector can be optionally performed as follows: inputting the point cloud data into an upsampling network to obtain the upsampled point cloud data output by the upsampling network, and inputting the high-dimensional feature vector into the upsampling network to obtain the upsampled high-dimensional feature vector output by the upsampling network, such as... Figure 11 As shown.

[0154] In this application, the upsampling network can be pre-trained or trained synchronously during the training of the prediction head network; no specific restrictions are imposed here. Furthermore, the training process for the upsampling network can be found in existing upsampling network training procedures, which will not be elaborated upon here.

[0155] This application employs an upsampling network to upsample the point cloud data and high-dimensional feature vectors output by the scene coordinate regression network. This process progressively upsamples the lower-resolution point cloud data and the corresponding images of the high-dimensional feature vectors to the same size as the higher-resolution first target view image, thereby achieving dimensionality unification for different image types. This enables pixel-level dense prediction tasks based on dimensionally unified images. Specifically, the different image types refer to the point cloud image corresponding to the point cloud data, the high-dimensional feature image corresponding to the high-dimensional feature vectors, and the first target view image.

[0156] In some embodiments, such as Figure 12 As shown, the upsampling network of this application may include an interpolation module and an optimization module. The input of the interpolation module is connected to the output vector of the scene coordinate regression network, and the output of the interpolation module is connected to the input of the optimization module.

[0157] So, in Figure 12Based on the network model structure shown, the aforementioned steps of inputting point cloud data into the upsampling network to obtain upsampled point cloud data output by the upsampling network, and inputting high-dimensional feature vectors into the upsampling network to obtain upsampled high-dimensional feature vectors output by the upsampling network, can include the following:

[0158] Step S1: Input the point cloud data into the interpolation module to obtain the processed point cloud data output by the interpolation module. Input the processed point cloud data into the optimization module to obtain the upsampled point cloud data output by the optimization module.

[0159] Step S2: Input the high-dimensional feature vector into the interpolation module to obtain the processed high-dimensional feature vector output by the interpolation module. Input the processed high-dimensional feature vector into the optimization module to obtain the upsampled high-dimensional feature vector corresponding to the high-dimensional feature vector output by the optimization module.

[0160] The interpolation module mentioned above can be a difference module that implements the interpolation function, and can be selected as a bilinear difference module or a bilinear difference layer.

[0161] The optimization module mentioned above can be a convolutional layer.

[0162] In other words, the upsampling network in this application employs a combination of bilinear interpolation and convolutional layers to implement an upsampling strategy. Compared to the traditional transposed convolution method, this upsampling strategy effectively avoids the checkerboard artifact problem while maintaining computational efficiency. Furthermore, it can effectively fill sparse regions in point cloud data and high-dimensional feature vectors, while preserving the continuity of the geometric structure.

[0163] The following is a reference to the appendix. Figure 13 This application describes an image generation apparatus based on an embodiment. Figure 13 As shown, the image generation device 500 includes: an image acquisition module 510, a first determination module 520, a second determination module 530, and a generation module 540.

[0164] The image acquisition module 510 is used to acquire a first target view image and the camera parameters corresponding to the first target view image.

[0165] The first determining module 520 is used to determine the point cloud data and high-dimensional feature vector corresponding to the first target view image based on the first target view image and the camera parameters corresponding to the first target view image.

[0166] The second determining module 530 is used to determine the three-dimensional scene representation parameters based on the first target view image, the point cloud data, and the high-dimensional feature vector;

[0167] The generation module 540 is used to render the three-dimensional scene representation parameters to obtain a second target view image.

[0168] In one optional implementation of this application embodiment, the first determining module 520 is specifically used for:

[0169] The first target view image and the camera parameters corresponding to the first target view image are input into the scene coordinate regression network to obtain the point cloud data and high-dimensional feature vector corresponding to the first target view image output by the scene coordinate regression network; wherein the scene coordinate regression network is trained based on multiple images of the same scene under different viewpoints.

[0170] In one optional implementation of this application embodiment, the scene coordinate regression network includes: a feature extraction module and a point cloud generation module, wherein the first determining module 520 is further configured to:

[0171] The first target view image and the camera parameters corresponding to the first target view image are input into the feature extraction module to obtain the high-dimensional feature vector corresponding to the first target view image output by the feature extraction module.

[0172] The high-dimensional feature vector corresponding to the first target view image is input into the point cloud generation module to obtain the point cloud data corresponding to the first target view image output by the point cloud generation module.

[0173] In one optional implementation of this application embodiment, the second determining module 530 is specifically used for:

[0174] The first target view image, the point cloud data, and the high-dimensional feature vector are input into the prediction head network to obtain the three-dimensional scene representation parameters output by the prediction head network; wherein the prediction head network is trained based on the scene coordinate regression network.

[0175] In one optional implementation of this application, the prediction head network includes multiple layers of convolutional layers.

[0176] In one optional implementation of this application embodiment, the image generating apparatus 500 further includes:

[0177] The sampling module is used to determine the upsampled point cloud data corresponding to the point cloud data, and the upsampled high-dimensional feature vector corresponding to the high-dimensional feature vector.

[0178] In one optional implementation of this application embodiment, the sampling module is specifically used for:

[0179] The point cloud data is input into the upsampling network to obtain the upsampled point cloud data output by the upsampling network, and the high-dimensional feature vector is input into the upsampling network to obtain the upsampled high-dimensional feature vector output by the upsampling network.

[0180] In one optional implementation of this application embodiment, the upsampling network includes: an interpolation module and an optimization module, wherein the sampling module is further configured to:

[0181] The point cloud data is input into the interpolation module to obtain the processed point cloud data output by the interpolation module. The processed point cloud data is then input into the optimization module to obtain the upsampled point cloud data output by the optimization module.

[0182] The high-dimensional feature vector is input into the interpolation module to obtain the processed high-dimensional feature vector output by the interpolation module. The processed high-dimensional feature vector is then input into the optimization module to obtain the upsampled high-dimensional feature vector output by the optimization module.

[0183] In an optional implementation of this application embodiment, the second determining module 530 is further configured to:

[0184] The three-dimensional scene representation parameters are determined based on the first target view image, the upsampled point cloud data, and the upsampled high-dimensional feature vector.

[0185] In one optional implementation of this application embodiment, the image generating apparatus 500 further includes:

[0186] The first acquisition module is used to acquire the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image.

[0187] The first training module is used to train the scene coordinate regression network to be trained based on the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image.

[0188] In one optional implementation of this application embodiment, the image generating apparatus 500 further includes:

[0189] The second acquisition module is used to acquire the second training multi-view image and the camera parameters corresponding to each view image in the second training multi-view image;

[0190] The processing module is used to input the second training multi-view image and the camera parameters corresponding to each view image in the second training multi-view image into the scene coordinate regression network to obtain the point cloud data output by the scene coordinate regression network and the high-dimensional feature vector corresponding to each view image in the second training multi-view image.

[0191] The second training module is used to train the prediction head network to be trained based on the second training multi-view image, the point cloud data, and each of the high-dimensional feature vectors.

[0192] It should be understood that the device embodiments and the foregoing method embodiments can correspond to each other, and similar descriptions can be found in the first aspect of the method embodiments. To avoid repetition, further details are omitted here. Specifically, Figure 13 The device 500 shown can perform Figure 1 The corresponding method embodiments, and the foregoing and other operations and / or functions of each module in device 500 are respectively implemented to achieve Figure 1 For the sake of brevity, the corresponding processes in each method are not described in detail here.

[0193] The apparatus 500 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the first aspect method embodiment in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the first aspect method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the first aspect method embodiment described above.

[0194] Figure 14 This is a schematic block diagram of an electronic device provided as an embodiment of this application. Figure 14 As shown, the electronic device 600 may include a memory 610 and a processor 620. The memory 610 stores computer programs and transmits the program code to the processor 620. In other words, the processor 620 can call and run the computer program from the memory 610 to implement the image generation method described in the first aspect of the embodiments of this application.

[0195] For example, the processor 620 can be used to execute the image generation method described in the first aspect above according to instructions in the computer program.

[0196] In some embodiments of this application, the processor 620 may include, but is not limited to:

[0197] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0198] In some embodiments of this application, the memory 610 includes, but is not limited to:

[0199] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0200] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 610 and executed by the processor 620 to complete the image generation method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device.

[0201] like Figure 14 As shown, the electronic device 600 may further include:

[0202] Transceiver 630, which can be connected to processor 620 or memory 610.

[0203] The processor 620 can control the transceiver 630 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 630 may include a transmitter and a receiver. The transceiver 630 may further include antennas, and the number of antennas may be one or more.

[0204] It should be understood that the various components in the electronic device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0205] This application also provides a computer storage medium having a computer program stored thereon, which, when executed by a computer, enables the computer to perform the image generation method described in the first aspect above.

[0206] This application also provides a computer program product containing program instructions that, when executed on an electronic device, cause the electronic device to perform the image generation method described in the first aspect above.

[0207] When implemented using software, it can be implemented entirely or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0208] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0209] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0210] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0211] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0212] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image generation method, characterized in that, include: Acquire the first target view image and the camera parameters corresponding to the first target view image; Based on the first target view image and the camera parameters corresponding to the first target view image, determine the point cloud data and high-dimensional feature vector corresponding to the first target view image; The three-dimensional scene representation parameters are determined based on the first target view image, the point cloud data, and the high-dimensional feature vector; The three-dimensional scene representation parameters are rendered to obtain a second target view image.

2. The method according to claim 1, characterized in that, The step of determining the point cloud data and high-dimensional feature vector corresponding to the first target view image based on the first target view image and the camera parameters corresponding to the first target view image includes: The first target view image and the camera parameters corresponding to the first target view image are input into the scene coordinate regression network to obtain the point cloud data and high-dimensional feature vector corresponding to the first target view image output by the scene coordinate regression network; wherein the scene coordinate regression network is trained based on multiple images of the same scene under different viewpoints.

3. The method according to claim 2, characterized in that, The scene coordinate regression network includes a feature extraction module and a point cloud generation module. The step of inputting the first target view image and the corresponding camera parameters into the scene coordinate regression network to obtain the point cloud data and high-dimensional feature vector corresponding to the first target view image output by the scene coordinate regression network includes: The first target view image and the camera parameters corresponding to the first target view image are input into the feature extraction module to obtain the high-dimensional feature vector corresponding to the first target view image output by the feature extraction module. The high-dimensional feature vector corresponding to the first target view image is input into the point cloud generation module to obtain the point cloud data corresponding to the first target view image output by the point cloud generation module.

4. The method according to claim 1, characterized in that, The step of determining the 3D scene representation parameters based on the first target view image, the point cloud data, and the high-dimensional feature vector includes: The first target view image, the point cloud data, and the high-dimensional feature vector are input into the prediction head network to obtain the three-dimensional scene representation parameters output by the prediction head network; wherein the prediction head network is trained based on the scene coordinate regression network.

5. The method according to claim 4, characterized in that, The prediction head network includes multiple layers of convolutional layers.

6. The method according to claim 1, characterized in that, The method further includes: Determine the upsampled point cloud data corresponding to the point cloud data, and the upsampled high-dimensional feature vector corresponding to the high-dimensional feature vector.

7. The method according to claim 6, characterized in that, The step of determining the upsampled point cloud data corresponding to the point cloud data and the upsampled high-dimensional feature vector corresponding to the high-dimensional feature vector includes: The point cloud data is input into the upsampling network to obtain the upsampled point cloud data output by the upsampling network, and the high-dimensional feature vector is input into the upsampling network to obtain the upsampled high-dimensional feature vector output by the upsampling network.

8. The method according to claim 7, characterized in that, The upsampling network includes an interpolation module and an optimization module. The steps of inputting the point cloud data into the upsampling network to obtain the upsampled point cloud data output by the upsampling network, and inputting the high-dimensional feature vector into the upsampling network to obtain the upsampled high-dimensional feature vector output by the upsampling network, include: The point cloud data is input into the interpolation module to obtain the processed point cloud data output by the interpolation module. The processed point cloud data is then input into the optimization module to obtain the upsampled point cloud data output by the optimization module. The high-dimensional feature vector is input into the interpolation module to obtain the processed high-dimensional feature vector output by the interpolation module. The processed high-dimensional feature vector is then input into the optimization module to obtain the upsampled high-dimensional feature vector output by the optimization module.

9. The method according to any one of claims 6 to 8, characterized in that, The determination of the 3D scene representation parameters also includes: The three-dimensional scene representation parameters are determined based on the first target view image, the upsampled point cloud data, and the upsampled high-dimensional feature vector.

10. The method according to claim 1, characterized in that, The method further includes: Obtain the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image; Based on the first training multi-view image and the camera parameters corresponding to each view image in the first training multi-view image, the scene coordinate regression network is trained to obtain the scene coordinate regression network.

11. The method according to claim 4, characterized in that, The method further includes: Obtain the second training multi-view image and the camera parameters corresponding to each view image in the second training multi-view image; The second training multi-view image and the camera parameters corresponding to each view image in the second training multi-view image are input into the scene coordinate regression network to obtain the point cloud data output by the scene coordinate regression network and the high-dimensional feature vector corresponding to each view image in the second training multi-view image. The prediction head network is obtained by training the prediction head network based on the second training multi-view image, the point cloud data, and each of the high-dimensional feature vectors.

12. An image generation apparatus, characterized in that, include: An image acquisition module is used to acquire a first target view image and the camera parameters corresponding to the first target view image; The first determining module is used to determine the point cloud data and high-dimensional feature vector corresponding to the first target view image based on the first target view image and the camera parameters corresponding to the first target view image. The second determining module is used to determine the three-dimensional scene representation parameters based on the first target view image, the point cloud data, and the high-dimensional feature vector; The generation module is used to render the three-dimensional scene representation parameters to obtain a second target view image.

13. An electronic device, characterized in that, include: A processor and a memory, the memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 11.

15. A computer program product containing program instructions, characterized in that, When the program instructions are executed on the electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1 to 11.

Images