Image generation model training method and image generation method, device and medium

Abstract

The present disclosure relates to the technical field of computer, in particular to an image generation model training method, an image generation method, an image generation device, a medium and an electronic device. The training method comprises: inputting an input image sample into a to-be-trained model, obtaining an image feature vector corresponding to the input image sample, obtaining a depth feature vector corresponding to the input image sample, and fusing to obtain a multi-modal feature vector corresponding to the input image sample; determining positions of a plurality of pixels corresponding to a predicted target image, generating virtual rays from a camera focal point to the positions of the plurality of pixels, and sampling a plurality of points on each virtual ray; predicting color information of the plurality of pixels corresponding to the predicted target image; rendering the plurality of pixels to obtain the predicted target image; and updating neural network parameters of the to-be-trained model to obtain an image generation model. Through the technical scheme of the embodiment of the present disclosure, the problem of defects in the new view image rendered by the prior art can be solved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to image generation model training methods, image generation methods, image generation model training devices, image generation devices, computer-readable storage media, and electronic devices. Background Technology

[0002] With the rapid development of software and hardware, computer graphics is being applied more and more widely in various fields. For example, computer graphics is extensively used in areas such as holography, metaverse, and digital twins. In computer graphics, new perspective image rendering is a key research direction.

[0003] In related technologies, two-dimensional features of the input image can be obtained, and the model can be trained using these two-dimensional features to achieve new perspective image rendering. However, the methods in these technologies, which use two-dimensional features to train the model, result in a lack of three-dimensional information, leading to defects such as blurring and artifacts in the rendered new perspective images.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide an image generation model training method, an image generation model training device, a computer-readable storage medium, and an electronic device, which can solve the problem of defects in the new perspective images rendered in the prior art.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part by practice of this disclosure.

[0007] According to a first aspect of this disclosure, an image generation model training method is provided, comprising: acquiring input image samples and target image samples; wherein the input image samples and target image samples are images captured in the same scene using different camera poses, and the target image samples correspond to a target camera pose; inputting the input image samples into a model to be trained, acquiring image feature vectors corresponding to the input image samples, acquiring depth feature vectors corresponding to the input image samples, and fusing the image feature vectors and depth feature vectors to obtain a multimodal feature vector corresponding to the input image samples; determining the positions of multiple pixels corresponding to the predicted target image based on the target camera pose, generating virtual rays from the camera focus to the positions of each pixel, and sampling multiple points on each virtual ray; wherein the target camera pose corresponds to a camera focus; determining the color information of multiple pixels corresponding to the predicted target image based on the multiple points sampled from the virtual rays and the multimodal feature vectors; rendering the multiple pixels based on the color information corresponding to the multiple pixels to obtain the predicted target image; and updating the neural network parameters of the model to be trained based on the target image samples and the predicted target image to obtain an image generation model.

[0008] Optionally, based on the aforementioned scheme, the depth feature vector corresponding to the input image sample is obtained by: performing depth estimation on the input image sample to obtain a sparse depth feature vector corresponding to the input image sample; and determining the depth feature vector corresponding to the input image sample based on the sparse depth feature vector; wherein the depth feature vector corresponding to the input image sample is a dense depth feature vector.

[0009] Optionally, based on the aforementioned scheme, the color information of multiple pixels corresponding to the predicted target image is determined according to multiple points obtained by virtual ray upsampling and multimodal feature vectors, including: determining the pixel coordinate values ​​of the pixels; determining the pixel feature vector corresponding to the pixel according to the multimodal feature vectors; performing coordinate encoding on the spatial coordinate values ​​of the multiple points obtained by virtual ray upsampling to obtain multiple spatial coordinate encoding vectors; determining the color information and density information of multiple points according to the pixel feature vector corresponding to the pixel and the multiple spatial coordinate encoding vectors; and determining the color information of multiple pixels according to the color information and density information of multiple points.

[0010] Optionally, based on the aforementioned scheme, the virtual ray has a ray direction, and the color information of multiple pixels is determined according to the color information and density information of multiple points, including: performing volume rendering along the ray direction to obtain multiple candidate points according to the color information and density information of multiple points; and determining the color information of multiple pixels according to the multiple candidate points on the virtual ray.

[0011] Optionally, based on the aforementioned scheme, the multiple points obtained by virtual ray upsampling have spatial coordinate values. Determining the pixel coordinate value of a pixel includes: transforming the spatial coordinate values ​​of the multiple points obtained by virtual ray upsampling from the world coordinate system to the camera coordinate system to obtain the first candidate coordinate values ​​of the multiple points; transforming the first candidate coordinate values ​​of the multiple points from the camera coordinate system to the image coordinate system to obtain the second candidate coordinate values ​​of the pixel; and transforming the second candidate coordinate values ​​of the pixel from the image coordinate system to the pixel coordinate system to obtain the pixel coordinate value of the pixel.

[0012] According to a second aspect of this disclosure, an image generation method is provided, the method comprising: acquiring an input image and predicting a camera pose; wherein the predicted camera pose is the camera pose corresponding to a target image, and the input image and the target image are images captured in the same scene using different camera poses; inputting the input image into an image generation model to obtain a target image; wherein the image generation model is obtained by an image generation model training method as described in any of the above.

[0013] According to a third aspect of this disclosure, an image generation model training apparatus is provided. The apparatus includes: a sample data acquisition unit configured to acquire input image samples and target image samples; wherein the input image samples and target image samples are images captured in the same scene using different camera poses, and the target image samples correspond to a target camera pose; a feature vector acquisition unit configured to input the input image samples into a model to be trained, acquire image feature vectors corresponding to the input image samples, acquire depth feature vectors corresponding to the input image samples, and fuse the image feature vectors and depth feature vectors to obtain multimodal feature vectors corresponding to the input image samples; and a virtual ray generation unit configured to perform a virtual ray generation based on the target image sample. The system comprises a camera pose determination unit, a color information acquisition unit, and a color information rendering unit. The color information acquisition unit is configured to determine the color information of the multiple pixels corresponding to the predicted target image based on the multiple points obtained from the virtual ray upsampling and the multimodal feature vector. The predicted image rendering unit is configured to render the multiple pixels based on the color information corresponding to the multiple pixels to obtain the predicted target image. The network parameter update unit is configured to update the neural network parameters of the training model based on the target image samples and the predicted target image to obtain the image generation model.

[0014] Optionally, based on the aforementioned scheme, to obtain the depth feature vector corresponding to the input image sample, the device further includes: a sparse depth feature vector acquisition unit, configured to perform depth estimation on the input image sample to obtain the sparse depth feature vector corresponding to the input image sample; and a depth feature vector acquisition unit, configured to determine the depth feature vector corresponding to the input image sample based on the sparse depth feature vector; wherein the depth feature vector corresponding to the input image sample is a dense depth feature vector.

[0015] Optionally, based on the aforementioned scheme, the device further includes: a pixel feature vector acquisition unit configured to determine the pixel coordinates of a pixel and the pixel feature vector corresponding to the pixel based on the multiple points obtained by upsampling the virtual ray and the multimodal feature vector; a spatial coordinate encoding vector acquisition unit configured to perform coordinate encoding on the spatial coordinates of the multiple points obtained by upsampling the virtual ray to obtain multiple spatial coordinate encoding vectors; a spatial coordinate encoding vector determination unit configured to determine the color information of multiple points and the density information of multiple points based on the pixel feature vector corresponding to the pixel and the multiple spatial coordinate encoding vectors; and a first color information determination unit configured to determine the color information of multiple pixels based on the color information of multiple points and the density information of multiple points.

[0016] Optionally, based on the aforementioned scheme, the virtual ray has a ray direction, and the color information of multiple pixels is determined based on the color information and density information of multiple points. The device further includes: a candidate point determination unit, configured to perform volume rendering along the ray direction based on the color information and density information of multiple points to obtain multiple candidate points; and a second color information determination unit, configured to perform color information determination of multiple pixels based on the multiple candidate points on the virtual ray.

[0017] Optionally, based on the aforementioned scheme, the multiple points obtained by virtual ray upsampling have spatial coordinate values. To determine the pixel coordinate values ​​of a pixel, the device further includes: a first candidate coordinate value determination unit, configured to perform the conversion of the spatial coordinate values ​​of the multiple points obtained by virtual ray upsampling from the world coordinate system to the camera coordinate system to obtain first candidate coordinate values ​​of the multiple points; a second candidate coordinate value determination unit, configured to perform the conversion of the first candidate coordinate values ​​of the multiple points from the camera coordinate system to the image coordinate system to obtain second candidate coordinate values ​​of the pixel; and a pixel coordinate value determination unit, configured to perform the conversion of the second candidate coordinate values ​​of the pixel from the image coordinate system to the pixel coordinate system to obtain the pixel coordinate values ​​of the pixel.

[0018] According to a fourth aspect of this disclosure, an image generation apparatus is provided, comprising: an input image acquisition unit configured to acquire an input image and predict a camera pose; wherein the predicted camera pose is a camera pose corresponding to a target image, and the input image and the target image are images captured in the same scene using different camera poses; and an image generation unit configured to input the input image into an image generation model to obtain a target image; wherein the image generation model is trained using an image generation model as described in any of the above.

[0019] According to a fifth aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the image generation model training method of the first aspect and the image generation method of the second aspect as described in the above embodiments.

[0020] According to a sixth aspect of this disclosure, an electronic device is provided, comprising:

[0021] Processor; and

[0022] A memory is used to store one or more programs that, when executed by one or more processors, enable the one or more processors to implement the image generation model training method of the first aspect and the image generation method of the second aspect as described in the above embodiments.

[0023] According to a seventh aspect of the present disclosure, a computer program product, a computer program / instruction, is provided, characterized in that when the computer program / instruction is executed by a processor, it implements the image generation model training method and the image generation method described above.

[0024] The technical solutions provided in this disclosure can include the following beneficial effects:

[0025] In one embodiment of the image generation model training method provided by this disclosure, input image samples and target image samples can be obtained. The input image samples are input into the model to be trained, and the image feature vectors and depth feature vectors corresponding to the input image samples are obtained. The image feature vectors and depth feature vectors are fused to obtain the multimodal feature vectors corresponding to the input image samples. Based on the target camera pose, the positions of multiple pixels corresponding to the predicted target image are determined. Virtual rays are generated from the camera focus to the positions of each pixel. Multiple points are sampled on each virtual ray. The color information of multiple pixels corresponding to the predicted target image is determined based on the multiple points sampled on the virtual ray and the multimodal feature vectors. The multiple pixels are rendered based on the color information corresponding to the multiple pixels to obtain the predicted target image. The neural network parameters of the model to be trained are updated based on the target image samples and the predicted target image to obtain the image generation model. In the embodiments of this disclosure, on the one hand, in the graphics generation model, the multimodal feature vector obtained by fusing image feature vectors and depth feature vectors can carry more information, enabling the reconstructed 3D scene to have viewpoint consistency; on the other hand, virtual rays can be generated to each pixel on the target image, and the color information of multiple pixels of the target image can be determined based on multiple points upsampled from the virtual rays, so as to realize the generation of the target image. At the same time, the model can be trained using two-dimensional and three-dimensional information, which can improve the accuracy of the generated target image and avoid defects such as blurring and artifacts in the target image.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0028] Figure 1 The illustration shows a schematic diagram of an exemplary system architecture for an image generation model training method in an exemplary embodiment of the present disclosure;

[0029] Figure 2 A flowchart illustrating an image generation model training method in an exemplary embodiment of this disclosure is shown schematically.

[0030] Figure 3 This schematically illustrates a flowchart of determining the depth feature vector corresponding to an input image sample based on a sparse depth feature vector in an exemplary embodiment of the present disclosure.

[0031] Figure 4 This schematically illustrates a flowchart of determining the color information of multiple pixels based on the color information of multiple points and the density information of multiple points in an exemplary embodiment of this disclosure;

[0032] Figure 5 This schematically illustrates a flowchart of determining color information of multiple pixels based on multiple candidate points on a virtual ray in an exemplary embodiment of this disclosure;

[0033] Figure 6 This schematically illustrates a flowchart of a process for converting the second candidate coordinate values ​​of a pixel from the image coordinate system to the pixel coordinate system to obtain the pixel coordinate values ​​of the pixel in an exemplary embodiment of this disclosure;

[0034] Figure 7 This schematically illustrates a flowchart of a process in an exemplary embodiment of the present disclosure, in which image feature vectors and depth feature vectors are fused to obtain multimodal feature vectors corresponding to input image samples.

[0035] Figure 8 This schematically illustrates a flowchart of sampling multiple points on each virtual ray in an exemplary embodiment of the present disclosure;

[0036] Figure 9 This schematically illustrates a flowchart of determining the color information and density information of multiple points based on the pixel feature vector corresponding to the pixel and multiple spatial coordinate encoding vectors in an exemplary embodiment of this disclosure;

[0037] Figure 10 This schematically illustrates a flowchart of rendering multiple pixels based on color information corresponding to multiple pixels in an exemplary embodiment of the present disclosure;

[0038] Figure 11 This schematic diagram illustrates an image generation system in an exemplary embodiment of the present disclosure.

[0039] Figure 12 This schematically illustrates a flowchart of an exemplary embodiment of the present disclosure in which an input image is input into an image generation model to obtain a target image;

[0040] Figure 13 This schematic diagram illustrates the composition of an image generation model training apparatus according to an exemplary embodiment of the present disclosure.

[0041] Figure 14 This schematic diagram illustrates the composition of an image generation apparatus according to an exemplary embodiment of the present disclosure;

[0042] Figure 15 The schematic diagram illustrates a structural schematic of a computer system suitable for implementing an electronic device according to exemplary embodiments of the present disclosure. Detailed Implementation

[0043] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The described feature vectors, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Furthermore, the described feature vectors, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known structures, methods, apparatuses, implementations, materials, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0044] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, or in one or more software-hardened modules, or in different network and / or processor devices and / or microcontroller devices.

[0045] Figure 1 A schematic diagram of an exemplary system architecture for an image generation model training method or an image generation method to which embodiments of the present disclosure can be applied is shown.

[0046] like Figure 1 As shown, system architecture 1000 may include one or more of terminal devices 1001, 1002, and 1003, network 1004, and server 1005. Network 1004 is used as a medium to provide a communication link between terminal devices 1001, 1002, and 1003 and server 1005. Network 1004 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0047] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included. For example, server 1005 could be a server cluster composed of multiple servers.

[0048] Users can use terminal devices 1001, 1002, and 1003 to interact with server 1005 via network 1004 to receive or send messages, etc. Terminal devices 1001, 1002, and 1003 can be various electronic devices with displays, including but not limited to smartphones, tablets, laptops, and desktop computers. Additionally, server 1005 can be a server providing various services.

[0049] In one embodiment, the execution entity of the image generation model training method of this disclosure can be a server 1005. The server 1005 can obtain input image samples and target image samples sent by terminal devices 1001, 1002, and 1003, input the input image samples into the model to be trained, obtain the image feature vector corresponding to the input image samples, obtain the depth feature vector corresponding to the input image samples, fuse the image feature vector and the depth feature vector to obtain the multimodal feature vector corresponding to the input image samples, determine the position of multiple pixels corresponding to the predicted target image based on the target camera pose, generate virtual rays from the camera focus to the position of each pixel, sample multiple points on each virtual ray, determine the color information of multiple pixels corresponding to the predicted target image based on the multiple points obtained from the virtual ray and the multimodal feature vector, render the multiple pixels according to the color information corresponding to the multiple pixels to obtain the predicted target image, and update the neural network parameters of the model to be trained according to the target image samples and the predicted target image to obtain the image generation model. Furthermore, the image generation model training method disclosed herein can be executed through terminal devices 1001, 1002, 1003, etc., to achieve the following: acquiring input image samples and target image samples; inputting the input image samples into the model to be trained; acquiring the image feature vector corresponding to the input image samples; acquiring the depth feature vector corresponding to the input image samples; fusing the image feature vector and the depth feature vector to obtain the multimodal feature vector corresponding to the input image samples; determining the positions of multiple pixels corresponding to the predicted target image based on the target camera pose; generating virtual rays from the camera focus to the positions of each pixel; sampling multiple points on each virtual ray; determining the color information of multiple pixels corresponding to the predicted target image based on the multiple points sampled from the virtual ray and the multimodal feature vector; rendering multiple pixels based on the color information corresponding to multiple pixels to obtain the predicted target image; and updating the neural network parameters of the model to be trained based on the target image samples and the predicted target image to obtain the image generation model.

[0050] Furthermore, the image generation model training method of this disclosure can also be implemented jointly by terminal devices 1001, 1002, 1003 and server 1005. For example, terminal devices 1001, 1002, and 1003 can input image samples and target image samples, and then send the acquired input image samples and target image samples to server 1005, so that server 1005 can input the input image samples into the model to be trained, obtain the image feature vector corresponding to the input image samples, obtain the depth feature vector corresponding to the input image samples, fuse the image feature vector and the depth feature vector to obtain the multimodal feature vector corresponding to the input image samples, determine the positions of multiple pixels corresponding to the predicted target image based on the target camera pose, generate virtual rays from the camera focus to the positions of each pixel, sample multiple points on each virtual ray, determine the color information of multiple pixels corresponding to the predicted target image based on the multiple points obtained from the virtual ray and the multimodal feature vector, render multiple pixels based on the color information corresponding to multiple pixels to obtain the predicted target image, and update the neural network parameters of the model to be trained based on the target image samples and the predicted target image to obtain the image generation model.

[0051] With the rapid development of software and hardware, computer graphics is being applied more and more widely in various fields. For example, computer graphics is extensively used in areas such as holography, metaverse, and digital twins. In computer graphics, new perspective image rendering is a key research direction.

[0052] In related technologies, two-dimensional features of the input image can be obtained, and the model can be trained using these two-dimensional features to achieve new perspective image rendering. However, the methods in these technologies, which use two-dimensional features to train the model, result in a lack of three-dimensional information, leading to defects such as blurring and artifacts in the rendered new perspective images.

[0053] According to the training method of the item classification model provided in this exemplary embodiment, input image samples and target image samples can be obtained. The input image samples are input into the model to be trained, and the image feature vectors and depth feature vectors corresponding to the input image samples are obtained. The image feature vectors and depth feature vectors are fused to obtain the multimodal feature vectors corresponding to the input image samples. Based on the target camera pose, the positions of multiple pixels corresponding to the predicted target image are determined. Virtual rays are generated from the camera focus to the positions of each pixel. Multiple points are sampled on each virtual ray. The color information of multiple pixels corresponding to the predicted target image is determined based on the multiple points sampled on the virtual ray and the multimodal feature vectors. The multiple pixels are rendered based on the color information to obtain the predicted target image. The neural network parameters of the model to be trained are updated based on the target image samples and the predicted target image to obtain the image generation model. Figure 2 As shown, the image generation model training method may include the following steps S210 to S260:

[0054] Step S210: Obtain input image samples and target image samples; wherein, the input image samples and target image samples are images captured in the same scene using different camera poses, and the target image samples correspond to the target camera pose;

[0055] Step S220: Input the input image sample into the model to be trained, obtain the image feature vector corresponding to the input image sample, obtain the depth feature vector corresponding to the input image sample, and fuse the image feature vector and the depth feature vector to obtain the multimodal feature vector corresponding to the input image sample.

[0056] Step S230: Based on the target camera pose, determine the positions of multiple pixels corresponding to the predicted target image, generate virtual rays from the camera focus to the positions of each pixel, and sample multiple points on each virtual ray; wherein, the target camera pose corresponds to the camera focus.

[0057] Step S240: Determine the color information of multiple pixels corresponding to the predicted target image based on multiple points obtained by upsampling the virtual ray and the multimodal feature vector;

[0058] Step S250: Render the multiple pixels according to the color information corresponding to the multiple pixels to obtain the predicted target image;

[0059] Step S260: Update the neural network parameters of the model to be trained based on the target image sample and the predicted target image to obtain the image generation model.

[0060] In one embodiment of the image generation model training method provided by this disclosure, input image samples and target image samples can be obtained. The input image samples are input into the model to be trained, and the image feature vectors and depth feature vectors corresponding to the input image samples are obtained. The image feature vectors and depth feature vectors are fused to obtain the multimodal feature vectors corresponding to the input image samples. Based on the target camera pose, the positions of multiple pixels corresponding to the predicted target image are determined. Virtual rays are generated from the camera focus to the positions of each pixel. Multiple points are sampled on each virtual ray. The color information of multiple pixels corresponding to the predicted target image is determined based on the multiple points sampled on the virtual ray and the multimodal feature vectors. The multiple pixels are rendered based on the color information corresponding to the multiple pixels to obtain the predicted target image. The neural network parameters of the model to be trained are updated based on the target image samples and the predicted target image to obtain the image generation model. In the embodiments of this disclosure, on the one hand, in the graphics generation model, the multimodal feature vector obtained by fusing image feature vectors and depth feature vectors can carry more information, enabling the reconstructed 3D scene to have viewpoint consistency; on the other hand, virtual rays can be generated to each pixel on the target image, and the color information of multiple pixels of the target image can be determined based on multiple points upsampled from the virtual rays, so as to realize the generation of the target image. At the same time, the model can be trained using two-dimensional and three-dimensional information, which can improve the accuracy of the generated target image and avoid defects such as blurring and artifacts in the target image.

[0061] The steps S210 to S260 of the image generation model training method in this exemplary embodiment will now be described in more detail with reference to the accompanying drawings and embodiments.

[0062] Step S210: Obtain input image samples and target image samples;

[0063] In one example embodiment of this disclosure, input image samples and target image samples can be obtained. The input image samples and target image samples are images captured in the same scene using different camera poses, and the target image sample corresponds to a target camera pose. Specifically, the input image samples and target image samples are images captured from the same subject at different shooting angles; the camera poses used are different when capturing images from different shooting angles. Specifically, the camera pose can include external camera parameters and internal camera parameters. External camera parameters can include camera position and camera angle, while internal camera parameters can include camera focal length and image size.

[0064] It should be noted that this disclosure does not impose any special restrictions on the specific parameter types for camera pose.

[0065] For example, an input image sample can be obtained by taking a picture of the subject using the input camera pose, and a target image sample can be obtained by taking a picture of the subject using the target camera pose.

[0066] It should be noted that this disclosure does not impose any special limitations on the generation methods of input image samples and target image samples.

[0067] In one exemplary embodiment of this disclosure, after acquiring the target image sample, the target camera pose corresponding to the target image sample can be obtained. For example, the target camera pose corresponding to the target image sample can be obtained using the SFM algorithm (structure from motion, recovering the 3D scene structure from motion information).

[0068] Step S220: Input the input image sample into the model to be trained, obtain the image feature vector corresponding to the input image sample, obtain the depth feature vector corresponding to the input image sample, and fuse the image feature vector and the depth feature vector to obtain the multimodal feature vector corresponding to the input image sample.

[0069] In one exemplary embodiment of this disclosure, after obtaining the input image sample and the target image sample through the above steps, and obtaining the target camera pose corresponding to the target image sample, the input image sample can be input into the model to be trained to obtain the image feature vector corresponding to the input image sample. Specifically, the image feature vector corresponding to the input image sample can be used to indicate the color of the input image sample, such as RGB (Red Green Blue) features.

[0070] It should be noted that this disclosure does not impose any special restrictions on the specific type of image feature vector corresponding to the input image sample or the specific method of obtaining the image feature vector corresponding to the input image sample.

[0071] In one exemplary embodiment of this disclosure, after inputting the input image sample into the model to be trained through the above steps, the depth feature vector corresponding to the input image sample can be obtained. Specifically, the depth feature vector corresponding to the input image sample can be used to indicate the distance between a point in the scene of the input image sample and the camera.

[0072] In one example embodiment of this disclosure, after obtaining the depth feature vector corresponding to the input image sample through the above steps, the depth feature vector (single channel) corresponding to the input image sample can be converted into a three-channel HSV (HueSaturation Value) format, and then input into a pre-trained convolutional neural network to extract the depth feature vector.

[0073] It should be noted that this disclosure does not impose any special limitations on the specific method for obtaining the depth feature vectors corresponding to the input image samples.

[0074] In one exemplary embodiment of this disclosure, after obtaining the image feature vector and the depth feature vector corresponding to the input image sample through the above steps, the image feature vector and the depth feature vector can be fused to obtain the multimodal feature vector corresponding to the input image sample. Specifically, the multimodal feature vector corresponding to the input image sample can be used to indicate the color of the input image sample and the distance between a point in the scene of the input image sample and the camera.

[0075] Furthermore, when fusing image feature vectors and depth feature vectors to obtain multimodal feature vectors corresponding to input image samples, an attention mechanism can be used for fusion. Image feature vectors provide semantic and texture information, while depth feature vectors provide spatial information. Since image feature vectors and depth feature vectors have different modalities, the convergence speed of the model under training differs. Therefore, using an attention mechanism to fuse image and depth feature vectors can combine the feature vectors from both modalities, thereby improving the training speed of the model.

[0076] In one example embodiment of this disclosure, the model to be trained refers to a model established to complete the image generation task. An image generation model can be obtained by training the model to be trained, thereby completing the image generation task. It should be noted that this disclosure does not impose any special limitations on the specific structure of the model to be trained.

[0077] The image generation task refers to taking an input image and a predicted camera pose as input, and generating a target image based on the predicted camera pose. The input image and the target image are images taken in the same scene using different camera poses.

[0078] It should be noted that this disclosure does not impose any special limitations on the specific method of fusing image feature vectors and depth feature vectors to obtain the multimodal feature vectors corresponding to the input image samples.

[0079] In one exemplary embodiment of this disclosure, feature learning for multi-camera pose can be performed using multimodal feature vectors corresponding to multiple input image samples. For example, the multimodal feature vectors corresponding to multiple input image samples can be input into a multi-view feature fusion module for feature learning of multi-camera pose.

[0080] Step S230: Based on the target camera pose, determine the positions of multiple pixels corresponding to the predicted target image, generate virtual rays from the camera focus to the position of each pixel, and sample multiple points on each virtual ray;

[0081] In one example embodiment of this disclosure, after determining the target camera pose corresponding to the target image sample through the above steps, the positions of multiple pixels corresponding to the predicted target image can be determined based on the target camera pose. Specifically, after determining the target camera pose, the shooting position corresponding to the predicted target image to be generated can be determined, and the positions of multiple pixels on the predicted target image can be determined accordingly.

[0082] It should be noted that this disclosure does not impose any special limitations on the specific method for determining the positions of multiple pixels corresponding to the predicted target image.

[0083] In one example embodiment of this disclosure, after determining the positions of multiple pixels corresponding to the predicted target image, virtual rays can be generated from the camera focus to the positions of each pixel, and multiple points can be sampled on each virtual ray. The target camera pose corresponds to a camera focus. Specifically, the target camera pose refers to the camera pose of a virtual camera in the virtual environment, which corresponds to a camera focus. Virtual rays can be emitted from the camera focus of the virtual camera to the multiple pixels corresponding to the predicted target image, and multiple points can be sampled on each virtual camera.

[0084] For example, if the length (number of pixels) and width (number of pixels) of the target image to be predicted are H and W respectively, then H*W virtual rays need to be emitted from the focus of the virtual camera to the multiple pixels corresponding to the target image to be predicted, and multiple points are sampled on each virtual camera.

[0085] Furthermore, when sampling multiple points on each virtual ray, multiple points can be sampled uniformly on the virtual ray; or, multiple points can be sampled non-uniformly on the virtual ray.

[0086] Step S240: Determine the color information of multiple pixels corresponding to the predicted target image based on multiple points obtained by upsampling the virtual ray and the multimodal feature vector;

[0087] In one exemplary embodiment of this disclosure, after generating virtual rays from the camera focus to the positions of each pixel through the above steps, sampling multiple points on each virtual ray, and fusing the image feature vector and depth feature vector to obtain the multimodal feature vector corresponding to the input image sample, the color information of multiple pixels corresponding to the predicted target image can be determined based on the multiple points sampled from the virtual ray and the multimodal feature vector. Specifically, the multiple points sampled from the virtual ray can be used to indicate three-dimensional information, and the multimodal feature vector can be used to indicate two-dimensional information. The color information of multiple pixels corresponding to the predicted target image can be determined by combining the three-dimensional information and the two-dimensional information.

[0088] Specifically, a virtual ray can correspond to a pixel on the target image. The color information of that pixel on the target image can be determined based on multiple points on the virtual ray and the multimodal feature vector. Similarly, the color information of multiple pixels corresponding to the target image can be obtained.

[0089] It should be noted that this disclosure does not impose any special limitations on the specific method for determining the color information of multiple pixels corresponding to the predicted target image based on multiple points obtained from virtual ray upsampling and multimodal feature vectors.

[0090] Step S250: Render the multiple pixels according to the color information corresponding to the multiple pixels to obtain the predicted target image;

[0091] In one example embodiment of this disclosure, after obtaining the color information of multiple pixels in the predicted target image through the above steps, the multiple pixels can be rendered according to the color information corresponding to the multiple pixels to obtain the predicted target image. Specifically, the color information of each pixel object can be used to render that pixel. After the multiple pixels are rendered, the predicted target image can be obtained. The predicted target image refers to the image captured based on the pose of the target camera.

[0092] Step S260: Update the neural network parameters of the model to be trained based on the target image sample and the predicted target image to obtain the image generation model.

[0093] In one exemplary embodiment of this disclosure, after obtaining the predicted target image through the above steps, the neural network parameters of the model to be trained can be updated based on the target image samples and the predicted target image to obtain an image generation model. Specifically, the model to be trained can be used to generate a target image based on the input image and the target camera position, where the target image refers to an image captured based on the target camera pose.

[0094] In one example embodiment of this disclosure, the model to be trained may include multiple hidden layers, which may include convolutional layers, normalization layers, activation layers, etc. The first fusion feature vector corresponding to the video sample data can be sequentially input into the multiple hidden layers of the model to be trained to obtain the hidden layer calculation results, and the predicted target image can be obtained through the hidden layer calculation results.

[0095] In one example embodiment of this disclosure, after obtaining the predicted target image through the above steps, the neural network parameters of the model to be trained can be updated based on the target image sample and the predicted target image to obtain an item classification model. Specifically, the predicted target image can be used to indicate a predicted image generated based on the target camera pose. The predicted target image is a predicted value, and at this time, the corresponding real value, i.e., the target image sample (label), can be obtained. This target image sample can be used to indicate the real image generated based on the target camera pose. At this time, the predicted target image (predicted value) can be compared with the target image sample (real value) to obtain the prediction difference between the predicted target image (predicted value) and the target image sample (real value), and the neural network parameters of the model to be trained can be updated based on the prediction difference to obtain an image generation model.

[0096] Specifically, the neural network parameters of the model to be trained may include the number of model layers, the number of feature vector channels, and the learning rate. When updating the neural network parameters of the model to be trained based on the prediction difference, the number of model layers, the number of feature vector channels, and the learning rate of the model to be trained can be updated to train the image generation model.

[0097] In one example embodiment of this disclosure, the neural network parameters of the model to be trained can be updated using the backpropagation algorithm, and an image generation model can be obtained after training.

[0098] It should be noted that this disclosure does not impose any special limitations on the specific method of updating the neural network parameters of the model to be trained based on the target image samples and the predicted target image.

[0099] In one exemplary embodiment of this disclosure, the neural network parameters of the model to be trained can be updated based on the target image sample and the predicted target image. When the model to be trained meets the convergence condition, it is determined to be an image generation model. Specifically, meeting the convergence condition means that the prediction accuracy of the model to be trained is high, and it can be applied. For example, the convergence condition may include the number of training iterations, such as ending training after the model to be trained N times; or the convergence condition may include the training duration, such as ending training after the model to be trained for a duration T.

[0100] It should be noted that this disclosure does not impose any special limitations on the specific content of the convergence conditions. By applying convergence conditions to the model, the training process of the model to be trained can be better controlled, the problem of overtraining of neural networks can be avoided, and the training efficiency of the model to be trained can be improved.

[0101] In one example embodiment of this disclosure, when training the model to be trained, the training scenarios in the training set can be increased to improve the generalization of the trained image generation model; and multiple input image samples can be used, each input image sample corresponding to an input image pose. Increasing the number of input image samples is beneficial to reconstructing a more realistic 3D scene and can provide more prior knowledge, thereby improving the accuracy of the trained image generation model.

[0102] In one exemplary embodiment of this disclosure, depth estimation can be performed on an input image sample to obtain a sparse depth feature vector corresponding to the input image sample, and the depth feature vector corresponding to the input image sample can be determined based on the sparse depth feature vector; wherein, the depth feature vector corresponding to the input image sample is a dense depth feature vector. (Refer to...) Figure 3 As shown, determining the depth feature vector corresponding to the input image sample based on the sparse depth feature vector may include the following steps S310 to S320:

[0103] Step S310: Perform depth estimation on the input image sample to obtain the sparse depth feature vector corresponding to the input image sample;

[0104] In one exemplary embodiment of this disclosure, after inputting the input image sample into the model to be trained through the above steps, depth estimation can be performed on the input image sample to obtain the sparse depth feature vector corresponding to the input image sample. Specifically, depth estimation refers to using the input image sample to estimate the distance of each pixel in the input image sample relative to the shooting source, which can be used to indicate the distance from the viewpoint to the object in the input image sample, and the viewpoint can be the camera that took the image.

[0105] The sparse depth feature vector corresponding to the input image sample refers to the vector in which most of the elements are equal to zero.

[0106] Furthermore, the sparse depth feature vector corresponding to the input image sample can be used to indicate the depth value of some pixels in the input image sample.

[0107] For example, depth estimation can be performed using a depth prediction method with a fully convolutional residual network to obtain sparse depth feature vectors corresponding to input image samples.

[0108] In one example embodiment of this disclosure, when performing depth estimation on an input image sample to obtain the sparse depth feature vector corresponding to the input image sample, the SFM algorithm can be used to perform sparse depth estimation to obtain the sparse depth feature vector corresponding to the input image sample.

[0109] It should be noted that this disclosure does not impose any special restrictions on the specific methods of depth estimation.

[0110] Step S320: Determine the depth feature vector corresponding to the input image sample based on the sparse depth feature vector;

[0111] In one exemplary embodiment of this disclosure, after obtaining the sparse depth feature vector corresponding to the input image sample through the above steps, the depth feature vector corresponding to the input image sample can be determined based on the sparse depth feature vector. Here, the depth feature vector corresponding to the input image sample is a dense depth feature vector. Specifically, depth padding can be performed on the sparse depth feature vector to obtain the depth feature vector corresponding to the input image sample. For example, dense depth estimation can be performed on the sparse depth feature vector to obtain the depth feature vector corresponding to the input image sample. In the depth feature vector (dense depth feature vector), a small portion of the stored vector elements are equal to zero.

[0112] Furthermore, the depth feature vector corresponding to the input image sample can be used to indicate the depth values ​​of multiple pixels in the input image sample.

[0113] It should be noted that this disclosure does not impose any special limitations on the specific method for determining the depth feature vector corresponding to the input image sample based on the sparse depth feature vector.

[0114] In one example embodiment of this disclosure, the sparse depth feature vector corresponding to the input image sample can be input into a pre-trained depth completion network to obtain the depth feature vector corresponding to the input image sample.

[0115] Through the steps S310-S320 described above, depth estimation can be performed on the input image samples to obtain the sparse depth feature vectors corresponding to the input image samples. Based on these sparse depth feature vectors, the depth feature vectors corresponding to the input image samples are determined. Through the embodiments of this disclosure, the sparse depth feature vectors of the input image samples can be converted into dense depth feature vectors, which improves the reference value of the depth feature vectors in model training, enhances the generalization ability of the model, and thus improves the accuracy of the generated target image.

[0116] In one example embodiment of this disclosure, the pixel coordinates of a pixel can be determined, the pixel feature vector corresponding to the pixel can be determined based on the multimodal feature vector, the spatial coordinates of multiple points obtained by upsampling from a virtual ray can be encoded to obtain multiple spatial coordinate encoded vectors, the color information and density information of multiple points can be determined based on the pixel feature vector corresponding to the pixel and the multiple spatial coordinate encoded vectors, and the color information of multiple pixels can be determined based on the color information and density information of multiple points. (Reference) Figure 4 As shown, determining the color information of multiple pixels based on the color information and density information of multiple points may include the following steps S410 to S440:

[0117] Step S410: Determine the pixel coordinates of the pixel and determine the pixel feature vector corresponding to the pixel based on the multimodal feature vector;

[0118] In one exemplary embodiment of this disclosure, after obtaining the multimodal feature vector corresponding to the input image sample through the above steps and sampling multiple points on each virtual ray, the pixel coordinate values ​​of the pixels can be determined. Specifically, in the above steps, the position of each pixel on the predicted target image has been determined, so the pixel coordinate values ​​of each pixel can be determined, and multiple pixels can be distinguished by the pixel coordinate values ​​of each pixel.

[0119] In one exemplary embodiment of this disclosure, after determining the pixel coordinates of each pixel, the pixel feature vector corresponding to each pixel can be determined based on the multimodal feature vector. Specifically, the multimodal feature vector can be used to indicate the color of the input image sample and the distance between a point in the scene of the input image sample and the camera. After determining the pixel coordinates of each pixel, the pixel feature vector corresponding to each pixel can be determined by combining the multimodal feature vector of the input image sample and the pixel coordinates of the pixels.

[0120] In one example embodiment of this disclosure, after obtaining the pixel coordinates and multimodal feature vectors of a pixel, the pixel feature vector corresponding to the pixel can be obtained by the bilinear interpolation method.

[0121] It should be noted that this disclosure does not impose any special limitations on the specific method for determining the pixel feature vector corresponding to a pixel based on the multimodal feature vector.

[0122] Step S420: Coordinate encoding is performed on the spatial coordinate values ​​of multiple points obtained by upsampling the virtual ray to obtain multiple spatial coordinate encoding vectors;

[0123] In one exemplary embodiment of this disclosure, after sampling multiple points on a virtual ray through the above steps, the spatial coordinate values ​​of the multiple points sampled from the virtual ray can be encoded to obtain multiple spatial coordinate encoding vectors. Specifically, after encoding the spatial coordinate values ​​of the multiple points sampled from the virtual ray, the spatial positions of the multiple points can be indicated by the multiple spatial coordinate encoding vectors, and the spatial positions of the multiple points can reflect the high-dimensional information of the pixels corresponding to the multiple points.

[0124] It should be noted that this disclosure does not impose any special limitations on the specific method of obtaining multiple spatial coordinate encoded vectors by encoding the spatial coordinate values ​​of multiple points obtained by upsampling virtual rays.

[0125] Step S430: Determine the color information and density information of multiple points based on the pixel feature vector corresponding to the pixel and multiple spatial coordinate encoding vectors;

[0126] In one exemplary embodiment of this disclosure, after obtaining the pixel feature vector and multiple spatial coordinate encoding vectors corresponding to a pixel through the above steps, the color information and density information of multiple points can be determined based on the pixel feature vector and multiple spatial coordinate encoding vectors. Specifically, the pixel feature vector corresponding to a pixel can be used to indicate the pixel information (e.g., position) of that pixel, and the spatial coordinate encoding vector can be used to indicate the spatial position of multiple points corresponding to the pixel. The color information and density information of each point on the virtual ray can be determined using the pixel feature vector corresponding to the pixel and the multiple spatial coordinate encoding vectors corresponding to the virtual ray. The color information of a point can be used to indicate the color of the point, and the density information of the point can be used to indicate the distance between each point on the virtual ray and the pixel.

[0127] It should be noted that this disclosure does not impose any special limitations on the specific method for determining the color information and density information of multiple points based on the pixel feature vector corresponding to the pixel and multiple spatial coordinate encoding vectors.

[0128] Step S440: Determine the color information of multiple pixels based on the color information and density information of multiple points.

[0129] In one exemplary embodiment of this disclosure, after obtaining the color information and density information of multiple points through the above steps, the color information of multiple pixels can be determined based on the color information and density information of the multiple points. Specifically, the color information of the pixels corresponding to the virtual rays corresponding to these points is determined, and similarly, the color information of the pixels corresponding to other virtual rays can be obtained, that is, the color information of multiple pixels is determined based on the color information and density information of multiple points. Specifically, each virtual ray corresponds to one pixel on the predicted target image, and multiple points on the virtual ray can be mapped to that pixel, and the color information of that pixel can be determined.

[0130] It should be noted that this disclosure does not impose any special limitations on the specific method for determining the color information of multiple pixels based on the color information and density information of multiple points.

[0131] Through the above steps S410-S440, the pixel coordinates of a pixel can be determined. The pixel feature vector corresponding to the pixel is determined based on the multimodal feature vector. Multiple spatial coordinate encoding vectors are obtained by coordinate encoding the spatial coordinates of multiple points sampled from the virtual ray. The color information and density information of multiple points are determined based on the pixel feature vector corresponding to the pixel and the multiple spatial coordinate encoding vectors. Finally, the color information of multiple pixels is determined based on the color information and density information of the multiple points. Through the embodiments of this disclosure, the color information of pixels corresponding to a virtual ray can be determined using the color and density information of multiple points on the virtual ray. Introducing three-dimensional information through multiple points on the virtual ray can improve the accuracy of the generated target image.

[0132] In one example embodiment of this disclosure, multiple candidate points can be obtained by volume rendering along a ray direction based on the color information and density information of multiple points, and the color information of multiple pixels can be determined based on the multiple candidate points on the virtual ray. (Refer to...) Figure 5 As shown, determining the color information of multiple pixels based on multiple candidate points on a virtual ray may include the following steps S510 to S520:

[0133] Step S510: Based on the color information and density information of multiple points, perform volume rendering along the ray direction to obtain multiple candidate points;

[0134] In one exemplary embodiment of this disclosure, after obtaining the color information and density information of multiple points corresponding to each virtual ray through the above steps, volume rendering can be performed along the ray direction based on the color information and density information of the multiple points to obtain multiple candidate points. Specifically, the ray direction can be the direction from the focal point to the pixel position, or the ray direction can be the direction from the pixel position to the focal point. Volume rendering can be performed sequentially on multiple points on the virtual ray along the ray direction to obtain multiple candidate points. The candidate points can be used to indicate the rendering status of multiple points on the virtual ray corresponding to the pixel.

[0135] For example, if there are n points on each virtual ray, n calculations are needed to determine n candidate points.

[0136] Step S520: Determine the color information of multiple pixels based on multiple candidate points on the virtual ray.

[0137] In one exemplary embodiment of this disclosure, after rendering candidate points from multiple points on each virtual ray through the above steps, the color information of multiple pixels can be determined based on the multiple candidate points on the virtual ray. Specifically, the multiple candidate points on the virtual ray can be superimposed, and the color information of the pixel corresponding to the virtual ray can be obtained based on the pixel feature vector corresponding to the pixel. The color information of the pixel can be used to indicate the color to be presented by the pixel.

[0138] For example, if the size (number of pixels) of the target image to be predicted is (H, W), the color information of one pixel can be determined each time based on multiple candidate points on the virtual ray. Similarly, it takes H*W times to determine the color information of multiple pixels.

[0139] Specifically, the color information of a pixel can be determined by the following expression, where C is the color information of the pixel, r is the virtual ray, d is the ray direction, c is the color information of the points on the virtual ray, σ is the density information of the points on the virtual ray, tn is the nearest point on the ray direction of the virtual ray, tf is the farthest point on the ray direction of the virtual ray, and T(t) is the cumulative transmittance to a point on the ray direction of the virtual ray:

[0140]

[0141] Through the above steps S510-S520, multiple candidate points can be obtained by volume rendering along the ray direction based on the color information and density information of multiple points. The color information of multiple pixels is then determined based on these candidate points on the virtual ray. Through the embodiments of this disclosure, multiple points on a virtual ray can be volume rendered, and the color information of pixels can be determined based on the multiple candidate points after volume rendering. This allows for the determination of the final color information of the pixels to be displayed based on the display status of the multiple candidate points, thereby improving the accuracy of the generated target image.

[0142] In one exemplary embodiment of this disclosure, the spatial coordinate values ​​of multiple points obtained by upsampling from a virtual ray can be transformed from the world coordinate system to the camera coordinate system to obtain first candidate coordinate values ​​of the multiple points. The first candidate coordinate values ​​of the multiple points can then be transformed from the camera coordinate system to the image coordinate system to obtain second candidate coordinate values ​​of the pixels. Finally, the second candidate coordinate values ​​of the pixels can be transformed from the image coordinate system to the pixel coordinate system to obtain the pixel coordinate values ​​of the pixels. (Refer to...) Figure 6 As shown, transforming the second candidate coordinate value of a pixel from the image coordinate system to the pixel coordinate system to obtain the pixel coordinate value may include the following steps S610 to S630:

[0143] Step S610: Transform the spatial coordinate values ​​of multiple points obtained by upsampling the virtual ray from the world coordinate system to the camera coordinate system to obtain the first candidate coordinate values ​​of multiple points;

[0144] Step S620: Transform the first candidate coordinate values ​​of multiple points from the camera coordinate system to the image coordinate system to obtain the second candidate coordinate values ​​of the pixels;

[0145] Step S630: Transform the second candidate coordinate value of the pixel from the image coordinate system to the pixel coordinate system to obtain the pixel coordinate value of the pixel.

[0146] In one exemplary embodiment of this disclosure, after obtaining the spatial coordinate values ​​of multiple points obtained by virtual ray upsampling through the above steps, the spatial coordinate values ​​of the multiple points obtained by virtual ray upsampling can first be transformed from the world coordinate system to the camera coordinate system to obtain the first candidate coordinate values ​​of the multiple points. Then, the first candidate coordinate values ​​of the multiple points can be transformed from the camera coordinate system to the image coordinate system to obtain the second candidate coordinate values ​​of the pixel. Finally, the second candidate coordinate values ​​of the pixel can be transformed from the image coordinate system to the pixel coordinate system to obtain the pixel coordinate value of the pixel. Specifically, the world coordinate system refers to a three-dimensional coordinate system with the object being photographed as the origin, the camera coordinate system refers to a three-dimensional coordinate system with the camera as the origin when the image is captured, the image coordinate system refers to a two-dimensional coordinate system with the center point of the image as the origin, and the pixel coordinate system refers to a two-dimensional coordinate system with a specific pixel in the image (e.g., the top-left pixel) as the origin.

[0147] Specifically, multiple points on a virtual ray can be mapped to a single pixel in a plane. Based on this, the spatial coordinates of the multiple points sampled from the virtual ray are transformed from the world coordinate system to the pixel coordinate system, and the pixel coordinates of that pixel are determined.

[0148] Specifically, the spatial coordinates of multiple points can be transformed from the world coordinate system to the pixel coordinate system by the target camera pose, and the pixel coordinates of that pixel can be determined.

[0149] Through the above steps S610 to S630, the spatial coordinate values ​​of multiple points obtained by virtual ray upsampling can be transformed from the world coordinate system to the camera coordinate system to obtain the first candidate coordinate values ​​of multiple points. The first candidate coordinate values ​​of multiple points can be transformed from the camera coordinate system to the image coordinate system to obtain the second candidate coordinate values ​​of pixels. The second candidate coordinate values ​​of pixels can be transformed from the image coordinate system to the pixel coordinate system to obtain the pixel coordinate values ​​of pixels.

[0150] In one exemplary embodiment of this disclosure, such as Figure 7The diagram shows a flowchart for fusing image feature vectors and depth feature vectors to obtain a multimodal feature vector corresponding to an input image sample. Specifically, S710, obtain the image feature vector corresponding to the input image sample; S720, perform depth estimation on the input image sample to obtain a sparse depth feature vector corresponding to the input image sample; S730, determine the depth feature vector corresponding to the input image sample based on the sparse depth feature vector; S740, fuse the image feature vector and the depth feature vector to obtain a multimodal feature vector corresponding to the input image sample; S750, fuse the multimodal feature vectors.

[0151] In one exemplary embodiment of this disclosure, such as Figure 8 The diagram shows a flowchart for sampling multiple points on each virtual ray. Specifically, in step S810, input image samples and target image samples are acquired; in step S820, the target camera pose corresponding to the target image sample is obtained through the SFM algorithm; in step S830, virtual rays are generated from the camera focus to the position of each pixel; and in step S840, multiple points are sampled on each virtual ray.

[0152] In one exemplary embodiment of this disclosure, such as Figure 9 The diagram shows a flowchart for determining the color information and density information of multiple points based on the pixel feature vector corresponding to each pixel and multiple spatial coordinate encoding vectors. Specifically, in step S910, the spatial coordinate values ​​of multiple points obtained by upsampling from the virtual ray are transformed from the world coordinate system to the pixel coordinate system; in step S920, the pixel feature vector corresponding to each pixel is determined by bilinear interpolation; in step S930, the spatial coordinate values ​​of the multiple points obtained by upsampling from the virtual ray are encoded to obtain multiple spatial coordinate encoding vectors; and in step S940, the color information and density information of multiple points are determined based on the pixel feature vector corresponding to each pixel and the multiple spatial coordinate encoding vectors.

[0153] In one exemplary embodiment of this disclosure, such as Figure 10 The diagram illustrates a flowchart for rendering multiple pixels based on their corresponding color information to obtain a predicted target image. Specifically, in step S1010, color information and density information of multiple points are acquired; in step S1020, volume rendering is performed along a ray direction based on the color and density information of the multiple points to obtain multiple candidate points, and the color information of multiple pixels is determined based on the multiple candidate points on the virtual ray; in step S1030, the multiple pixels are rendered based on their corresponding color information to obtain the predicted target image.

[0154] In one exemplary embodiment of this disclosure, such as Figure 11 The diagram shown is a schematic of an image generation system, which includes an input preprocessing module, a depth extraction module, and a multimodal fusion module.

[0155] Specifically, the input preprocessing module includes a camera pose estimation unit, a 3D virtual ray generation unit, and a 3D spatial point sampling unit. Specifically: the camera pose estimation unit is used to obtain the target camera pose corresponding to the target image sample; the 3D reconstruction unit is used to obtain the target camera pose corresponding to the target image sample using the SFM algorithm; the 3D virtual ray generation unit is used to generate virtual rays from the camera focus to the positions of each pixel; and the 3D spatial point sampling unit is used to sample multiple points on each virtual ray.

[0156] Specifically, the depth extraction module includes a sparse depth extraction unit, a depth feature extraction unit, a depth completion unit, an image feature extraction unit, a multimodal attention unit, a multi-view feature fusion unit, and a target view query unit. Specifically: the sparse depth extraction unit performs depth estimation on the input image samples to obtain the corresponding sparse depth map; the depth feature extraction unit extracts the corresponding sparse depth feature vectors of the input image samples; and the depth completion unit determines the corresponding depth feature vector of the input image samples based on the sparse depth feature vectors.

[0157] Specifically, the multimodal fusion module includes an image feature extraction unit, a multimodal attention unit, a multi-view feature fusion unit, and a target view query unit. Specifically: the image feature extraction unit obtains the image feature vector corresponding to the input image sample; the multimodal attention unit fuses the image feature vector with the depth feature vector to obtain the multimodal feature vector corresponding to the input image sample; the multi-view feature fusion unit performs multi-camera pose feature learning using the multimodal feature vectors corresponding to multiple input image samples, and determines the pixel feature vector corresponding to each pixel based on the multimodal feature vectors; the target view query unit searches for the features corresponding to the target camera position.

[0158] The image generation system also includes: a neural radiation field unit, used to determine the color information and density information of multiple points based on the pixel feature vector corresponding to the pixel and multiple spatial coordinate encoding vectors; and a three-dimensional volume rendering unit, used to perform volume rendering along the ray direction based on the color information and density information of multiple points to obtain multiple candidate points.

[0159] In one example embodiment of this disclosure, an input image and a predicted camera pose can be acquired, and the input image can be fed into an image generation model to obtain a target image; wherein, the image generation model is obtained through the image generation model training method described above. (Refer to...) Figure 12 As shown, inputting the input image into the image generation model to obtain the target image may include the following steps S1210 to S1220:

[0160] Step S1210: Obtain the input image and predict the camera pose;

[0161] Step S1220: Input the input image into the image generation model to obtain the target image.

[0162] In one example embodiment of this disclosure, an input image and a predicted camera pose can be acquired. The predicted camera pose is the camera pose corresponding to the target image, and the input image and the target image are images captured in the same scene using different camera poses. Specifically, the target image can be generated in an image generation model by inputting the input image and the predicted camera pose, and the camera pose corresponding to this target image is the predicted camera pose.

[0163] For example, if the subject is a refrigerator, the input image is an image taken directly in front of the refrigerator, and the predicted phase pose is the left side of the refrigerator, a model can be generated from the input image and the predicted camera pose input image to obtain the target image, which is an image taken on the left side of the refrigerator.

[0164] For example, when determining the predicted camera pose, the predicted camera pose can be generated by rotating the virtual camera, adjusting the virtual camera depth of field, controlling the virtual camera spiral, and moving the virtual camera left and right.

[0165] It should be noted that this disclosure does not impose any special restrictions on the specific type of camera pose prediction.

[0166] Through the above steps S1210 to S1220, video data can be obtained, and the video data can be input into the item classification model to obtain the item category.

[0167] In one embodiment of the image generation model training method provided by this disclosure, input image samples and target image samples can be obtained. The input image samples are input into the model to be trained, and the image feature vectors and depth feature vectors corresponding to the input image samples are obtained. The image feature vectors and depth feature vectors are fused to obtain the multimodal feature vectors corresponding to the input image samples. Based on the target camera pose, the positions of multiple pixels corresponding to the predicted target image are determined. Virtual rays are generated from the camera focus to the positions of each pixel. Multiple points are sampled on each virtual ray. The color information of multiple pixels corresponding to the predicted target image is determined based on the multiple points sampled on the virtual ray and the multimodal feature vectors. The multiple pixels are rendered based on the color information corresponding to the multiple pixels to obtain the predicted target image. The neural network parameters of the model to be trained are updated based on the target image samples and the predicted target image to obtain the image generation model. In the embodiments of this disclosure, on the one hand, in the graphics generation model, the multimodal feature vector obtained by fusing image feature vectors and depth feature vectors can carry more information, enabling the reconstructed 3D scene to have viewpoint consistency; on the other hand, virtual rays can be generated to each pixel on the target image, and the color information of multiple pixels of the target image can be determined based on multiple points upsampled from the virtual rays, so as to realize the generation of the target image. At the same time, the model can be trained using two-dimensional and three-dimensional information, which can improve the accuracy of the generated target image and avoid defects such as blurring and artifacts in the target image.

[0168] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0169] Furthermore, in an exemplary embodiment of this disclosure, an image generation model training apparatus is also provided. (Refer to...) Figure 13 As shown, an image generation model training device 1300 includes: a sample data acquisition unit 1310, a feature vector acquisition unit 1320, a virtual ray generation unit 1330, a color information acquisition unit 1340, a predicted image rendering unit 1350, and a network parameter update unit 1360.

[0170] The sample data acquisition unit is configured to acquire input image samples and target image samples; wherein the input image samples and target image samples are images captured in the same scene using different camera poses, and the target image sample corresponds to the target camera pose; the feature vector acquisition unit is configured to input the input image samples into the model to be trained, acquire the image feature vector corresponding to the input image samples, acquire the depth feature vector corresponding to the input image samples, and fuse the image feature vector and the depth feature vector to obtain the multimodal feature vector corresponding to the input image samples; the virtual ray generation unit is configured to determine the predicted target image pair based on the target camera pose. The system generates virtual rays from the camera focus to the positions of multiple pixels corresponding to the target camera pose, and samples multiple points on each virtual ray. The target camera pose corresponds to the camera focus. The color information acquisition unit is configured to determine the color information of multiple pixels corresponding to the target image based on the multiple points obtained from the virtual ray sampling and the multimodal feature vector. The predicted image rendering unit is configured to render multiple pixels based on the color information corresponding to the multiple pixels to obtain the predicted target image. The network parameter update unit is configured to update the neural network parameters of the model to be trained based on the target image samples and the predicted target image to obtain the image generation model.

[0171] Optionally, based on the aforementioned scheme, to obtain the depth feature vector corresponding to the input image sample, the device further includes: a sparse depth feature vector acquisition unit, configured to perform depth estimation on the input image sample to obtain the sparse depth feature vector corresponding to the input image sample; and a depth feature vector acquisition unit, configured to determine the depth feature vector corresponding to the input image sample based on the sparse depth feature vector; wherein the depth feature vector corresponding to the input image sample is a dense depth feature vector.

[0172] Optionally, based on the aforementioned scheme, the device further includes: a pixel feature vector acquisition unit configured to determine the pixel coordinates of a pixel and the pixel feature vector corresponding to the pixel based on the multiple points obtained by upsampling the virtual ray and the multimodal feature vector; a spatial coordinate encoding vector acquisition unit configured to perform coordinate encoding on the spatial coordinates of the multiple points obtained by upsampling the virtual ray to obtain multiple spatial coordinate encoding vectors; a spatial coordinate encoding vector determination unit configured to determine the color information of multiple points and the density information of multiple points based on the pixel feature vector corresponding to the pixel and the multiple spatial coordinate encoding vectors; and a first color information determination unit configured to determine the color information of multiple pixels based on the color information of multiple points and the density information of multiple points.

[0173] Optionally, based on the aforementioned scheme, the virtual ray has a ray direction, and the color information of multiple pixels is determined based on the color information and density information of multiple points. The device further includes: a candidate point determination unit, configured to perform volume rendering along the ray direction based on the color information and density information of multiple points to obtain multiple candidate points; and a second color information determination unit, configured to perform color information determination of multiple pixels based on the multiple candidate points on the virtual ray.

[0174] Optionally, based on the aforementioned scheme, the multiple points obtained by virtual ray upsampling have spatial coordinate values. To determine the pixel coordinate values ​​of a pixel, the device further includes: a first candidate coordinate value determination unit, configured to perform the conversion of the spatial coordinate values ​​of the multiple points obtained by virtual ray upsampling from the world coordinate system to the camera coordinate system to obtain first candidate coordinate values ​​of the multiple points; a second candidate coordinate value determination unit, configured to perform the conversion of the first candidate coordinate values ​​of the multiple points from the camera coordinate system to the image coordinate system to obtain second candidate coordinate values ​​of the pixel; and a pixel coordinate value determination unit, configured to perform the conversion of the second candidate coordinate values ​​of the pixel from the image coordinate system to the pixel coordinate system to obtain the pixel coordinate values ​​of the pixel.

[0175] Since the functional modules of the image generation model training device in the example embodiments of this disclosure correspond to the steps of the above-described example embodiments of the image generation model training method, for details not disclosed in the device embodiments of this disclosure, please refer to the above-described embodiments of the image generation model training method of this disclosure.

[0176] Furthermore, in an exemplary embodiment of this disclosure, an image generation model training apparatus is also provided. (Refer to...) Figure 14 As shown, an image generation apparatus 1400 includes an input image acquisition unit 1410 and an image generation unit 1420.

[0177] The input image acquisition unit is configured to acquire an input image and predict the camera pose; wherein the predicted camera pose is the camera pose corresponding to the target image, and the input image and the target image are images captured in the same scene using different camera poses; the image generation unit is configured to input the input image into an image generation model to obtain the target image; wherein the image generation model is trained using an image generation model as described above.

[0178] Since the functional modules of the image generation apparatus in the example embodiments of this disclosure correspond to the steps of the example embodiments of the image generation method described above, for details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the image generation method described above.

[0179] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the feature vectors and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the feature vectors and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0180] Furthermore, in an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described image generation model training method is also provided.

[0181] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be embodied in the following forms: a completely hardware embodiment, a completely software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0182] The following reference Figure 15 To describe an electronic device 1500 according to such an embodiment of the present disclosure. Figure 15 The electronic device 1500 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0183] like Figure 15 As shown, the electronic device 1500 is manifested in the form of a general-purpose computing device. The components of the electronic device 1500 may include, but are not limited to: at least one processing unit 1510, at least one storage unit 1520, a bus 1530 connecting different system components (including storage unit 1520 and processing unit 1510), and a display unit 1540.

[0184] The storage unit stores program code, which can be executed by the processing unit 1510 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1510 can perform actions such as... Figure 2Step S210, as shown, involves acquiring input image samples and target image samples; wherein the input image samples and target image samples are images captured in the same scene using different camera poses, and the target image sample corresponds to the target camera pose; Step S220, the input image samples are input into the model to be trained, and the image feature vector and depth feature vector corresponding to the input image samples are acquired, and the image feature vector and depth feature vector are fused to obtain the multimodal feature vector corresponding to the input image samples; Step S230, based on the target camera pose, the positions of multiple pixels corresponding to the predicted target image are determined, and virtual rays are generated from the camera focus to the positions of each pixel, and multiple points are sampled on each virtual ray; wherein the target camera pose corresponds to the camera focus; Step S240, the color information of multiple pixels corresponding to the predicted target image is determined according to the multiple points sampled from the virtual rays and the multimodal feature vector; Step S250, the multiple pixels are rendered according to the color information corresponding to the multiple pixels to obtain the predicted target image; Step S260, the neural network parameters of the model to be trained are updated according to the target image samples and the predicted target image to obtain the image generation model.

[0185] Alternatively, it can also perform such as Figure 12 In step S1210, the input image is acquired and the camera pose is predicted; in step S1220, the input image is input into the image generation model to obtain the target image.

[0186] For example, electronic devices can achieve such Figure 2 and Figure 12 The steps shown.

[0187] Storage unit 1520 may include readable media in the form of volatile storage units, such as random access memory (RAM) 1521 and / or cache memory 1522, and may further include read-only memory (ROM) 1523.

[0188] Storage unit 1520 may also include a program / utility 1524 having a set (at least one) program module 1525, such program module 1525 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0189] Bus 1530 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0190] Electronic device 1500 can also communicate with one or more external devices 1570 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 1500, and / or with any device that enables electronic device 1500 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1550. Furthermore, electronic device 1500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1560. As shown, network adapter 1560 communicates with other modules of electronic device 1500 via bus 1530. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0191] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0192] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions that can be executed by a processor of the device to perform the described method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0193] In an exemplary embodiment, a computer program product is also provided, including a computer program / instructions, which, when executed by a processor, implement the image generation model training method or image generation method described above.

[0194] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0195] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for training an image generation model, characterized in that, The method includes: Obtain input image samples and target image samples; wherein, the input image samples and the target image samples are images captured in the same scene using different camera poses, and the target image samples correspond to the target camera pose; The input image sample is input into the model to be trained, and the image feature vector corresponding to the input image sample and the depth feature vector corresponding to the input image sample are obtained. An attention mechanism is used to fuse the image feature vector and the depth feature vector to obtain the multimodal feature vector corresponding to the input image sample. The multimodal feature vector is used to indicate the color of the input image sample and the distance between a point in the scene of the input image sample and the camera. Based on the target camera pose, the positions of multiple pixels corresponding to the predicted target image are determined, and a virtual ray is generated from the camera focus to the position of each pixel. Multiple points are sampled on each virtual ray; wherein, the target camera pose corresponds to a camera focus. The color information of multiple pixels corresponding to the predicted target image is determined based on multiple points obtained by upsampling the virtual ray and the multimodal feature vector. The multiple pixels are rendered based on the color information corresponding to the multiple pixels to obtain the predicted target image; The neural network parameters of the model to be trained are updated based on the target image sample and the predicted target image to obtain the image generation model.

2. The method according to claim 1, characterized in that, The step of obtaining the depth feature vector corresponding to the input image sample includes: Depth estimation is performed on the input image sample to obtain the sparse depth feature vector corresponding to the input image sample; The depth feature vector corresponding to the input image sample is determined based on the sparse depth feature vector; wherein, the depth feature vector corresponding to the input image sample is a dense depth feature vector.

3. The method according to claim 1, characterized in that, The step of determining the color information of multiple pixels corresponding to the predicted target image based on multiple points obtained by upsampling the virtual ray and the multimodal feature vector includes: Determine the pixel coordinates of the pixel, and determine the pixel feature vector corresponding to the pixel based on the multimodal feature vector; The spatial coordinate values ​​of multiple points obtained by upsampling the virtual ray are encoded to obtain multiple spatial coordinate encoded vectors; The color information and density information of the plurality of points are determined based on the pixel feature vector corresponding to the pixel and the plurality of spatial coordinate encoding vectors; The color information of the plurality of pixels is determined based on the color information and density information of the plurality of points.

4. The method according to claim 3, characterized in that, The virtual ray has a ray direction, and determining the color information of the plurality of pixels based on the color information and density information of the plurality of points includes: Based on the color information and density information of the multiple points, volume rendering is performed along the ray direction to obtain multiple candidate points; The color information of a plurality of pixels is determined based on the plurality of candidate points on the virtual ray.

5. The method according to claim 3, characterized in that, The multiple points obtained by upsampling the virtual ray have spatial coordinate values, and determining the pixel coordinate value of the pixel includes: The spatial coordinate values ​​of multiple points obtained by upsampling the virtual ray are transformed from the world coordinate system to the camera coordinate system to obtain the first candidate coordinate values ​​of multiple points; The first candidate coordinate values ​​of the plurality of points are transformed from the camera coordinate system to the image coordinate system to obtain the second candidate coordinate values ​​of the pixel; The pixel coordinates are obtained by transforming the second candidate coordinates of the pixel from the image coordinate system to the pixel coordinate system.

6. An image generation method, characterized in that, The method includes: The input image and the predicted camera pose are obtained; wherein the predicted camera pose is the camera pose corresponding to the target image, and the input image and the target image are images captured in the same scene using different camera poses; The input image is input into the image generation model to obtain the target image; wherein the image generation model is trained by the image generation model training method as described in any one of claims 1-5.

7. An image generation model training device, characterized in that, include: The sample data acquisition unit is configured to acquire input image samples and target image samples; wherein the input image samples and the target image samples are images captured in the same scene using different camera poses, and the target image samples correspond to a target camera pose; The feature vector acquisition unit is configured to input the input image sample into the model to be trained, acquire the image feature vector corresponding to the input image sample, acquire the depth feature vector corresponding to the input image sample, and fuse the image feature vector and the depth feature vector using an attention mechanism to obtain the multimodal feature vector corresponding to the input image sample; wherein, the multimodal feature vector is used to indicate the color of the input image sample and the distance between a point in the scene of the input image sample and the camera; The virtual ray generation unit is configured to perform the following actions: based on the target camera pose, determine the positions of multiple pixels corresponding to the predicted target image, generate virtual rays from the camera focus to the positions of each pixel, and sample multiple points on each virtual ray; wherein the target camera pose corresponds to a camera focus. The color information acquisition unit is configured to determine the color information of multiple pixels corresponding to the predicted target image based on multiple points obtained by upsampling the virtual ray and the multimodal feature vector; The predictive image rendering unit is configured to render the plurality of pixels according to the color information corresponding to the plurality of pixels to obtain a predicted target image; The network parameter update unit is configured to update the neural network parameters of the model to be trained based on the target image sample and the predicted target image to obtain an image generation model.

8. An image generation apparatus, characterized in that, include: The input image acquisition unit is configured to acquire an input image and predict the camera pose; wherein the predicted camera pose is the camera pose corresponding to the target image, and the input image and the target image are images captured in the same scene using different camera poses; An image generation unit is configured to perform an operation of inputting the input image into an image generation model to obtain a target image; wherein the image generation model is trained using the image generation model training method as described in any one of claims 1-5.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the executable instructions to implement the image generation model training method as described in any one of claims 1 to 5 or the image generation method as described in claim 6.

10. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the image generation model training method as claimed in any one of claims 1 to 5 or the image generation method as claimed in claim 6.

Images