Image generation method, and deep learning model training method and device

By generating point cloud images and converting them into Gaussian points collection, new view synthesis is performed using 3D Gaussian sputtering method, the problem of insufficient universality and real-time rendering capabilities of new view synthesis in the existing technology is solved, and efficient new perspective images are achieved Generation and 3D scene reconstruction.

CN119048630APending Publication Date: 2024-11-29BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411246777.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-05
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

The existing new view synthesis technology relies on complex high-resolution image processing and lengthy optimization processes, limiting versatility and real-time rendering capabilities, and 3D scene reconstruction depends on paired reconstruction of image pairs, requiring real camera pose optimization, limiting Application in real-world scenarios.

Method used

By generating point cloud images corresponding to the input image and converting them into a Gaussian point collection, 3D scene reconstruction and new perspective image generation are used to perform three-dimensional scene reconstruction and new perspective image generation, implementing explicit point-based scene representation, supporting real-time rendering and Efficient optimization.

Benefits of technology

Improve image generation quality and efficiency, reduce scene limitations, and realize real-time rendering and efficient synthesis of new perspective images in real-world scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048630A_ABST
    Figure CN119048630A_ABST
Patent Text Reader

Abstract

The invention provides an image generation method, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to AIGC content generation and other scenes based on artificial intelligence. According to the specific implementation scheme, according to depth information of an input image, a point cloud image corresponding to the input image is generated, and the input image comprises a plurality of images of different visual angles for a target scene; according to the input image and the point cloud image, scene description of a new view angle for the target scene is generated, and the new view angle is a view angle different from the view angle of the input image; and generating a new image corresponding to the new view angle according to the scene description. The invention further provides an image generation device, electronic equipment and a storage medium.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to computer vision, deep learning, large models and other technical fields, and can be applied to scenarios such as AIGC content generation based on artificial intelligence. More specifically, the present disclosure provides an image generation method, device, electronic device, storage medium and computer program product. Background Art

[0002] With the rapid development of artificial intelligence, the application of AI-based image generation is becoming more and more abundant. For example, the new view synthesis method that generates new view images based on existing view images can obtain a new view and bring a new visual experience to users. Summary of the invention

[0003] The present disclosure provides an image generation method, an apparatus, an electronic device, a storage medium, and a computer program product.

[0004] According to a first aspect, an image generation method is provided, the method comprising: generating a point cloud image corresponding to the input image based on depth information of the input image, wherein the input image comprises a plurality of images of different perspectives for a target scene; generating a scene description of a new perspective for the target scene based on the input image and the point cloud image, wherein the new perspective is a perspective different from that of the input image; and generating a new image corresponding to the new perspective based on the scene description.

[0005] According to a second aspect, a training method for a deep learning model is provided, the method comprising: generating a point cloud image corresponding to the sample image based on depth information of the sample image, wherein the sample image includes multiple images of different perspectives for a sample scene; inputting the sample image and the point cloud image into the deep learning model to generate a set of Gaussian points for a target perspective of the sample scene, wherein the target perspective is a perspective different from that of the sample image; generating a target image corresponding to the target perspective based on the set of Gaussian points; determining the loss of the deep learning model based on the difference between the target image and a supervision image, wherein the supervision image is an image of the target perspective of the sample scene; and adjusting parameters of the deep learning model based on the loss.

[0006] According to a third aspect, an image generating device is provided, which includes: a first point cloud image generating module, used to generate a point cloud image corresponding to an input image based on depth information of the input image, wherein the input image includes multiple images of different perspectives for a target scene; a scene description generating module, used to generate a scene description of a new perspective for the target scene based on the input image and the point cloud image, wherein the new perspective is a perspective different from that of the input image; and a first image generating module, used to generate a new image corresponding to the new perspective based on the scene description.

[0007] According to a fourth aspect, a training device for a deep learning model is provided, the device comprising: a second point cloud image generation module, for generating a point cloud image corresponding to the sample image based on depth information of the sample image, wherein the sample image comprises a plurality of images at different perspectives of the sample scene; a Gaussian point generation module, for inputting the sample image and the point cloud image into the deep learning model, and generating a set of Gaussian points for a target perspective of the sample scene, wherein the target perspective is a perspective different from that of the sample image; a second image generation module, for generating a target image corresponding to the target perspective based on the set of Gaussian points; a loss determination module, for determining the loss of the deep learning model based on the difference between the target image and a supervision image, wherein the supervision image is an image of the target perspective of the sample scene; and an adjustment module, for adjusting the parameters of the deep learning model based on the loss.

[0008] According to a fifth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.

[0009] According to a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0010] According to a seventh aspect, a computer program product is provided, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and wherein the computer program implements the method provided according to the present disclosure when executed by a processor.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 is a schematic diagram of an exemplary system architecture to which an image generation method and a training method for a deep learning model can be applied according to an embodiment of the present disclosure;

[0014] Figure 2 is a flowchart of an image generating method according to an embodiment of the present disclosure;

[0015] Figure 3is a schematic diagram of an image generation method according to an embodiment of the present disclosure;

[0016] Figure 4 is a flowchart of a method for training a deep learning model according to an embodiment of the present disclosure;

[0017] Figure 5 is a comparison diagram of image generation effects according to an embodiment of the present disclosure;

[0018] Figure 6 is a block diagram of an image generating apparatus according to an embodiment of the present disclosure;

[0019] Figure 7 is a block diagram of a training apparatus for a deep learning model according to an embodiment of the present disclosure;

[0020] Figure 8 It is a block diagram of an electronic device according to at least one of the image generation method and the deep learning model training method of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] New view synthesis refers to generating new view images based on existing view images. For example, if the existing view image is the left view of a scene, the generated new view image may include the left view, right view and middle view of the scene.

[0023] Currently, new view synthesis often relies on the real camera pose, and the new view is synthesized by splicing images of consecutive frames according to the camera pose. This requires complex high-resolution image processing and lengthy optimization processes, which limits the versatility and real-time rendering capabilities.

[0024] A new view synthesis method based on Neural Radiance Field (NeRF) uses key point matching between consecutive frames to predict camera pose. Although the camera pose does not need to be known in advance, the implicit modeling inherent in NeRF makes it complicated to optimize both camera pose and scene simultaneously. Implicit modeling of NeRF refers to synthesizing new views directly based on input images without a representation of the displayed 3D scene. This also means that the NeRF model needs to be trained for different scenes in order to synthesize views in different scenes.

[0025] A new view synthesis method based on 3D Gaussian Splatting (3D-GS). By reconstructing the 3D scene of the input image, a new view is generated according to the Gaussian points in the reconstructed 3D scene. Compared with the new view synthesis method based on neural radiation field, the new view synthesis method based on 3D Gaussian Splatting provides an explicit point-based scene representation, realizing real-time rendering and efficient optimization. It can be applied in products or projects that require real-time 3D graphics rendering and computer vision integration.

[0026] However, the current reconstruction of three-dimensional scenes still relies on paired reconstruction of image pairs, which requires real camera poses and a lengthy pose optimization process, limiting the application of 3D Gaussian sputtering in real-world scenes.

[0027] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0028] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0029] Figure 1 is a schematic diagram of an exemplary system architecture to which an image generation method and a training method for a deep learning model can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0030] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0031] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices, including but not limited to smart phones, tablet computers, laptops, etc.

[0032] At least one of the image generation method and the deep learning model training method provided in the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, at least one of the image generation device and the deep learning model training device provided in the embodiments of the present disclosure can generally be set in the server 105. The image generation method and the deep learning model training method provided in the embodiments of the present disclosure can also be executed by a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the image generation device and the deep learning model training device provided in the embodiments of the present disclosure can also be set in a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0033] Figure 2 is a flowchart of an image generating method according to an embodiment of the present disclosure.

[0034] like Figure 2 As shown, the image generating method 200 includes operations S210 to S230.

[0035] In operation S210, a point cloud image corresponding to the input image is generated according to depth information of the input image.

[0036] The input image may be a plurality of images of different perspectives of the target scene. For example, it may be an image pair consisting of two images of different perspectives of the target scene. The two images in the image pair may be images taken by the acquisition device at different angles at the target position, or images taken at different target positions in the target scene.

[0037] The target scene may be an indoor scene, an outdoor scene, a scene for a certain object, a scene for a certain environment, etc. The target position may be a position in the target scene.

[0038] The depth information of the input image refers to the distance of each pixel in the input image relative to the shooting source, which can be obtained using a pre-trained depth estimation model. For example, the depth information of each pixel in the input image can be obtained using the depth estimation model, and the position information of the point in space corresponding to each pixel can be determined based on the depth information of each pixel in the input image. The point in space can be called a spatial point. By determining the spatial point corresponding to each pixel in the input image, a point cloud image can be obtained.

[0039] In operation S220 , a scene description of a new perspective for a target scene is generated according to the input image and the point cloud image.

[0040] For example, based on the pixel information in the input image and the point cloud information in the point cloud image, the target scene can be reconstructed in three dimensions to obtain a three-dimensional scene description. The reconstructed three-dimensional scene may include a scene with the same viewing angle as the input image, or a scene with a different viewing angle than the input image.

[0041] For example, the input image may include images taken by the acquisition device at a target position rotated by 20° and 60° respectively. The reconstructed 3D scene may include 3D scenes at 20° and 60°, and may also include 3D scenes at viewing angles between 20° and 60°.

[0042] Gaussian points are a way of expressing scenes. Scene descriptions can be represented as a set of Gaussian points. Each Gaussian point in a Gaussian point set can contain information such as position, color, density (or opacity), and scale. Gaussian points can also be called Gaussian spheres or Gaussian bodies.

[0043] A Gaussian point set can be generated by a 3D Gaussian generative model. The input of the 3D Gaussian generative model of this embodiment includes input images of different perspectives of the target scene and point cloud images of the input images, and the output is a Gaussian point set of a new perspective. In other words, the 3D Gaussian generative model reconstructs a three-dimensional scene based on input images and point cloud images of different perspectives to obtain a three-dimensional scene of a new perspective, and the three-dimensional scene of the new perspective is represented by a Gaussian point set.

[0044] The 3D Gaussian generation model can convert each spatial point in the point cloud image into a Gaussian point. Specifically, each spatial point can be converted into a 3D Gaussian distribution function, which contains information such as position, color, density, and scale.

[0045] The 3D Gaussian generation model can also predict Gaussian points of new perspectives based on the generated Gaussian points. For example, the positions of spatial points in the point cloud image can be fine-tuned to obtain the positions of new Gaussian points, and the color, scale, and density of new Gaussian points can be predicted based on the color, scale, and density of the already generated Gaussian points, thereby generating new Gaussian points.

[0046] The Gaussian points obtained by transforming the spatial points and the predicted new Gaussian points can form a Gaussian point set of a new perspective.

[0047] In operation S230, a new image corresponding to the new viewing angle is generated according to the scene description.

[0048] For example, the Gaussian points can be projected onto a two-dimensional plane, and the Gaussian points can be rendered according to the color, scale, and density information to obtain a two-dimensional image, that is, an image of the target scene from a new perspective.

[0049] The embodiments of the present disclosure generate a point cloud image based on the depth information of an input image, reconstruct a three-dimensional scene based on the input image and the point cloud image, obtain a scene description of a new perspective, and generate an image of a new perspective based on the scene description of the new perspective. The method of generating an image of a new perspective by reconstructing the three-dimensional scene can improve the image generation quality and generation efficiency.

[0050] Figure 3 is a schematic diagram of an image generating method according to an embodiment of the present disclosure.

[0051] like Figure 3 As shown, this embodiment includes a depth estimation model 310 and a Gaussian generation model 320. The input image 301 is a plurality of images of different perspectives of the target scene. The input image is input into the depth estimation model 310 to obtain a point cloud image 302.

[0052] According to an embodiment of the present disclosure, based on the depth information of the pixel points in the input image, the position information of the spatial points corresponding to the pixel points is determined; and based on the position information of the spatial points corresponding to the pixel points, a point cloud image corresponding to the input image is generated.

[0053] The depth estimation large model 310 is used to perform depth estimation on the image and generate a point cloud image based on the estimated depth information. The depth estimation large model 310 can be a pre-trained large-scale model with high credibility. Therefore, the input image 301 can be input into the depth estimation large model 310 in the frozen state of the depth estimation large model 310 to obtain a point cloud image 302. The depth estimation large model 310 in the frozen state means that the depth estimation large model 310 does not participate in the training of the Gaussian generation task of the present disclosure, that is, during the training process of the Gaussian generation model 320, the parameters of the depth estimation large model 310 remain unchanged.

[0054] The depth estimation large model 310 does not need to know the information of camera calibration or viewpoint posture in advance, and can transform the paired reconstruction problem into the arrangement of point cloud images, relaxing the hard constraints of the traditional projection camera model. In the case of providing multiple images, every two consecutive frames can be used as an image pair and input into the depth estimation large model 310 to obtain the point cloud image of the image pair. The alignment of point cloud images is simpler and more effective, and all pairs of point cloud images can be arranged according to spatial positions so that all point cloud images are expressed in a common scene.

[0055] The network architecture of the depth estimation large model 310 can be based on a standard Transformer encoder and decoder and obtained through pre-training. The point cloud information provided by the highly reliable depth estimation large model 310 can provide a reliable data source for subsequent 3D reconstruction and improve the 3D scene reconstruction effect.

[0056] In addition, the large depth estimation model is not restricted by the scene and can generate highly reliable point cloud images for images of any scene. Compared with the neural radiation field model, which needs to optimize the synthesis of views for different scenes, the embodiments of the present disclosure provide point cloud images through the large depth estimation model, which can reduce the restrictions on the scene and synthesize new views in a wider range of scenes.

[0057] The input image 301 and the point cloud image 302 are input into the Gaussian generation model 320 to obtain a Gaussian point set 303 .

[0058] According to an embodiment of the present disclosure, based on pixel information of an input image and distribution characteristics of a point cloud in the point cloud image, spatial points in a point cloud image are converted into Gaussian points to obtain Gaussian points corresponding to the input image; and based on the Gaussian points corresponding to the input image, a set of Gaussian points for a new perspective on a target scene is generated.

[0059] The Gaussian generation model 320 can convert each spatial point in the point cloud image 302 into a Gaussian point to obtain a Gaussian point set, which can be expressed as , where k represents the kth Gaussian point, For location, is the covariance matrix, is the opacity, is a spherical harmonic function. The spherical harmonic function characterizes the color distribution of the Gaussian point surface, the covariance matrix characterizes the scale of the Gaussian point, which may include the rotation angle, size, etc., the opacity, i.e., the density, characterizes the degree to which the Gaussian point blocks light, and the position characterizes the position of the center of the Gaussian point in space, which can be directly converted from the position of the point cloud in the point cloud image.

[0060] According to an embodiment of the present disclosure, the color information of the Gaussian points is determined based on the pixel information; and the position, scale and density of the Gaussian points are determined based on the distribution characteristics of the point cloud in the point cloud image.

[0061] For example, the position information of each Gaussian point can be determined by the position information of the spatial point, the color distribution information can be determined according to the color of the pixel points in the input image, and the density and scale information can be determined according to the distribution shape characteristics of the point cloud.

[0062] The input image 301 may be an image pair with different perspectives, and the point cloud image 302 may be a point cloud image pair corresponding to the input image pair. A Gaussian point set corresponding to the input image pair may be obtained by performing three-dimensional scene reconstruction based on the input image 301 and the point cloud image 302. In addition, the Gaussian generation model 320 may also generate a Gaussian point set of a new perspective based on the Gaussian point set corresponding to the input image pair.

[0063] According to an embodiment of the present disclosure, the positions of Gaussian points corresponding to the input image are adjusted to obtain new positions of Gaussian points, and the color, scale and density of the new Gaussian points are determined to obtain new Gaussian points; and based on the Gaussian points corresponding to the input image and the new Gaussian points, a set of Gaussian points for a new perspective of the target scene is generated.

[0064] In one example, the positions of the spatial points in the point cloud image 302 may be fine-tuned to obtain new spatial points, so that the point cloud in the point cloud image 302 is denser and covers a new perspective of the target scene. A set of Gaussian points corresponding to the new perspective may be generated based on the point cloud image 302 covering the new perspective.

[0065] In another example, the positions of the generated Gaussian points corresponding to the input image can be fine-tuned to obtain new Gaussian points, which can cover the new viewing angle. The color, scale, and density of the new Gaussian points can be predicted based on the color, scale, and density of the already generated Gaussian points to generate a new set of Gaussian points. The already generated set of Gaussian points corresponding to the input image and the predicted set of Gaussian points constitute the set of Gaussian points corresponding to the new viewing angle.

[0066] The Gaussian generation model 320 may include a U-Net network, which is an encoder-decoder based network structure and can predict the color, scale and density of Gaussian points based on the color information in the input image 301 and the point cloud shape distribution information in the point cloud image 302.

[0067] There can be multiple images of different perspectives for the target scene, and building a complete three-dimensional scene may require hundreds, thousands, or even tens of thousands of images. Every two adjacent frames of images can be combined into an image pair, and depth estimation, point cloud image generation, and three-dimensional Gaussian point set generation can be performed on each image pair in turn. In the above process, after generating a Gaussian point set for a part of the image pairs (such as 100 image pairs), the Gaussian points of the part of the image can be released, and then a new image pair can be obtained to generate Gaussian points. This dynamic acquisition and release mechanism can reduce the processing pressure of computing resources.

[0068] After obtaining the Gaussian point set 303 , rendering may be performed according to the Gaussian point set 303 to obtain an image with a new perspective.

[0069] According to an embodiment of the present disclosure, Gaussian points in a Gaussian point set are projected onto a two-dimensional plane based on position and scale information; and the Gaussian points on the two-dimensional plane are rendered based on color and density information to obtain a new image corresponding to a new perspective.

[0070] For example, the scale includes geometric information such as shape, size, and rotation angle, and the position includes depth information. The Gaussian points are sorted according to the depth, and the sorted Gaussian points are projected according to the above geometric information, and can be projected onto a two-dimensional plane. On the two-dimensional plane, the Gaussian points are rendered according to their positions and opacity to obtain a new image. The new image is an image corresponding to the new perspective.

[0071] Different from traditional implicit neural network rendering, the embodiments of the present disclosure use explicit 3D Gaussian points to represent the reconstructed three-dimensional scene, which makes the rendering process more efficient and scalable, and realizes real-time rendering and efficient synthesis of new views.

[0072] According to an embodiment of the present disclosure, the present disclosure also provides a training method for a deep learning model.

[0073] The deep learning model training method disclosed in the present invention can be applied to the training of the above-mentioned Gaussian generation model.

[0074] Figure 4 It is a flowchart of a method for training a deep learning model according to an embodiment of the present disclosure.

[0075] like Figure 4 As shown, the training method 400 of the deep learning model includes operations S410~S450.

[0076] In operation S410, a point cloud image corresponding to the sample image is generated according to depth information of the sample image.

[0077] For example, the sample image may include multiple images with different perspectives for the sample scene. A pre-trained depth estimation model may be used to accurately estimate the depth of the sample image, and a point cloud image may be generated based on the estimated depth information.

[0078] For example, the training set may include multiple frames of continuous sample images for the sample scene, and each two consecutive frames of images may be combined into a sample image pair, and the sample image pair may be input into a pre-trained depth estimation model to obtain a point cloud image pair corresponding to the sample image pair.

[0079] In operation S420, the sample image and the point cloud image are input into a deep learning model to generate a set of Gaussian points for a target perspective of the sample scene.

[0080] The deep learning model can be a Gaussian generative model, which can include a U-Net network. The U-Net network is an encoder-decoder based network structure, which can convert spatial points in a point cloud image into Gaussian points based on color information in the sample image and point cloud shape distribution characteristics in the point cloud image.

[0081] For example, the position of the Gaussian point is determined according to the position of the spatial point in the point cloud image, the color distribution information of the Gaussian point is determined according to the pixel color information of the sample image, and the scale, size, rotation angle, opacity (density) and other information of the Gaussian point are determined according to the point cloud shape distribution characteristics.

[0082] The Gaussian generation model can also fine-tune the position of the Gaussian point as the position of the new Gaussian point, and determine the color, scale and density of the new Gaussian point based on the color, scale, density and other information of the generated Gaussian point.

[0083] The Gaussian points corresponding to the sample image and the predicted new Gaussian points constitute a Gaussian point set, which is an expression of the sample scene at the target perspective. The target perspective is a perspective different from the perspective of the sample image. For example, it may include the perspective of two sample images and the perspective between the perspectives of the two sample images.

[0084] In operation S430, a target image corresponding to a target viewing angle is generated according to the Gaussian point set.

[0085] For example, the Gaussian points in the Gaussian point set of the target perspective may be projected onto a two-dimensional plane, and then the Gaussian points may be rendered according to color and opacity to obtain a target image.

[0086] In operation S440, a loss of the deep learning model is determined according to the difference between the target image and the supervision image.

[0087] In operation S450, parameters of the deep learning model are adjusted according to the loss.

[0088] The target image is an image of the target perspective of the sample scene obtained by rendering based on the Gaussian point set. The supervision image is the true value image of the target perspective of the target sample scene in the training set. The loss of the deep learning model can be determined based on the difference between the target image and the supervision image. The parameters of the deep learning model are adjusted according to the loss to complete a round of training.

[0089] The above process is repeated using multiple sample image pairs in the training set for multiple rounds of training to obtain a trained deep learning model.

[0090] The trained deep learning model of the embodiment of the present disclosure can generate a new perspective scene description for any scene, that is, a set of Gaussian points, thereby rendering a new perspective image, thereby improving the generation quality and efficiency of the new perspective image.

[0091] According to an embodiment of the present disclosure, the depth loss, luminosity loss and color loss are determined based on the differences between the depth, luminosity and color information of the target image and the depth, luminosity and color information of the supervisory image, respectively; and the overall loss is determined based on the depth loss, luminosity loss and color loss.

[0092] For the target image and the supervision image, the depth loss, photometric loss and color loss can be calculated respectively.

[0093] In one example, a large depth estimation model can be used to estimate the depth information of the target image and the supervision image respectively, and the depth loss is determined according to the difference between the two.

[0094] The luminosity of an image refers to how bright it is, and can be obtained by calculating the average brightness of the image. After obtaining the luminosity of the target image and the supervised image, the difference between the luminosities of the two can be calculated to determine the luminosity loss.

[0095] The RGB difference between the target image and the supervised image can be calculated to obtain the color loss.

[0096] Next, the depth loss, luminosity loss, and color loss may be integrated including dynamic weight coefficients to obtain the overall loss. For example, the overall loss may be obtained according to the following formula (1).

[0097] (1)

[0098] Where t represents the number of training steps, It is a dynamic weight coefficient that can be adjusted during training. represents the depth loss, represents the light loss, Indicates color loss.

[0099] The embodiments of the present disclosure adjust the parameters of the deep learning model according to the losses in multiple aspects such as depth, luminosity and color, which can improve the accuracy of the Gaussian point set generated by the model.

[0100] According to the embodiments of the present disclosure, the Gaussian generative model provided by the embodiments of the present disclosure has achieved advanced results on multiple public data sets.

[0101] Figure 5 is a comparison diagram of image generation effects according to an embodiment of the present disclosure.

[0102] like Figure 5 As shown, the first row of images 510 are images generated according to the image generation method provided by the embodiment of the present disclosure. The middle row of images 520 are images generated according to the new view synthesis method of the related art (such as the new view synthesis method based on Gaussian radiation field). The third row of images 530 are true value images.

[0103] The quality of the images generated by the image generation method provided by the embodiments of the present disclosure in various scenarios is higher than that of the images 520 generated by the new view synthesis method based on the Gaussian radiation field. For example, the peak signal-to-noise ratio (PSNR) of each image in the first row of images 510 is higher than that of the corresponding image in the second row of images 520. PSNR is an indicator for measuring image quality. The higher the PSNR, the smaller the image distortion, that is, the higher the image quality.

[0104] Each image in the second row of images 520 has a distorted portion, such as the portion framed by a red frame, which has distortions such as blur and low resolution.

[0105] According to the embodiments of the present disclosure, the image generation method provided by the embodiments of the present disclosure can generate high-quality images and provide real-time rendering for immersive experience. It can be applied to fields that require real-time 3D graphics and computer vision technology, such as virtual reality, augmented reality, autonomous driving, robot navigation, 3D modeling and rendering.

[0106] According to an embodiment of the present disclosure, the present disclosure also provides an image generating device and a training device for a deep learning model.

[0107] Figure 6 is a block diagram of an image generating apparatus according to an embodiment of the present disclosure.

[0108] like Figure 6 As shown, the image generating device 600 includes a first point cloud image generating module 610 , a scene description generating module 620 and a first image generating module 630 .

[0109] The first point cloud image generation module 610 is used to generate a point cloud image corresponding to the input image according to the depth information of the input image, wherein the input image includes a plurality of images of different viewing angles for the target scene.

[0110] The scene description generation module 620 is used to generate a scene description of a new perspective for a target scene based on the input image and the point cloud image, wherein the new perspective is a perspective different from that of the input image.

[0111] The first image generation module 630 is used to generate a new image corresponding to a new viewing angle according to the scene description.

[0112] The first point cloud image generating module 610 includes a space point determining unit and a point cloud image determining unit.

[0113] The spatial point determination unit is used to determine the position information of the spatial point corresponding to the pixel point according to the depth information of the pixel point in the input image.

[0114] The point cloud image determination unit is used to generate a point cloud image corresponding to the input image according to the position information of the space point corresponding to the pixel point.

[0115] The scene description includes a set of Gaussian points. The scene description generation module 620 includes a first Gaussian point generation unit and a second Gaussian point generation unit.

[0116] The first Gaussian point generation unit is used to convert the spatial points in the point cloud image into Gaussian points according to the pixel information of the input image and the distribution characteristics of the point cloud in the point cloud image, so as to obtain Gaussian points corresponding to the input image.

[0117] The second Gaussian point generating unit is used to generate a Gaussian point set of a new viewing angle for a target scene according to the Gaussian points corresponding to the input image.

[0118] The Gaussian point includes position, color, scale and density. The first Gaussian point generation unit includes a first parameter determination subunit and a second parameter determination subunit.

[0119] The first parameter determination subunit is used to determine the color information of the Gaussian point according to the pixel information.

[0120] The second parameter determination subunit is used to determine the position, scale and density of the Gaussian points according to the distribution characteristics of the point cloud in the point cloud image.

[0121] The second Gaussian point generating unit includes a parameter adjusting subunit and a third parameter determining subunit.

[0122] The parameter adjustment subunit is used to adjust the position of the Gaussian point corresponding to the input image to obtain the position of the new Gaussian point, and determine the color, scale and density of the new Gaussian point to obtain the new Gaussian point.

[0123] The third parameter determination subunit is used to generate a Gaussian point set of a new viewing angle for the target scene according to the Gaussian points corresponding to the input image and the new Gaussian points.

[0124] The first image generating module 630 includes a projection unit and a rendering unit.

[0125] The projection unit is used to project the Gaussian points in the Gaussian point set onto a two-dimensional plane according to position and scale information.

[0126] The rendering unit is used to render Gaussian points on a two-dimensional plane according to color and density information to obtain a new image corresponding to a new viewing angle.

[0127] Figure 7 is a block diagram of a training device for a deep learning model according to an embodiment of the present disclosure.

[0128] like Figure 7As shown, the training device 700 of the deep learning model includes a second point cloud image generation module 710, a Gaussian point generation module 720, a second image generation module 730, a loss determination module 740 and an adjustment module 750.

[0129] The second point cloud image generating module 710 is used to generate a point cloud image corresponding to the sample image according to the depth information of the sample image, wherein the sample image includes a plurality of images of different viewing angles for the sample scene.

[0130] The Gaussian point generation module 720 is used to input the sample image and the point cloud image into the deep learning model to generate a set of Gaussian points for a target perspective of the sample scene, wherein the target perspective is a perspective different from that of the sample image.

[0131] The second image generation module 730 is used to generate a target image corresponding to the target viewing angle according to the Gaussian point set.

[0132] The loss determination module 740 is used to determine the loss of the deep learning model based on the difference between the target image and the supervision image, wherein the supervision image is an image of the target perspective of the sample scene.

[0133] The adjustment module 750 is used to adjust the parameters of the deep learning model according to the loss.

[0134] The loss determination module includes a first loss determination unit and a second loss determination unit.

[0135] The first loss determination unit is used to determine the depth loss, the photometric loss and the color loss according to the differences between the depth, photometric and color information of the target image and the depth, photometric and color information of the supervision image, respectively.

[0136] The second loss determination unit is used to determine the overall loss according to the depth loss, the luminosity loss and the color loss.

[0137] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0138] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0139] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0140] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0141] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as at least one of the image generation method and the training method of the deep learning model. For example, in some embodiments, at least one of the image generation method and the training method of the deep learning model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of at least one of the image generation method and the training method of the deep learning model described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute at least one of the image generation method and the deep learning model training method in any other appropriate manner (for example, by means of firmware).

[0142] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0144] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0145] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0146] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0147] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0148] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0149] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating an image, comprising: Generate a point cloud image corresponding to the input image according to depth information of the input image, wherein the input image includes a plurality of images of different viewing angles for a target scene; Generating a scene description of a new perspective for the target scene according to the input image and the point cloud image, wherein the new perspective is a perspective different from that of the input image; and According to the scene description, a new image corresponding to the new perspective is generated.

2. The method according to claim 1, wherein: Generating a point cloud image corresponding to the input image according to the depth information of the input image comprises: Determining, based on the depth information of the pixel in the input image, the position information of the spatial point corresponding to the pixel; and A point cloud image corresponding to the input image is generated according to the position information of the spatial point corresponding to the pixel point.

3. The method according to claim 1, wherein: The scene description includes a Gaussian point set; and generating a scene description of a new perspective for the target scene according to the input image and the point cloud image includes: According to the pixel information of the input image and the distribution characteristics of the point cloud in the point cloud image, converting the spatial points in the point cloud image into Gaussian points to obtain Gaussian points corresponding to the input image; and A Gaussian point set of a new viewing angle for the target scene is generated according to the Gaussian points corresponding to the input image.

4. The method according to claim 3, wherein: The Gaussian point includes position, color, scale and density; and converting the spatial points in the point cloud image into Gaussian points according to the pixel information of the input image and the distribution characteristics of the point cloud in the point cloud image includes: Determining color information of the Gaussian point according to the pixel information; and The position, scale and density of the Gaussian points are determined according to the distribution characteristics of the point cloud in the point cloud image.

5. The method according to claim 3, wherein: The step of generating a Gaussian point set of a new viewing angle for the target scene according to the Gaussian points corresponding to the input image comprises: Adjusting the position of the Gaussian point corresponding to the input image to obtain the position of a new Gaussian point, and determining the color, scale and density of the new Gaussian point to obtain the new Gaussian point; and A Gaussian point set of a new viewing angle for the target scene is generated according to the Gaussian points corresponding to the input image and the new Gaussian points.

6. The method according to claim 5, wherein: The generating a new image corresponding to the new perspective according to the scene description comprises: Projecting the Gaussian points in the Gaussian point set onto a two-dimensional plane according to the position and scale information; and The Gaussian points on the two-dimensional plane are rendered according to the color and density information to obtain a new image corresponding to the new viewing angle.

7. A method for training a deep learning model, comprising: Generating a point cloud image corresponding to the sample image according to the depth information of the sample image, wherein the sample image includes a plurality of images of different viewing angles for the sample scene; Inputting the sample image and the point cloud image into a deep learning model to generate a Gaussian point set for a target perspective of the sample scene, wherein the target perspective is a perspective different from that of the sample image; Generating a target image corresponding to the target viewing angle according to the Gaussian point set; Determining the loss of the deep learning model according to a difference between the target image and a supervision image, wherein the supervision image is an image of a target perspective of the sample scene; and Based on the loss, parameters of the deep learning model are adjusted.

8. The method according to claim 7, wherein: Determining the loss of the deep learning model according to the difference between the target image and the supervision image includes: Determine a depth loss, a photometric loss, and a color loss according to the differences between the depth, photometric, and color information of the target image and the depth, photometric, and color information of the supervisory image, respectively; Based on the depth loss, luminosity loss and color loss, an overall loss is determined.

9. An image generating device, comprising: A first point cloud image generation module, configured to generate a point cloud image corresponding to an input image according to depth information of the input image, wherein the input image includes a plurality of images of different viewing angles for a target scene; a scene description generating module, configured to generate a scene description of a new perspective for the target scene according to the input image and the point cloud image, wherein the new perspective is a perspective different from that of the input image; and The first image generation module is used to generate a new image corresponding to the new viewing angle according to the scene description.

10. The device according to claim 9, wherein: The first point cloud image generation module includes: a spatial point determination unit, configured to determine position information of a spatial point corresponding to the pixel point according to depth information of the pixel point in the input image; and The point cloud image determination unit is used to generate a point cloud image corresponding to the input image according to the position information of the space point corresponding to the pixel point.

11. The device according to claim 9, wherein: The scene description includes a Gaussian point set; and the scene description generation module includes: a first Gaussian point generating unit, configured to convert the spatial points in the point cloud image into Gaussian points according to the pixel information of the input image and the distribution characteristics of the point cloud in the point cloud image, so as to obtain Gaussian points corresponding to the input image; and The second Gaussian point generating unit is used to generate a Gaussian point set of a new viewing angle for the target scene according to the Gaussian points corresponding to the input image.

12. The device according to claim 11, wherein The Gaussian points include position, color, scale and density; The first Gaussian point generating unit comprises: A first parameter determination subunit, configured to determine color information of the Gaussian point according to the pixel information; as well as The second parameter determination subunit is used to determine the position, scale and density of the Gaussian points according to the distribution characteristics of the point cloud in the point cloud image.

13. The device according to claim 11, wherein: The second Gaussian point generating unit comprises: a parameter adjustment subunit, configured to adjust the position of the Gaussian point corresponding to the input image to obtain the position of a new Gaussian point, and determine the color, scale and density of the new Gaussian point to obtain the new Gaussian point; and The third parameter determination subunit is used to generate a Gaussian point set of a new viewing angle for the target scene according to the Gaussian points corresponding to the input image and the new Gaussian points.

14. The device according to claim 13, wherein: The first image generation module comprises: a projection unit, configured to project the Gaussian points in the Gaussian point set onto a two-dimensional plane according to the position and scale information; and A rendering unit is used to render the Gaussian points on the two-dimensional plane according to the color and density information to obtain a new image corresponding to the new viewing angle.

15. A training device for a deep learning model, comprising: A second point cloud image generating module, configured to generate a point cloud image corresponding to the sample image according to the depth information of the sample image, wherein the sample image includes a plurality of images of different viewing angles for the sample scene; A Gaussian point generation module, used to input the sample image and the point cloud image into a deep learning model to generate a Gaussian point set for a target perspective of the sample scene, wherein the target perspective is a perspective different from that of the sample image; A second image generation module, configured to generate a target image corresponding to the target viewing angle according to the Gaussian point set; a loss determination module, configured to determine the loss of the deep learning model according to a difference between the target image and a supervision image, wherein the supervision image is an image of a target perspective of the sample scene; and An adjustment module is used to adjust the parameters of the deep learning model according to the loss.

16. The device according to claim 15, wherein: The loss determination module comprises: A first loss determination unit, configured to determine a depth loss, a photometric loss, and a color loss according to differences between the depth, photometric, and color information of the target image and the depth, photometric, and color information of the supervisory image, respectively; The second loss determination unit is used to determine the overall loss according to the depth loss, luminosity loss and color loss.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 8.

19. A computer program product, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Citation Information

Cited By

  • Large model-based three-dimensional scene image generation method and apparatus, and electronic device

    CN119784940A