Image generation method and device, storage medium and electronic device

Abstract

The present disclosure provides an image generation method, an image generation device, a storage medium, and an electronic device, relating to the field of image processing technology. The image generation method includes: obtaining a camera pose corresponding to the image to be generated, determining rays corresponding to each pixel in the image to be generated based on the camera pose; for each sampling point on the ray, determining the spatial semantic information of the sampling point based on the position information of the sampling point; determining the color information and density information of the sampling point based on the position information and spatial semantic information of the sampling point; and using the color information and density information of each sampling point on the ray to determine the color information of the pixel corresponding to the ray in the image to be generated, thereby generating the image to be generated. The present disclosure can improve the effect of generating new perspective images.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image generation method, an image generation device, a storage medium, and an electronic device. Background Art

[0002] 3D reconstruction and new-perspective image rendering have long been at the core of computer graphics and a key research focus in recent years. With the emergence of new concepts such as digital twins, holographic communications, and the metaverse, the demand for these technologies is increasing across industry.

[0003] Currently, new perspective image generation schemes generally have problems such as weak scene resolution and inflexible schemes.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0005] The purpose of the present disclosure is to provide an image generation method, an image generation device, a storage medium and an electronic device, thereby overcoming, at least to a certain extent, the problems of weak scene resolution and inflexibility of image generation schemes.

[0006] According to a first aspect of the present disclosure, an image generation method is provided, comprising: obtaining a camera pose corresponding to an image to be generated, and determining, based on the camera pose, a ray corresponding to each pixel in the image to be generated; determining, for each sampling point on the ray, spatial semantic information of the sampling point based on position information of the sampling point; determining, based on the position information and spatial semantic information of the sampling point, color information and density information of the sampling point; and determining, using the color information and density information of each sampling point on the ray, color information of the pixel corresponding to the ray in the image to be generated, to generate the image to be generated.

[0007] Optionally, determining the spatial semantic information of the sampling point based on the position information of the sampling point includes: pre-constructing a spatial cube, initializing the configuration of spatial semantic information for each pixel in the spatial cube; using the initialized spatial semantic information as a learnable parameter, and training the spatial semantic information using a training image of the same scene as the image to be generated to obtain the spatial semantic information corresponding to each pixel in the spatial cube; when the sampling point is a pixel in the spatial cube, determining the corresponding spatial semantic information based on the position information of the sampling point.

[0008] Optionally, the image generation method also includes: when the sampling point is not a pixel point in the spatial cube, determining the pixel points adjacent to the sampling point in the spatial cube; and interpolating the spatial semantic information corresponding to the pixel points adjacent to the sampling point to obtain the spatial semantic information of the sampling point.

[0009] Optionally, the color information and density information of the sampling point are determined based on the position information and spatial semantic information of the sampling point, including: inputting the position information and spatial semantic information of the sampling point into a fully connected network, and determining the color information and density information of the sampling point through a prediction process of the fully connected network.

[0010] Optionally, the image generation method also includes: pre-acquiring a training image of the same scene as the image to be generated; inputting the position information and spatial semantic information of the sampling points on the ray corresponding to the training image into the fully connected network; generating an intermediate image based on the color information and density information output by the fully connected network during the training process; comparing the intermediate image with a pre-selected real image, calculating loss information, and using the loss information to adjust the network parameters in the fully connected network.

[0011] Optionally, the training images participating in the network training include training images whose camera poses are different from the camera poses corresponding to the images to be generated.

[0012] Optionally, the color information and density information of each sampling point on the ray are used to determine the color information of the pixel points corresponding to the ray in the image to be generated, including: using the density information of each sampling point on the ray as a weight, performing an integration operation on the corresponding color information to determine the color information of the pixel points corresponding to the ray in the image to be generated.

[0013] According to a second aspect of the present disclosure, an image generation device is provided, comprising: a ray determination module for acquiring a camera pose corresponding to an image to be generated, and determining, based on the camera pose, rays corresponding to each pixel in the image to be generated; a spatial feature determination module for determining, for each sampling point on a ray, spatial semantic information of the sampling point based on the position information of the sampling point; an information determination module for determining, based on the position information and spatial semantic information of the sampling point, color information and density information of the sampling point; and an image generation module for determining, by using the color information and density information of each sampling point on the ray, color information of the pixel corresponding to the ray in the image to be generated, so as to generate the image to be generated.

[0014] Optionally, the spatial feature determination module can be configured to perform: pre-constructing a spatial cube, initializing and configuring spatial semantic information for each pixel in the spatial cube; using the initialized spatial semantic information as a learnable parameter, and training the spatial semantic information using training images of the same scene as the image to be generated to obtain the spatial semantic information corresponding to each pixel in the spatial cube; when the sampling point is a pixel in the spatial cube, determining the corresponding spatial semantic information based on the position information of the sampling point.

[0015] Optionally, the spatial feature determination module can also be configured to perform: when the sampling point is not a pixel point in the spatial cube, determine the pixel points adjacent to the sampling point in the spatial cube; and interpolate the spatial semantic information corresponding to the pixel points adjacent to the sampling point to obtain the spatial semantic information of the sampling point.

[0016] Optionally, the information determination module may be configured to execute: inputting the position information and spatial semantic information of the sampling point into a fully connected network, and determining the color information and density information of the sampling point through a prediction process of the fully connected network.

[0017] Optionally, the image generation device also includes a training module, which can be configured to perform: pre-acquiring a training image of the same scene as the image to be generated; inputting the position information and spatial semantic information of the sampling points on the ray corresponding to the training image into the fully connected network; generating an intermediate image based on the color information and density information output by the fully connected network during the training process; comparing the intermediate image with a pre-selected real image, calculating the loss information, and using the loss information to adjust the network parameters in the fully connected network.

[0018] Optionally, the training images participating in the network training include training images whose camera poses are different from the camera poses corresponding to the images to be generated.

[0019] Optionally, the image generation module may be configured to perform: using density information of each sampling point on the ray as a weight, performing an integration operation on the corresponding color information to determine color information of a pixel corresponding to the ray in the image to be generated.

[0020] According to a third aspect of the present disclosure, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned image generation method is implemented.

[0021] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; the processor is configured to implement the above-mentioned image generation method by executing the executable instructions.

[0022] In the technical solutions provided in some embodiments of the present disclosure, the color information and density information of the sampling points are determined based on the position information and spatial semantic information of the sampling points on the ray, and then the color information and density information of each sampling point on the ray are used to determine the color information of the pixel points corresponding to the ray on the image to be generated to generate the image. On the one hand, the disclosed solution combines spatial semantic information to generate new perspective images, improves the scene resolution capability, and is not limited by the implementation scene and has universality. On the other hand, since the spatial semantic information is taken into account, the generated image can better match the perspective (camera pose), and the image generation accuracy is high.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0025] Figure 1 A schematic diagram showing an image generation process according to an embodiment of the present disclosure;

[0026] Figure 2 A schematic diagram showing various stages involved in the image generation scheme of an embodiment of the present disclosure;

[0027] Figure 3 A flowchart schematically illustrates an image generation method according to an exemplary embodiment of the present disclosure;

[0028] Figure 4 Schematically shows a flow chart of determining spatial semantic information of a sampling point according to an embodiment of the present disclosure;

[0029] Figure 5 A flowchart schematically illustrates a scene editing process according to an embodiment of the present disclosure;

[0030] Figure 6 A block diagram schematically illustrates an image generating apparatus according to an exemplary embodiment of the present disclosure;

[0031] Figure 7 Schematically shows a block diagram of an image generating apparatus according to another exemplary embodiment of the present disclosure;

[0032] Figure 8 A block diagram schematically illustrates an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or that other methods, components, devices, steps, etc. may be employed. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0034] In addition, the accompanying drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0035] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to actual circumstances.

[0036] It should be noted that the image generation scheme of the embodiment of the present disclosure can be implemented by an electronic device. That is, each step of the image generation method of the embodiment of the present disclosure can be performed by an electronic device, and the image generation device can be configured in the electronic device.

[0037] The electronic devices mentioned in this disclosure include but are not limited to smart phones, tablet computers, smart wearable devices, etc. In addition, the electronic device can also be a server, which is not limited in this disclosure.

[0038] Figure 1 Schematic diagram showing the image generation process of an embodiment of the present disclosure. Figure 1If the image generation process of the disclosed embodiment is considered as a whole, then the input of this whole is the new perspective of the scene, and the output is the corresponding image under this new perspective. In addition, using images taken from other perspectives of the same scene, the parameters used in the image generation process can be trained to ensure that the corresponding image can be output under the input perspective. The perspective refers to the observer's observation angle. In the process of implementing the solution processing based on the computer in the disclosed embodiment, the camera pose can be used to represent the perspective.

[0039] For a real scene, images from different perspectives can be pre-captured and used to train the learnable parameters used in the image generation process of the disclosed embodiments to obtain adjusted results. When a new perspective is input, the information of the new perspective is processed by the disclosed image generation solution to generate an image corresponding to the new perspective.

[0040] It is understandable that the above-mentioned generated images and the images pre-used for training may be images of the same scene but different perspectives.

[0041] Figure 2 A schematic diagram showing various stages involved in the image generation solution of an embodiment of the present disclosure is shown.

[0042] The image generation scheme of the embodiment of the present disclosure may include forward propagation and backward propagation processes. For forward propagation, after inputting the target perspective, a target image corresponding to the target perspective may be generated through the processing of the three-dimensional virtual ray generation stage, the three-dimensional space point sampling stage, the three-dimensional space feature determination stage, the space point feature acquisition stage, the neural rendering network processing stage, and the three-dimensional volume rendering stage. For backward propagation, the learnable parameters included in the three-dimensional space feature determination stage and the neural rendering network processing stage may be adjusted through the three-dimensional volume rendering stage, the neural rendering network processing stage, the space point feature acquisition stage, and the three-dimensional space feature determination stage.

[0043] Below Figure 2 Each processing stage involved is described separately.

[0044] For the three-dimensional virtual ray generation stage: different viewing angles correspond to different camera poses. When the target viewing angle is obtained, that is, when the camera pose is obtained, rays are projected in the observation direction with the camera pose as the starting point. If the size of the target image is H×W (height×width), H×W rays are generated with the distance between pixels as the offset.

[0045] Regarding the 3D spatial point sampling stage: Due to the limitations of electronic device computing power and time-consuming operations, it is usually impossible to query information for every point in space. Therefore, spatial point sampling is required. Specifically, a fixed number of points are sampled along each ray, using a random sampling method, for example. Through 3D spatial point sampling, for example, H × W × N sampling points can be determined.

[0046] During the 3D spatial feature determination phase, a spatial cube of H×W×D (height×width×depth) is pre-set. Each pixel in the cube is initialized with a feature vector, which is then used as a learnable parameter. During training, these feature vectors are updated using images from the same scene. After training, each point in the space is assigned a feature vector representing semantic information, which can be used in subsequent inference processes.

[0047] Regarding the spatial point feature acquisition stage: the disclosed embodiment uses three-dimensional local features rather than global features. Therefore, the feature vector of the sampling point can be indexed according to the coordinate position of the sampling point in the three-dimensional space.

[0048] Regarding the neural rendering network processing stage: The neural rendering network used in this disclosure can be a fully connected network. The network input is the location of the sampling point in space and its feature vector, and the output is the color value and density value of the sampling point. The density value represents the transparency of the sampling point and can be used as a weight for the color value in subsequent processing.

[0049] For the 3D volume rendering stage: each ray in the 3D space corresponds to a pixel point in the 2D image from the new perspective. The rendering process is performed point by point, that is, the integration operation is performed on the 3D points sampled along the ray. The specific formula is as follows:

[0050]

[0051] Where C is the RGB value of a pixel in the target image, r represents the ray, d is the ray direction vector, c represents the RGB value of the spatial sampling point, and σ is the density value. tn and tf are the nearest and farthest points integrated along the ray, respectively. T(t) calculates the cumulative projection rate of the ray to a point on the ray.

[0052] The disclosed embodiment introduces learnable three-dimensional spatial semantic features to generate new perspective images. The introduction of this feature makes the system at least universal and controllable. Universality is reflected in the ability of the network model input to distinguish different scenes under the guidance of features as a prerequisite, and can adaptively output different values, so a trained network model can generate three-dimensional models of various scenes. Controllability is reflected in the fact that the three-dimensional spatial semantic features can obtain features point by point, so the feature vectors at different positions can be obtained accordingly and corresponding operations can be performed.

[0053] In addition, the image generation solution of the embodiment of the present disclosure can be an end-to-end rendering system, and the training of learnable parameters can be deployed on a cloud platform. For the same scene, the user only needs to input the desired perspective to obtain the image corresponding to the perspective in the scene.

[0054] In addition, users can also input different effects to achieve operations such as superimposing and moving objects in the scene.

[0055] Figure 3 The flowchart of the image generation method according to the exemplary embodiment of the present disclosure is schematically shown. Figure 3 , the image generation method may include the following steps:

[0056] S32. Obtain a camera pose corresponding to the image to be generated, and determine a ray corresponding to each pixel point in the image to be generated according to the camera pose.

[0057] The camera pose corresponds to the viewing angle, i.e., the observer's viewing position and posture. In one embodiment, the user can manually input the camera pose based on the desired image. In another embodiment, the electronic device can automatically generate the camera pose based on the image rendering effect. Factors involved in the image rendering effect include, but are not limited to, depth of field configuration, rotation, and movement.

[0058] After obtaining the camera pose, H×W rays can be emitted based on the camera pose. It should be understood that the rays mentioned in the embodiments of this disclosure are virtual rays, which are algorithmic means used in computer processing and do not imply that the rays actually exist in space. The size of the image to be generated is H×W, that is, each ray corresponds to a pixel in the image to be generated.

[0059] In addition, sampling can be performed on each ray to determine sampling points. For example, N sampling points are selected for each ray. In this case, a total of H×W×N sampling points are collected.

[0060] S34. For each sampling point on the ray, determine the spatial semantic information of the sampling point according to the position information of the sampling point.

[0061] According to some embodiments of the present disclosure, a spatial cube can be pre-constructed. Similarly, this cube is a means used during computer processing and does not imply the actual existence of a cube in space. Spatial semantic information is initialized and configured for each pixel in the spatial cube. This initialized spatial semantic information is used as a learnable parameter and trained using training images of the same scene as the image to be generated, thereby obtaining spatial semantic information corresponding to each pixel in the spatial cube.

[0062] For each sampling point on a ray, if the sampling point is a pixel point in a spatial cube, the corresponding spatial semantic information can be determined based on the position information of the sampling point; if the sampling point is not a pixel point in a spatial cube, the pixel points adjacent to the sampling point in the spatial cube can be determined, and the spatial semantic information corresponding to the pixel points adjacent to the sampling point can be interpolated to obtain the spatial semantic information of the sampling point.

[0063] Figure 4 The flowchart of determining the spatial semantic information of a sampling point according to an embodiment of the present disclosure is schematically shown.

[0064] In step S402, a spatial cube is constructed to initialize the spatial semantic information of the pixels.

[0065] In step S404, the spatial semantic information is trained using images of the same scene to determine the spatial semantic information corresponding to each pixel point on the spatial cube.

[0066] It should be understood that step S402 and step S404 may be pre-processing steps, that is, steps performed before the process of generating the image to be generated. In addition, these two steps may also be performed on the cloud platform.

[0067] In step S406, the electronic device may determine the location information of the sampling point.

[0068] In step S408, the electronic device can determine whether the sampling point is a pixel point of the spatial cube based on the position information of the sampling point. If it is a pixel point of the spatial cube, step S410 is executed to directly determine the corresponding spatial semantic information. If it is not a pixel point of the spatial cube, step S412 is executed to interpolate the spatial semantic information of the pixels adjacent to the sampling point in the spatial cube, and the interpolation result is determined as the spatial semantic information corresponding to the sampling point.

[0069] S36. Determine the color information and density information of the sampling point based on the position information and spatial semantic information of the sampling point.

[0070] In an exemplary embodiment of the present disclosure, the electronic device may input the location information and spatial semantic information of the sampling point into a fully connected network, namely the neural rendering network mentioned above, and determine the color information and density information of the sampling point through the prediction process of the fully connected network.

[0071] It is understood that the fully connected network is a network determined after training. The training process can be performed by the electronic device or by the cloud platform.

[0072] For pre-training, a training image of the same scene as the image to be generated is obtained. The positional information and spatial semantics of the sampling points on the rays corresponding to the training image are input into the fully connected network. Based on the color and density information output by the fully connected network during training, an intermediate image is generated. This intermediate image is then compared with a pre-selected real image to calculate loss information, which is used to adjust the network parameters in the fully connected network.

[0073] It is understood that the training images used in network training include training images with camera poses different from those corresponding to the to-be-generated images. In other words, the training images and the to-be-generated images are images of the same scene but from different perspectives.

[0074] S38. Using the color information and density information of each sampling point on the ray, determine the color information of the pixel points corresponding to the ray in the image to be generated, so as to generate the image to be generated.

[0075] The electronic device can determine the color information and density information of each sampling point on the outgoing ray according to step S36. In this case, the density information of each sampling point on the ray is used as a weight to perform an integration operation on the corresponding color information to determine the color information of the pixel corresponding to the ray in the to-be-generated image.

[0076] According to some embodiments of the present disclosure, the color information, ie, the RGB value, of the pixel corresponding to the ray in the image to be generated may be determined in combination with the following formula.

[0077]

[0078] Where C is the RGB value of a pixel in the target image, r represents the ray, d is the ray direction vector, c represents the RGB value of the spatial sampling point, and σ is the density value. tn and tf are the nearest and farthest points integrated along the ray, respectively. T(t) calculates the cumulative projection rate of the ray to a point on the ray.

[0079] For each ray, the color information of the corresponding pixel in the image to be generated is determined. Given the one-to-one correspondence between rays and pixels in the image to be generated, once the color information of each pixel in the image to be generated is determined, the image to be generated is generated, that is, an image of a new perspective of the scene is generated.

[0080] The following will refer to Figure 5 The process of superimposing an object in one scene into another scene according to an embodiment of the present disclosure is described.

[0081] In step S502 , the electronic device acquires an image of scene 1 .

[0082] In step S504, the electronic device segments the target object from the image of scene 1 and generates a volume mask. The target object may be an object of a predetermined type, such as a ball, a chair, etc., which is not limited in the present disclosure.

[0083] In step S506 , the electronic device determines the three-dimensional spatial characteristics of scene 1 .

[0084] In step S508 , the electronic device determines the three-dimensional spatial features of the target object using the three-dimensional spatial features of scene 1 and the volume mask of the target object.

[0085] In step S510 , the electronic device may determine the three-dimensional spatial characteristics of scene 2 .

[0086] In step S512, the electronic device may fuse the three-dimensional spatial features of the target object with the three-dimensional spatial features of scene 2 to determine the overall three-dimensional spatial features.

[0087] In step S514, the electronic device may determine the above-mentioned overall three-dimensional spatial features as spatial semantic information and execute the above-mentioned image generation process.

[0088] In step S516 , the electronic device may output an image of scene 2 superimposed with the target object.

[0089] In the processing of the image generation method of the above embodiment of the present disclosure, a new perspective image is generated by introducing a learnable three-dimensional spatial semantic feature. The introduction of this feature makes the system at least universal and controllable. Universality is manifested in that the input of the network model has the ability to distinguish different scenes under the guidance of the feature as a prerequisite, and can adaptively output different values, so a trained network model can generate three-dimensional models of various scenes. Controllability is manifested in that the three-dimensional spatial semantic feature can obtain features point by point, so the feature vectors at different positions can be obtained accordingly and corresponding operations can be performed.

[0090] Furthermore, the image generation solution of the disclosed embodiments can be an end-to-end rendering system, where the training of learnable parameters can be deployed on a cloud platform. For the same scene, users only need to input the desired perspective to obtain an image corresponding to that perspective. Users can also input different effects to achieve operations such as overlaying and moving objects in the scene.

[0091] In addition, the solution of the embodiment of the present disclosure can be used to segment the object in the image to generate a mask, and obtain the feature vector of the object in the three-dimensional space feature, thereby realizing scene editing.

[0092] It should be noted that although the steps of the method of the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all steps must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0093] Furthermore, this exemplary embodiment also provides an image generating device.

[0094] Figure 6 FIG2 schematically shows a block diagram of an image generating apparatus according to an exemplary embodiment of the present disclosure. Figure 6 The image generating device 6 according to an exemplary embodiment of the present disclosure may include a ray determining module 61 , a spatial feature determining module 63 , an information determining module 65 , and an image generating module 67 .

[0095] Specifically, the ray determination module 61 can be used to obtain the camera posture corresponding to the image to be generated, and determine the rays corresponding to each pixel point in the image to be generated based on the camera posture; the spatial feature determination module 63 can be used to determine the spatial semantic information of the sampling point on each ray based on the position information of the sampling point; the information determination module 65 can be used to determine the color information and density information of the sampling point based on the position information and spatial semantic information of the sampling point; the image generation module 67 can be used to use the color information and density information of each sampling point on the ray to determine the color information of the pixel point corresponding to the ray in the image to be generated, so as to generate the image to be generated.

[0096] According to an exemplary embodiment of the present disclosure, the spatial feature determination module 63 can be configured to perform: pre-constructing a spatial cube, and initializing and configuring spatial semantic information for each pixel point in the spatial cube; using the initialized spatial semantic information as a learnable parameter, and training the spatial semantic information using a training image of the same scene as the image to be generated to obtain the spatial semantic information corresponding to each pixel point in the spatial cube; when the sampling point is a pixel point in the spatial cube, determining the corresponding spatial semantic information based on the position information of the sampling point.

[0097] According to an exemplary embodiment of the present disclosure, the spatial feature determination module 63 can also be configured to perform: when the sampling point is not a pixel point in the spatial cube, determine the pixel points adjacent to the sampling point in the spatial cube; and interpolate the spatial semantic information corresponding to the pixel points adjacent to the sampling point to obtain the spatial semantic information of the sampling point.

[0098] According to an exemplary embodiment of the present disclosure, the information determination module 65 may be configured to execute: inputting the location information and spatial semantic information of the sampling points into a fully connected network, and determining the color information and density information of the sampling points through a prediction process of the fully connected network.

[0099] According to an exemplary embodiment of the present disclosure, referring to Figure 7 Compared with the image generating device 6 , the image generating device 7 may further include a training module 71 .

[0100] Specifically, the training module 71 can be configured to perform: pre-acquiring a training image of the same scene as the image to be generated; inputting the position information and spatial semantic information of the sampling points on the ray corresponding to the training image into the fully connected network; generating an intermediate image based on the color information and density information output by the fully connected network during the training process; comparing the intermediate image with a pre-selected real image, calculating the loss information, and using the loss information to adjust the network parameters in the fully connected network.

[0101] According to an exemplary embodiment of the present disclosure, the training images participating in the network training include training images whose camera poses are different from the camera poses corresponding to the to-be-generated images.

[0102] According to an exemplary embodiment of the present disclosure, the image generation module 67 can be configured to perform: using the density information of each sampling point on the ray as a weight, performing an integration operation on the corresponding color information to determine the color information of the pixel point corresponding to the ray in the image to be generated.

[0103] Since the functional modules of the image generating device in the embodiment of the present disclosure are the same as those in the above-mentioned method embodiment, they will not be described in detail here.

[0104] In exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, on which is stored a program product capable of implementing the aforementioned methods of this specification. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to execute the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present disclosure.

[0105] The program product for implementing the above-mentioned method according to an embodiment of the present disclosure may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0106] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical disk, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0107] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0108] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0109] The program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, and the like, as well as conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0110] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0111] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods, or program products. Therefore, various aspects of the present disclosure may be implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0112] Refer to the following Figure 8 800 according to this embodiment of the present disclosure will be described. Figure 8 The electronic device 800 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0113] like Figure 8 As shown, electronic device 800 is implemented as a general-purpose computing device. Components of electronic device 800 may include, but are not limited to, the aforementioned at least one processing unit 810, the aforementioned at least one storage unit 820, a bus 830 connecting various system components (including storage unit 820 and processing unit 810), and a display unit 840.

[0114] The storage unit stores program code that can be executed by the processing unit 810, causing the processing unit 810 to perform the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of the present disclosure. For example, the processing unit 810 can perform various steps of the image generation method according to an embodiment of the present disclosure.

[0115] The storage unit 820 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 8201 and / or a cache memory unit 8202 , and may further include a read-only memory unit (ROM) 8203 .

[0116] The storage unit 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, such program modules 8205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0117] Bus 830 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0118] The electronic device 800 can also communicate with one or more external devices 900 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 800, and / or any device that enables the electronic device 800 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 850. Furthermore, the electronic device 800 can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 860. As shown, the network adapter 860 communicates with other modules of the electronic device 800 via a bus 830. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 800, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0119] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0120] Furthermore, the figures above are merely illustrative of the processes included in the methods according to exemplary embodiments of the present disclosure and are not intended to be limiting. It is readily understood that the processes illustrated in the figures above do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0121] It should be noted that although several modules or units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0122] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing what is disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0123] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image generation method, characterized in that: include: Obtaining a camera pose corresponding to an image to be generated, and determining a ray corresponding to each pixel in the image to be generated according to the camera pose; For each sampling point on the ray, determining the spatial semantic information of the sampling point according to the position information of the sampling point; Determining color information and density information of the sampling point according to the position information and spatial semantic information of the sampling point; Determining color information of a pixel corresponding to the ray in the image to be generated by using the color information and density information of each sampling point on the ray, so as to generate the image to be generated; Wherein, determining the spatial semantic information of the sampling point according to the position information of the sampling point includes: Pre-building a spatial cube, wherein each pixel in the spatial cube is initialized with configuration spatial semantic information; Using the initialized spatial semantic information as a learnable parameter, the spatial semantic information is trained using a training image of the same scene as the image to be generated, so as to obtain spatial semantic information corresponding to each pixel in the spatial cube; In a case where the sampling point is a pixel point in the spatial cube, determining corresponding spatial semantic information according to position information of the sampling point; If the sampling point is not a pixel point in the spatial cube, determine pixel points in the spatial cube that are adjacent to the sampling point; and interpolate spatial semantic information corresponding to the pixel points adjacent to the sampling point to obtain spatial semantic information of the sampling point; The step of determining the color information and density information of the sampling point according to the position information and spatial semantic information of the sampling point includes: Inputting the position information and spatial semantic information of the sampling points into a fully connected network, and determining the color information and density information of the sampling points through a prediction process of the fully connected network; The image generation method further includes: Pre-acquiring a training image of the same scene as the image to be generated; Inputting the position information and spatial semantic information of the sampling points on the ray corresponding to the training image into the fully connected network; Generate an intermediate image based on the color information and density information output by the fully connected network during the training process; The intermediate image is compared with a pre-selected real image, loss information is calculated, and the network parameters in the fully connected network are adjusted using the loss information.

2. The image generation method according to claim 1, wherein: The training images participating in the network training include training images whose camera poses are different from the camera poses corresponding to the images to be generated.

3. The image generation method according to claim 1, wherein: Determining the color information of the pixel corresponding to the ray in the image to be generated by using the color information and density information of each sampling point on the ray includes: The density information of each sampling point on the ray is used as a weight to perform an integration operation on the corresponding color information to determine the color information of the pixel corresponding to the ray in the image to be generated.

4. An image generating device, characterized in that: include: A ray determination module is used to obtain a camera pose corresponding to the image to be generated, and determine a ray corresponding to each pixel in the image to be generated according to the camera pose; a spatial feature determination module, configured to determine, for each sampling point on the ray, spatial semantic information of the sampling point according to position information of the sampling point; An information determination module, configured to determine the color information and density information of the sampling point based on the position information and spatial semantic information of the sampling point; an image generation module, configured to determine the color information of the pixel points corresponding to the ray in the image to be generated by using the color information and density information of each sampling point on the ray, so as to generate the image to be generated; The spatial feature determination module is configured to: pre-construct a spatial cube, wherein each pixel in the spatial cube is initialized with spatial semantic information; use the initialized spatial semantic information as a learnable parameter, and train the spatial semantic information using a training image of the same scene as the image to be generated to obtain spatial semantic information corresponding to each pixel in the spatial cube; when the sampling point is a pixel in the spatial cube, determine the corresponding spatial semantic information based on the position information of the sampling point; when the sampling point is not a pixel in the spatial cube, determine the pixel in the spatial cube that is adjacent to the sampling point; Performing interpolation processing on spatial semantic information corresponding to pixel points adjacent to the sampling point to obtain spatial semantic information of the sampling point; The information determination module is configured to: input the position information and spatial semantic information of the sampling point into a fully connected network, and determine the color information and density information of the sampling point through a prediction process of the fully connected network; The image generating device further includes: A training module, configured to pre-acquire a training image of the same scene as the image to be generated; Inputting the position information and spatial semantic information of the sampling points on the ray corresponding to the training image into the fully connected network; Generate an intermediate image based on the color information and density information output by the fully connected network during the training process; The intermediate image is compared with a pre-selected real image, loss information is calculated, and the network parameters in the fully connected network are adjusted using the loss information.

5. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image generating method according to any one of claims 1 to 3 is implemented.

6. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to implement the image generating method according to any one of claims 1 to 3 by executing the executable instructions.

Images