Image generation method and apparatus, electronic device, and storage medium

Abstract

The present disclosure provides an image generation method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring at least two images at different viewing angles corresponding to a target to be generated, wherein each image comprises corresponding camera parameter information; performing fitting on the basis of the images to generate a parametric mesh model; defining N local coordinate systems on the basis of N vertices of the parametric mesh model, wherein N is an integer greater than or equal to 2; on the basis of the N local coordinate systems, determining local position information of position points in a three-dimensional space in each of the local coordinate systems, wherein the three-dimensional space is a space comprising the target to be generated; respectively defining a NeRF model on the basis of each local coordinate system among the N local coordinate systems; and generating a target generation image on the basis of at least two NeRF models and local position information in corresponding local coordinate systems. By fusing output results of at least two NeRF models, the quality of the generated target generation image can be effectively improved.

Description

Image generation method and device, electronic device, and storage medium

[0001] Cross-reference to Related Applications

[0002] The present disclosure claims priority to a Chinese patent application No. 202411006977.4, filed on July 25, 2024, and entitled "Image generation method and device", the entire content of which is incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates to the technical field of image generation, and in particular to an image generation method and device, an electronic device, and a storage medium. BACKGROUND

[0004] At present, 3D virtual object synthesis is becoming more and more common, but the existing technology is not realistic enough in local rendering effect, especially in the case of dynamic movement in the rendered local area, the local rendering effect is poor, and therefore needs to be improved. For example, in the process of generating a hand region of a human body, the relative positions between fingers can change at any time, resulting in poor display effect of the generated hand region. SUMMARY

[0005] The present disclosure provides an image generation method and device, an electronic device, and a storage medium, which can effectively improve the quality of target generated image generation.

[0006] According to an aspect of the present disclosure, an image generation method is provided, comprising:

[0007] obtaining at least two images of different perspectives corresponding to a target to be generated, wherein each image contains corresponding camera parameter information;

[0008] fitting a parameterized mesh model based on the images;

[0009] defining N local coordinate systems based on N vertices of the parameterized mesh model, wherein N is an integer greater than or equal to 2;

[0010] determining local position information of a position point in a three-dimensional space under each local coordinate system based on N local coordinate systems, wherein the three-dimensional space is a space containing the target to be generated;

[0011] defining one NeRF model for each of the N local coordinate systems based on the N local coordinate systems;

[0012] generating a target generated image based on at least two NeRF models and local position information under the corresponding local coordinate systems.

[0013] According to another aspect of the present disclosure, an image generation apparatus is provided, comprising:

[0014] an image acquisition module configured to acquire at least two images of different perspectives corresponding to a target to be generated, wherein each image contains corresponding camera parameter information;

[0015] a parameterized mesh model generation module configured to generate a parameterized mesh model based on the images;

[0016] a local coordinate system definition module configured to define N local coordinate systems based on N vertices of the parameterized mesh model, wherein N is an integer greater than or equal to 2;

[0017] a local position information determination module configured to determine local position information of a position point in a three-dimensional space in each of the local coordinate systems based on the N local coordinate systems; wherein the three-dimensional space is a space containing the target to be generated;

[0018] a NeRF model definition module configured to define a NeRF model based on each of the N local coordinate systems;

[0019] a target generated image generation module configured to generate a target generated image based on at least two NeRF models and the local position information in the corresponding local coordinate systems.

[0020] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0021] at least one processor; and

[0022] a memory communicatively connected to the at least one processor; wherein

[0023] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the image generation method according to any one of the embodiments of the present disclosure.

[0024] According to another aspect of the present disclosure, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to execute the image generation method according to any one of the embodiments of the present disclosure when executed.

[0025] The image generation scheme of the embodiments of the present disclosure includes: acquiring at least two images of different perspectives corresponding to a target to be generated, wherein each image contains corresponding camera parameter information; fitting a parameterized mesh model based on the images; defining N local coordinate systems based on N vertices of the parameterized mesh model, wherein N is an integer greater than or equal to 2; obtaining local position information of a position point in a three-dimensional space in each local coordinate system, wherein the three-dimensional space is a space containing the target to be generated; defining an NeRF model for each of the N local coordinate systems; and generating a target generated image based on at least two NeRF models and the local position information in the local coordinate systems. By fusing the output results of at least two NeRF models, the quality of the target generated image can be effectively improved.

[0026] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art without creative labor.

[0028] FIG. 1 is a flowchart of an image generation method according to an embodiment of the present disclosure;

[0029] FIG. 2 is a schematic diagram of an image acquisition principle in an image set according to an embodiment of the present disclosure;

[0030] FIG. 3 is a flowchart of an image generation method according to an embodiment of the present disclosure;

[0031] FIG. 4 is a flowchart of an image generation method according to an embodiment of the present disclosure;

[0032] FIG. 5 is a structural schematic diagram of an image generation device according to an embodiment of the present disclosure;

[0033] FIG. 6 is a structural schematic diagram of an electronic device implementing the image generation method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] In the following, the technical solutions in the embodiments of the present disclosure will be described clearly and completely in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative efforts should fall into the protection scope of the present disclosure.

[0035] It should be noted that the terms "first", "second" and the like in the description and claims of the present disclosure and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in other sequences than those illustrated or described herein. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0036] Embodiment one

[0037] FIG. 1 is a flowchart of an image generation method provided by Embodiment One of the present disclosure. The present embodiment can be applied to the case of generating an image. The method can be executed by an image generation device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device. As shown in FIG. 1, the method comprises:

[0038] S110, acquiring at least two images of different perspectives corresponding to a target to be generated, wherein each image contains corresponding camera parameter information.

[0039] In the embodiments of the present disclosure, the image acquisition device is controlled to acquire images of different perspectives of the target to be generated by image acquisition device. The image acquisition device can include various digital cameras, smart phones, portable wearable devices, unmanned aerial vehicles and other devices with optical image information acquisition function. Each image contains corresponding camera parameter information, which is the observation position coordinates and observation direction of the image acquisition device. The target to be generated can be a human body, a local area of a human body, other animals, vehicles or other target objects. It should be noted that the embodiments of the present disclosure do not limit the target to be generated.

[0040] For example, FIG. 2 is a schematic diagram of the acquisition principle of an image in an image set according to an embodiment of the present disclosure. As shown in FIG. 2, each image in the image set is an image taken by a camera with a fixed position in a global coordinate system (world coordinate system) with a fixed field of view direction. The coordinates of the camera position can be referred to as a viewpoint, and the RGB of each point in the image is the projection result of the RGB and transparency of each point on a certain ray with the viewpoint as the origin.

[0041] In S120, a parameterized mesh model is generated based on the image fitting.

[0042] In the embodiment of the present disclosure, the at least two images with different visual effects obtained in S110 are fitted to generate a parameterized mesh model, where the parameterized mesh model is a mesh graph used to represent the 3-dimensional topological structure of the target to be generated, and the parameterized mesh model can also be referred to as a 3D Mesh. The parameterized mesh model is composed of a plurality of vertex information and triangular face information, where the triangular face is a triangular face composed of every three vertices, and the vertex information includes the coordinates of each vertex, which can be represented as (x, y, z).

[0043] For example, a parameterized model such as MANO is used to generate a parameterized mesh model according to the images in the image set, where the parameterized mesh model contains vertex information and connection information of the parameterized mesh model. The vertex information is a ray passing through a key point in an image with the viewpoint of each image as the origin, and the intersection point of the same key point in different images is the position coordinates of the key point in the three-dimensional space, that is, the vertex of the parameterized mesh model. The connection information can be to connect a plurality of adjacent points according to a rule to form the parameterized mesh model, or to connect the key points to form the parameterized mesh model according to the image recognition technology after recognizing the target to be generated.

[0044] Optionally, the generating the parameterized mesh model based on the images comprises: determining a plurality of key points of the target to be generated in each of the images; and fitting the parameterized mesh model of the target to be generated in a three-dimensional space according to the key points of each of the images. In the embodiments of the present disclosure, a plurality of key points of the target to be generated in each of the at least two images of different perspectives are determined, wherein the key points can be feature points of the target to be generated. For example, the target to be generated is a face region of a human body, and the key points can be features of the face, such as a mouth, a left eye, a nose, a right eye, an ear, a brow, and the like. For another example, the target to be generated is a hand region of a human body, and the key points can be feature points of the fingers, such as a fingertip, a knuckle, and a palm, and the like. Optionally, the plurality of key points of the target to be generated in each of the images can be determined by a feature point recognition technology. The parameterized mesh model of the target to be generated in a three-dimensional space is fitted according to the key points of each of the images. For example, a parameterized mesh model (for example, MANO) can be fitted by a 2D photo, and the fitting method is not limited to matching the detected 2D key points of the fingers with each mesh vertex. Optionally, the parameterized mesh model is a triangular mesh.

[0045] S130, defining N local coordinate systems based on N vertices of the parameterized mesh model, wherein N is an integer greater than or equal to 2.

[0046] For example, all the vertices in the parameterized mesh model are determined, and then a local coordinate system is defined based on each of the vertices in the parameterized mesh model, wherein the number of the local coordinate systems is the same as the number of the vertices in the parameterized mesh model. For another example, all the vertices in the parameterized mesh model are determined, and then a part (at least two) of the vertices in the parameterized mesh model are randomly selected, and a local coordinate system is defined based on each of the selected part of the vertices, wherein the number of the local coordinate systems is the same as the number of the selected part of the vertices.

[0047] S140, determining local position information of a position point in a three-dimensional space in each of the local coordinate systems based on the N local coordinate systems, wherein the three-dimensional space is a space containing the target to be generated.

[0048] In the embodiments of the present disclosure, a three-dimensional space containing a target to be generated is determined, wherein the three-dimensional space is a space of a global coordinate system (world coordinate system), and is a space of a real world that has been defined when the images in the image set are captured. For example, a space region where a model defined when the images are captured is a cubic meter space with an origin as a vertex, there are 8 cameras, and the cameras are distributed at 8 vertices, the camera coordinates are (0, 0, 0), (1, 0, 0), (0, 1, 0), (0, 0, 1), (1, 1, 0), (1, 0, 1), (0, 1, 1), and (1, 1, 1), and the visual angle directions all point to the center of the cube, that is, any point in the three-dimensional model, the camera, or the space outside the camera is a point that can be defined in the global coordinate system of the three-dimensional space. For each image in the at least two images with different visual angles, local position information of a corresponding position point of each point in the image in the three-dimensional space in the N local coordinate systems is determined.

[0049] Optionally, the determining of the local position information of the position point in the three-dimensional space in each of the N local coordinate systems includes: determining global position information of the corresponding position point in the three-dimensional space in the global coordinate system according to the information of each point in the image; determining a linear transformation relationship between the global coordinate system and each of the N local coordinate systems; and converting the global position information of the position point in the three-dimensional space into the local position information of the position point in the local coordinate system based on the linear transformation relationship.

[0050] In the embodiments of the present disclosure, a global coordinate system is established, and position information of a corresponding position point in the three-dimensional space in the global coordinate system is determined according to the information of each point in the image. For the convenience of description, the position information of the corresponding position point in the three-dimensional space in the global coordinate system is referred to as global position information. It can be understood that the global position information of the position point in the three-dimensional space is position information in the same global coordinate system. For each of the N local coordinate systems, a linear transformation relationship between the global coordinate system and the local coordinate system is determined, wherein the linear transformation relationship is used to reflect a change relationship between two position points when the coordinate information of any position point in the global coordinate system is converted into the coordinate information of the corresponding position point in the local coordinate system. The global position information of the position point in the three-dimensional space is converted into the local position information of the position point in the corresponding local coordinate system according to the linear transformation relationship between the global coordinate system and the local coordinate system. In this way, the local position information of the corresponding position point of each point in the image in each of the local coordinate systems in the three-dimensional space can be obtained.

[0051] For example, the target to be generated is a hand region, and during the movement of the hand of the human body, the position and orientation of each vertex are dynamically changed, causing the global coordinates of a point to change even if the coordinates of the point in the local coordinate system remain unchanged (the point in the local coordinate system changes with the vertex). However, given the coordinates of the point in the local coordinate system, the characteristics of the point will not change regardless of the change in the global coordinates of the point. Assuming that the vertex corresponding to the local coordinate system is the tip of the index finger, the relative position relationship between the index finger nail and the tip of the index finger is fixed, and therefore the coordinates of the index finger nail in the local coordinate system of the tip of the index finger are unchanged, even if the index finger moves, causing the global coordinates of the tip of the index finger to change, and the characteristics given by the local coordinates will not change. Therefore, in the embodiments of the present disclosure, N local coordinate systems are defined based on N vertices of the parameterized mesh model, and the local position information of each point in the image corresponding to the position point in the three-dimensional space is determined based on the N local coordinate systems.

[0052] S150, define one NeRF model based on each of the N local coordinate systems.

[0053] In the embodiments of the present disclosure, one neural radiance field (NeRF) model is defined in each of the N local coordinate systems. It can be understood that the NeRF model corresponds to the local coordinate system one-to-one, and since there are N local coordinate systems, N corresponding NeRF models can be defined.

[0054] S160, generate a target generated image based on at least two NeRF models and the local position information in the corresponding local coordinate system.

[0055] In the embodiments of the present disclosure, the target generated image is generated according to N NeRF models and local position information of each position point in at least two images of different perspectives in a local coordinate system. Optionally, the target generated image is generated based on at least two NeRF models and local position information in the local coordinate system, including: generating an initial generated image based on each NeRF model and local position information in the local coordinate system; and fusing the initial generated images generated by the at least two NeRF models to generate the target generated image. For each local coordinate system of the N local coordinate systems, an initial generated image is generated according to the local position information of each point in the at least two images of different perspectives in a three-dimensional space corresponding to the position point in the local coordinate system and the NeRF model corresponding to the local coordinate system. It can be understood that the local position information of each point in the at least two images of different perspectives in a three-dimensional space corresponding to the position point in the local coordinate system is reconstructed by the NeRF model corresponding to the local coordinate system, and the reconstructed generated image is taken as the initial generated image. For example, the local position information of each point in the at least two images of different perspectives in a three-dimensional space corresponding to the position point in the local coordinate system is input into the NeRF model corresponding to the local coordinate system, and the image output by the NeRF model is taken as the initial generated image corresponding to the local coordinate system. In the above manner, the initial generated images corresponding to the N NeRF models can be obtained, and then the initial generated images generated by the N NeRF models are fused based on image fusion technology, and the fused generated image is taken as the target generated image. For example, the initial generated images output by the N NeRF models are linearly combined to generate the target generated image.

[0056] It should be emphasized that the NeRF model is essentially a function related to spatial position (x, y, z), observation direction, color information (RGB), and volume density information (transparency), the first two (spatial position (x, y, z) and observation direction) are input quantities, and the last two (color information and volume density information) are output quantities. The model training process is to sample points in the image, determine the coordinates, observation direction, RGB, and transparency of the sampling points, fit the model, then calculate the loss value of the model through the loss function, and correct the model based on the loss value, then sample, and repeatedly perform the above process until the desired trained model is obtained, for example, the loss value is within an acceptable range. The completion of model training indicates the completion of three-dimensional modeling. After the model is trained, given an observation point and an observation angle, the model will take the observation point as the origin to make a ray, and the RGB and transparency of each point (sampling point) on the ray will be superimposed to generate a pixel point on the initial generated image. Multiple pixel points generate an initial generated image, and at least two initial generated images generate a target generated image.

[0057] Optionally, the generating the initial generated image based on each NeRF model and the local position information under the corresponding local coordinate system comprises: determining, for the position point in the three-dimensional space, a vertex distance between the position point and a vertex of the parameterized mesh model corresponding to each local coordinate system; and generating the initial generated image based on each NeRF model and the local position information under the local coordinate system corresponding to the vertex distance being less than the first preset distance threshold.

[0058] In the embodiments of the present disclosure, since the local coordinate system is a corresponding coordinate system defined based on N vertices of the parameterized mesh model, and the local coordinate system corresponds to the N vertices of the parameterized mesh model one by one, therefore, the vertex distance between the position point and the vertex of the parameterized mesh model corresponding to each local coordinate system is calculated respectively by traversing the position point in the three-dimensional space. The initial generated image is generated based on each NeRF model and the local position information under the local coordinate system corresponding to the vertex distance being less than the first preset distance threshold. It can be understood that each NeRF model only reconstructs the position point based on the local position information under the local coordinate system corresponding to the vertex distance being less than the first preset distance threshold, thereby obtaining the initial generated position point output by each NeRF model, and the image composed of the initial generated position points is taken as the initial generated image. It can be understood that the initial generated position points output by all NeRF models are further fused to generate target generated position points, and the image composed of the target generated position points is taken as the target generated image.

[0059] Optionally, the generating the initial generated image based on each NeRF model and the local position information under the corresponding local coordinate system comprises: determining, for the position point in the three-dimensional space, a vertex distance between the position point and a vertex of the parameterized mesh model corresponding to each local coordinate system; selecting, from the local coordinate system, a first target local coordinate system corresponding to a first target vertex having a vertex distance less than a first preset distance threshold; and generating the initial generated image based on each NeRF model corresponding to the first target local coordinate system and the local position information under the first target local coordinate system corresponding to the vertex distance being less than the first preset distance threshold.

[0060] In the embodiments of the present disclosure, since the local coordinate system is a corresponding coordinate system defined based on N vertices of the parameterized mesh model, and the local coordinate system corresponds to the N vertices of the parameterized mesh model one by one, therefore, by traversing the position points in the three-dimensional space, the vertex distance of the position points and the vertices of the parameterized mesh model corresponding to each local coordinate system is calculated respectively. The local coordinate system corresponding to the first target vertex with a vertex distance less than a first preset distance threshold is selected from the N local coordinate systems as a first target local coordinate system, wherein the first target local coordinate system can be one or multiple. An initial generated image is generated according to the NeRF model corresponding to each first target local coordinate system and the local position information with a vertex distance less than the first preset distance threshold under the first target local coordinate system. It can be understood that the NeRF model corresponding to each first local position coordinate system only performs position point reconstruction on the local position information with a vertex distance less than the first preset distance threshold under the first local position coordinate system, so as to obtain the initial generated position points output by the NeRF model corresponding to each first target local coordinate system, and the image composed of the initial generated position points is taken as the initial generated image. It can be understood that the initial generated position points output by all the NeRF models corresponding to the first local coordinate system are further fused to generate target generated position points, and the image composed of the target generated position points is the target generated image.

[0061] The image generation method of the embodiments of the present disclosure acquires at least two images of different perspectives corresponding to a target to be generated, wherein each image contains corresponding camera parameter information; a parameterized mesh model is fitted based on the images; N local coordinate systems are defined based on N vertices of the parameterized mesh model, wherein N is an integer greater than or equal to 2; local position information of a position point in a three-dimensional space under each local coordinate system is determined based on the N local coordinate systems; wherein the three-dimensional space is a space containing the target to be generated; one NeRF model is defined based on each local coordinate system of the N local coordinate systems; and a target generated image is generated based on at least two NeRF models and local position information under the corresponding local coordinate systems. By fusing the output results of at least two NeRF models, the quality of the target generated image can be effectively improved.

[0062] Embodiment Two

[0063] FIG. 3 is a flowchart of an image generation method provided by Embodiment Two of the present disclosure, as shown in FIG. 3, the method comprises:

[0064] S310, at least two images of different perspectives corresponding to a target to be generated are acquired, wherein each image contains corresponding camera parameter information.

[0065] S320, a parameterized mesh model is fitted based on the images.

[0066] S330, defining N local coordinate systems based on N vertices of the parameterized mesh model, where N is an integer greater than or equal to 2.

[0067] S340, determining local position information of a position point in a three-dimensional space in each of the N local coordinate systems based on the N local coordinate systems; wherein the three-dimensional space is a space containing the target to be generated.

[0068] S350, defining one NeRF model for each of the N local coordinate systems based on each of the N local coordinate systems.

[0069] S360, determining a vertex distance between the position point and a vertex of the parameterized mesh model corresponding to each of the local coordinate systems for the position point in the three-dimensional space.

[0070] S370, setting a model weight of the NeRF model corresponding to each of the local coordinate systems according to the vertex distance.

[0071] In the embodiments of the present disclosure, for each position point in the three-dimensional space, a corresponding model weight of the NeRF model corresponding to each local coordinate system is set according to a vertex distance between the position point and a vertex of the parameterized mesh model corresponding to each local coordinate system. The model weight is negatively correlated with the corresponding vertex distance, that is, the larger the vertex distance, the smaller the corresponding model weight. The sum of the model weights of all the NeRF models may or may not be 1.

[0072] S380, fusing the initial generated images generated by the at least two NeRF models based on the at least two NeRF models and the corresponding model weights, to generate a target generated image.

[0073] In the embodiments of the present disclosure, at least two NeRF models are selected from the N NeRF models according to the model weights, for the convenience of description, the selected at least two NeRF models can be referred to as target NeRF models, and the local position point information is input into each target NeRF model respectively, to obtain the initial generated image output by each target NeRF model, and then the initial generated image is fused based on the at least two target NeRF models and the corresponding model weights to generate a target generated image. For example, for a certain position point, the model weight of the NeRF model 1 is 90, the model weight of the NeRF model 2 is 80, and the model weights of other NeRF models do not exceed 10, then the NeRF model 1 and the NeRF model 2 can be used as target NeRF models. The local position information corresponding to the position point is analyzed based on the NeRF model 1 and the NeRF model 2 respectively, to obtain the initial generated position point corresponding to the position point, and then the initial generated position points output by the NeRF model 1 and the NeRF model 2 are fused based on the model weights of the NeRF model 1 and the NeRF model 2, to generate a target generated position point. Similarly, the target generated position point corresponding to each position point can be determined in the above manner, and the target generated image is formed based on all target generated position points. Optionally, the local position point information can be input into the N NeRF models respectively, to obtain the initial generated image output by the N NeRF models respectively, and then the initial generated image is fused based on the N NeRF models and the corresponding model weights to generate a target generated image.

[0074] It can be understood that in the embodiments of the present disclosure, although there are also 8 pictures, if fitting is performed in one model, firstly, the space region to be fitted is large, and the number of sampled points is large, and if the same accuracy is to be achieved, the calculation amount is large, and the training time is long; secondly, in the dynamic modeling process, sampling training needs to be started from zero for each action moment (each action will cause the global coordinates to change), the calculation amount is large, and the time is long. Through the technical scheme provided in the embodiments of the present disclosure, N local coordinate systems and N NeRF models are defined, for example, one local coordinate system and a corresponding NeRF model are defined for each finger joint, and the position of the finger joint in the local coordinate system is basically unchanged regardless of the position of the finger joint in the three-dimensional space, so that when the local model is trained, the finger joint at the position can have better reconstruction effect, especially in the dynamic modeling process, the finger joint at the position has little change for each action moment, and sampling training does not need to start from zero, so that the training amount is greatly reduced, and the training efficiency is improved. At the same time, each local model only trains the points within a certain distance range around the origin, which greatly reduces the training amount, and the local rendering effect can also be improved by increasing the sampling density and / or the sampling times in the local space.

[0075] The image generation method of the embodiments of the present disclosure can guarantee the locality of each NeRF model and reduce the total calculation amount of image generation, and improve the quality of the generated target image by fusing the output results of at least two NeRF models.

[0076] Embodiment three

[0077] FIG. 4 is a flowchart of an image generation method provided by embodiment three of the present disclosure. As shown in FIG. 4, the method comprises:

[0078] S410, acquiring at least two images of different perspectives corresponding to a target to be generated, wherein each image contains corresponding camera parameter information.

[0079] S420, fitting a parameterized mesh model based on the images.

[0080] S430, defining N local coordinate systems based on N vertices of the parameterized mesh model, wherein N is an integer greater than or equal to 2.

[0081] S440, determining local position information of a position point in a three-dimensional space under each local coordinate system based on N local coordinate systems; wherein the three-dimensional space is a space containing the target to be generated.

[0082] S450, defining one NeRF model for each of the N local coordinate systems.

[0083] S460, for the position point in the three-dimensional space, determining the vertex distance between the position point and the vertex of the parameterized mesh model corresponding to each local coordinate system.

[0084] In the embodiments of the present disclosure, since the local coordinate system is a corresponding coordinate system defined based on the N vertices of the parameterized mesh model, and the local coordinate system corresponds one-to-one to the N vertices of the parameterized mesh model, therefore, for each position point in the three-dimensional space, the vertex distance between the position point and the vertex of the parameterized mesh model corresponding to each local coordinate system is calculated respectively.

[0085] S470, selecting a second target local coordinate system corresponding to a second target vertex with a vertex distance less than a second preset distance threshold from the local coordinate systems.

[0086] In the embodiments of the present disclosure, the local coordinate system corresponding to the second target vertex with a vertex distance less than a second preset distance threshold is selected from the N local coordinate systems as a second target local coordinate system, wherein the second target local coordinate system can be one or multiple. It should be noted that the present disclosure does not limit the size relationship between the first preset distance threshold and the second preset distance threshold.

[0087] S480, generating a target generated image based on the NeRF model corresponding to at least two second target local coordinate systems and local position information under the second target local coordinate system.

[0088] In the embodiments of the present disclosure, the target generated image is generated according to the NeRF model corresponding to at least two second target local coordinate systems and the local position information under the second target local coordinate system. It can be understood that the NeRF model corresponding to each second local position coordinate system reconstructs the local position information under the second local position coordinate system to obtain the initial generated position points output by the NeRF model corresponding to each second target local coordinate system, and then the initial generated position points output by the NeRF model corresponding to all second local coordinate systems are fused to generate target generated position points, and then the image composed of each target generated position point is taken as the target generated image.

[0089] The image generation method of the embodiments of the present disclosure can guarantee the locality of each NeRF model and reduce the total calculation amount of image generation, and at the same time, through the fusion of the output results of at least two NeRF models, the quality of the target generated image generation is improved.

[0090] In some embodiments, the determining, based on the N local coordinate systems, the local position information of each position point in the three-dimensional space under each local coordinate system, further includes: determining, based on the N local coordinate systems, relative local pose information of at least two local coordinate systems; or determining, based on the N local coordinate systems, the local pose information of the position point in the three-dimensional space under each local coordinate system; and the generating, based on at least two NeRF models and the local position information under the corresponding local coordinate system, a target generated image, further includes: each NeRF model taking the local pose information as an additional input signal, and generating a target generated image based on at least two NeRF models and the local position information and the local pose information under the corresponding local coordinate system. The advantage of such a setting is that each local NeRF takes the pose as an additional input signal, and the corresponding relationship between the pose of each position point and the joint activity angle around it can be obtained, so that the subtle changes of the local surface of the target to be generated under different poses can be reconstructed, and the quality of the target generated image can be further improved.

[0091] In the embodiments of the present disclosure, for at least two of the N local coordinate systems, the relative local pose information between the at least two local coordinate systems is determined, for example, the relative pose information between the coordinate origins (including the directions of the three coordinate axes) of the at least two local coordinate systems can be taken as the relative local pose information between the at least two local coordinate systems. The relative local pose information between the at least two local coordinate systems can be referred to as first local pose information. Alternatively, for each image in all images of different perspectives, the local pose information of each point in the image corresponding to the position point in the three-dimensional space in the N local coordinate systems is determined respectively. For example, the global pose information of each point in the image corresponding to the position point in the three-dimensional space in the global coordinate system is determined, and the global pose information of the position point in the three-dimensional space is converted into the local pose information of the position point in the corresponding local coordinate system according to the linear transformation relationship between the global coordinate system and each local coordinate system. The local pose information of the position point in the three-dimensional space in each local coordinate system can be referred to as second local pose information. The local pose information (i.e., the first local pose information and / or the second local pose information) and the local position information are input into at least two NeRF models respectively, the initial generated images output by the NeRF models are obtained respectively, and the initial generated images corresponding to the at least two NeRF models are fused to generate the target generated image. For example, taking the first local pose information as an additional input signal of each NeRF model, and taking two local coordinate systems as the fingertip local coordinate system and the first knuckle local coordinate system, the relative angle information between the fingertip local coordinate system and the first knuckle local coordinate system is determined, the wrinkle condition of the skin under different joint angles is obtained, and a training model is generated to realize accurate generation of local skin under unsampled joint angles. For another example, the target to be generated is a hand region of a human body, and the local pose information such as the angle pose between a point on the local skin surface and the surrounding joints is taken as an additional input signal of each NeRF model, so that the target generated image generated contains the change of the local skin surface of the hand under different poses, for example, the local skin wrinkle when the finger joints are curled, and the quality of the target generated image can be further improved. It can be understood that the motion of the target to be generated is continuous, and the sampling is discrete, for example, the joints in the motion are photographed and sampled at 30 frames per second to generate images at each time, and the joint angles in each time image are different and discontinuous. In the embodiments of the present disclosure, the training model is better generated by additionally inputting the local pose information to realize image generation under continuous joint motion angles.

[0092] Embodiment four

[0093] FIG. 5 is a structural schematic diagram of an image generation device provided in an embodiment of the present disclosure. As shown in FIG. 5, the device comprises:

[0094] An image acquisition module 510 is configured to acquire at least two images of different perspectives of a target to be generated, wherein each image contains corresponding camera parameter information;

[0095] A parameterized mesh model generation module 520 is configured to generate a parameterized mesh model based on the images;

[0096] A local coordinate system definition module 530 is configured to define N local coordinate systems based on N vertices of the parameterized mesh model, wherein N is an integer greater than or equal to 2;

[0097] A local position information determination module 540 is configured to determine local position information of a position point in a three-dimensional space in each local coordinate system based on the N local coordinate systems;

[0098] A NeRF model definition module 550 is configured to define a NeRF model for each of the N local coordinate systems;

[0099] A target generated image generation module 560 is configured to generate a target generated image based on at least two NeRF models and local position information in the corresponding local coordinate systems.

[0100] Optionally, the parameterized mesh model generation module is configured to:

[0101] determine a plurality of key points of the target to be generated in each image;

[0102] fit to generate the parameterized mesh model of the target to be generated in a three-dimensional space according to the key points of each image.

[0103] Optionally, the parameterized mesh model is a triangular mesh.

[0104] Optionally, the local position information determination module is configured to:

[0105] determine global position information of a corresponding position point in a global coordinate system in the three-dimensional space according to information of each point in the image;

[0106] determine a linear transformation relationship between the global coordinate system and each local coordinate system in the N local coordinate systems;

[0107] convert the global position information of the position point in the three-dimensional space into local position information of each position point in the local coordinate system based on the linear transformation relationship.

[0108] Optionally, the target generated image generation module includes:

[0109] An initial generated image generation unit is configured to generate an initial generated image based on each NeRF model and local position information corresponding to the local coordinate system;

[0110] A target generated image generation unit is configured to fuse the initial generated images generated by the at least two NeRF models to generate a target generated image.

[0111] Optionally, the initial generated image generation unit is configured to:

[0112] For the position point in the three-dimensional space, a vertex distance between the position point and a vertex of the parameterized mesh model corresponding to each local coordinate system is determined respectively;

[0113] A first target local coordinate system corresponding to a first target vertex with a vertex distance less than a first preset distance threshold is selected from the local coordinate systems;

[0114] The initial generated image is generated based on the NeRF model corresponding to each first target local coordinate system and local position information corresponding to the first target local coordinate system and having a vertex distance less than the first preset distance threshold.

[0115] Optionally, the target generated image generation module is configured to:

[0116] For the position point in the three-dimensional space, a vertex distance between the position point and a vertex of the parameterized mesh model corresponding to each local coordinate system is determined respectively;

[0117] A model weight of the NeRF model corresponding to each local coordinate system is set according to the vertex distance;

[0118] The initial generated images generated by the at least two NeRF models are fused based on the at least two NeRF models and the model weights corresponding to the at least two NeRF models to generate a target generated image.

[0119] Optionally, the target generated image generation module is configured to:

[0120] For the position point in the three-dimensional space, a vertex distance between the position point and a vertex of the parameterized mesh model corresponding to each local coordinate system is determined respectively;

[0121] A second target local coordinate system corresponding to a second target vertex with a vertex distance less than a second preset distance threshold is selected from the local coordinate systems;

[0122] generate a target generated image based on the at least two NeRF models and the local position information corresponding to the second target local coordinate system.

[0123] Optionally, the local position information determination module is further configured to:

[0124] determine relative local pose information of at least two of the local coordinate systems based on the N local coordinate systems; or

[0125] determine local pose information of the position point in the three-dimensional space in each of the local coordinate systems based on the N local coordinate systems;

[0126] The target generated image generation module is further configured to:

[0127] Each of the NeRF models takes the local pose information as an additional input signal, and generates a target generated image based on at least two of the NeRF models and the local position information and the local pose information corresponding to the local coordinate system.

[0128] The image generation apparatus provided by the embodiments of the present disclosure can perform the image generation method provided by any of the embodiments of the present disclosure, and has the corresponding function modules and beneficial effects of performing the method.

[0129] Embodiment five

[0130] FIG. 6 shows a structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (such as headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0131] As shown in FIG. 6, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., communicatively connected to the at least one processor 11, where the memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded into the random access memory (RAM) 13 from the storage unit 18. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0132] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, a speaker, etc., a storage unit 18, such as a magnetic disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0133] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the image generation method.

[0134] In some embodiments, the image generation method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the image generation method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the image generation method by any other appropriate means, such as by means of firmware.

[0135] The various embodiments of the systems and techniques described above can be implemented in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a load programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0136] Computer programs used to implement the methods of the present disclosure can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, can cause instructions defined in the flow charts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine, or entirely on a remote machine or server.

[0137] In the context of the present disclosure, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0138] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0139] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0140] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0141] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, without departing from the desired results of the technical solutions of the present disclosure, and are not limited herein.

[0142] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image generation method, wherein, The method comprises: obtaining at least two images of different perspectives corresponding to a target to be generated, wherein each image contains corresponding camera parameter information; fitting a parameterized mesh model based on the images; defining N local coordinate systems based on N vertices of the parameterized mesh model, wherein N is an integer greater than or equal to 2; determining local position information of a position point in a three-dimensional space in each local coordinate system based on the N local coordinate systems, wherein the three-dimensional space is a space containing the target to be generated; defining one NeRF model for each of the N local coordinate systems; generating a target generated image based on at least two NeRF models and local position information in the corresponding local coordinate systems.

2. The method of claim 1, wherein, The method of fitting a parameterized mesh model based on the images comprises: determining a plurality of key points of the target to be generated in each image; fitting the parameterized mesh model of the target to be generated in a three-dimensional space according to the key points of each image.

3. The method of claim 2, wherein, The parameterized mesh model is a triangular mesh.

4. The method of claim 1, wherein, The method of determining local position information of a position point in a three-dimensional space in each local coordinate system based on the N local coordinate systems comprises: determining global position information of a corresponding position point in a global coordinate system in the three-dimensional space according to information of each point in the image; determining a linear transformation relationship between the global coordinate system and each local coordinate system in the N local coordinate systems; converting the global position information of the position point in the three-dimensional space into local position information of the position point in the local coordinate system based on the linear transformation relationship.

5. The method of claim 1, wherein, The method of generating a target generated image based on at least two NeRF models and local position information in the corresponding local coordinate systems comprises: generating an initial generated image based on each NeRF model and local position information in the corresponding local coordinate system; fusing the initial generated images generated by the at least two NeRF models to generate a target generated image.

6. The method of claim 5, wherein, The method of generating an initial generated image based on each NeRF model and local position information in the corresponding local coordinate system comprises: determining a vertex distance between the position point and a vertex of the parameterized mesh model corresponding to each local coordinate system for the position point in the three-dimensional space; generating the initial generated image based on the vertex distance between each NeRF model and local position information in the corresponding local coordinate system being less than the first preset distance threshold.

7. The method of claim 5, wherein, The method of generating an initial generated image based on each NeRF model and local position information in the corresponding local coordinate system comprises: determining a vertex distance between the position point and a vertex of the parameterized mesh model corresponding to each local coordinate system for the position point in the three-dimensional space; selecting a first target local coordinate system corresponding to a first target vertex with a vertex distance less than a first preset distance threshold from the local coordinate systems; Generate the initial generated image based on the NeRF model corresponding to each of the first target local coordinate systems and the local position information under the first target local coordinate system corresponding to the vertex distance less than the first preset distance threshold.

8. The method of claim 1, wherein, The method for generating a target generated image based on at least two NeRF models and local position information under the local coordinate system comprises: For the position point in the three-dimensional space, the vertex distance between the position point and the vertex of the parameterized mesh model corresponding to each local coordinate system is determined respectively; The model weight of the NeRF model corresponding to each local coordinate system is set according to the vertex distance; The initial generated images generated by at least two NeRF models are fused based on at least two NeRF models and the model weight corresponding to the NeRF model to generate a target generated image.

9. The method of claim 1, wherein, The method for generating a target generated image based on at least two NeRF models and local position information under the local coordinate system comprises: For the position point in the three-dimensional space, the vertex distance between the position point and the vertex of the parameterized mesh model corresponding to each local coordinate system is determined respectively; A second target local coordinate system corresponding to a second target vertex with a vertex distance less than a second preset distance threshold is selected from the local coordinate system; A target generated image is generated based on the NeRF model corresponding to at least two second target local coordinate systems and the local position information under the second target local coordinate system corresponding to the NeRF model.

10. The method of any one of claims 1-9, wherein, The method for determining the local position information of the position point in the three-dimensional space under each local coordinate system based on N local coordinate systems further comprises: The relative local pose information of at least two local coordinate systems is determined based on N local coordinate systems; or The local pose information of the position point in the three-dimensional space under each local coordinate system is determined based on N local coordinate systems; The method for generating a target generated image based on at least two NeRF models and local position information under the local coordinate system further comprises: Each NeRF model takes the local pose information as an additional input signal, and generates a target generated image based on at least two NeRF models, the local position information under the local coordinate system corresponding to the NeRF model, and the local pose information.

11. The method of claim 1, wherein, The parameterized mesh model comprises vertex information and connection information. The vertex information is that a ray is drawn through a key point in an image with the viewpoint of each image as the origin, and the intersection point of the same key point in different images is the position coordinate of the key point in the three-dimensional space, that is, the vertex of the parameterized mesh model. The connection information is that adjacent points are connected according to a rule to form a parameterized mesh model, or key points are connected to form a parameterized mesh model after a target to be generated is identified through image recognition technology.

12. The method of claim 1, wherein, The method for defining N local coordinate systems based on N vertices of the parameterized mesh model comprises: All vertices in the parameterized mesh model are determined; Each of all vertices in the parametric mesh model is defined as a local coordinate system; wherein the number of local coordinate systems is the same as the number of vertices contained in the parametric mesh model; or, All vertices in the parametric mesh model are determined; Part of the vertices in the parametric mesh model are selected; wherein the selected part of the vertices includes at least two vertices; Each of the selected part of the vertices is defined as a local coordinate system; wherein the number of local coordinate systems is the same as the number of selected part of the vertices.

13. The method of claim 1, wherein, The initial generated images generated by the at least two NeRF models are fused to generate a target generated image, including: The initial generated images output by the at least two NeRF models are linearly combined to generate a target generated image.

14. The method of claim 1, wherein, The training process of the NeRF model includes: Sampling points in the image; Fitting the model by determining the coordinates, viewing direction, RGB and transparency of the sampling points; Calculating the loss value of the model by the loss function, and correcting the model based on the loss value; Repeat the above process until the model training is completed.

15. The method of claim 10, wherein, The relative local pose information between the at least two local coordinate systems is determined based on the N local coordinate systems, including: The relative pose information between the coordinate origins of the at least two local coordinate systems containing three coordinate axis directions is taken as the relative local pose information between the at least two local coordinate systems.

16. The method of claim 10, wherein, The local pose information of the position point in the three-dimensional space under each local coordinate system is determined based on the N local coordinate systems, including: Determine the global pose information of the corresponding position point of each point in the image in the three-dimensional space under the global coordinate system; According to the linear transformation relationship between the global coordinate system and each local coordinate system, the global pose information of the position point in the three-dimensional space is converted into the local pose information of the position point under the corresponding local coordinate system.

17. The method of claim 10, wherein, The relative local pose information between the at least two local coordinate systems is taken as the first local pose information, and the local pose information of the position point in the three-dimensional space under each local coordinate system is taken as the second local pose information. The target generated image is generated based on the at least two NeRF models and the local position information under the corresponding local coordinate system, including: The first local pose information and / or the second local pose information and the local position information are input into the at least two NeRF models respectively, and the initial generated images output by the NeRF models are obtained respectively; The initial generated images corresponding to the at least two NeRF models are fused to generate a target generated image.

18. An image generation apparatus, wherein, It includes: An image acquisition module for acquiring at least two images of different perspectives corresponding to a target to be generated, wherein each image contains corresponding camera parameter information; A parametric mesh model generation module for generating a parametric mesh model based on the images; A local coordinate system definition module for defining N local coordinate systems based on N vertices of the parametric mesh model, wherein N is an integer greater than or equal to 2; A local position information determination module is configured to determine local position information of a position point in a three-dimensional space in each of the local coordinate systems based on the N local coordinate systems; wherein the three-dimensional space is a space containing the target to be generated; A NeRF model definition module is configured to define one NeRF model for each of the N local coordinate systems respectively based on the local coordinate system. A target generation image generation module is configured to generate a target generation image based on at least two of the NeRF models and the local position information in the corresponding local coordinate systems.

19. An electronic device, comprising: It comprises: A processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, the machine readable instructions are executed by the processor to execute the steps of the method in any one of claims 1 to 17.

20. A computer readable storage medium, wherein, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to execute the steps of the method in any one of claims 1 to 17.

Images