Action image generation method, model building method, device and storage medium

Abstract

The present application provides an action image generation method, a model construction method, a computer device and a storage medium, wherein the model construction method comprises: obtaining a first image set and a second image set of a target person, wherein the first image set and the second image set include sequence images of the target person's actions; performing three-dimensional reconstruction according to the sequence images of the first image set to obtain a plurality of first 3D models; obtaining a texture map of the target person, mapping the plurality of first 3D models according to the texture map to obtain a plurality of texture maps; projecting the plurality of first 3D models to obtain a plurality of 2D projection images; and constructing an action generation model according to the sequence images, 2D projection images and texture maps of the second image set. The action generation model can generate more realistic action images, that is, the action images include more details, thereby improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an action image generation method, a model building method, a computer device and a storage medium. Background Art

[0002] At present, motion generation solutions generally use human key points or posture estimation to achieve human motion generation, but this solution is difficult to generate action images with high-definition details, such as less facial information. In addition, different human bodies have different shapes, such as different heights and fatness, and these details cannot be reflected based on human key points or posture estimation. Summary of the invention

[0003] The present application provides a method for constructing an action generation model, a method for generating an action image, a device and a storage medium, which can generate an action image including more details.

[0004] In a first aspect, the present application provides a method for constructing an action generation model, the method comprising:

[0005] Acquire a training set of a target person, the training set comprising a first image set and a second image set, the first image set and the second image set comprising sequence images about actions of the target person;

[0006] Performing three-dimensional reconstruction based on the sequence images of the first image set to obtain a plurality of first 3D models;

[0007] Acquire a texture map of the target person, and map the multiple first 3D models according to the texture map to obtain multiple texture maps;

[0008] Projecting the multiple first 3D models to obtain multiple 2D projection images;

[0009] An action generation model is constructed according to the sequence images of the second image set, the 2D projection image and the texture map.

[0010] In a second aspect, the present application further provides a method for generating an action image, the method comprising:

[0011] Acquire multiple images of a user, and determine a texture map of the user based on the multiple images;

[0012] Perform three-dimensional reconstruction based on the multiple images to obtain a 3D model of the user;

[0013] Mapping the 3D model according to the texture map to obtain a texture map;

[0014] An action image of the user is generated based on the multiple images and the texture map.

[0015] In a third aspect, the present application further provides a computer device, the computer device comprising:

[0016] Memory and processor;

[0017] Wherein, the memory is connected to the processor and is used to store programs;

[0018] The processor is used to implement the steps of the method for constructing an action generation model as described in any one of the embodiments of the present application, or to implement the steps of the method for generating an action image as described in any one of the embodiments of the present application, by running the program stored in the memory.

[0019] In a fourth aspect, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the steps of a method for constructing an action generation model as described in any one of the embodiments of the present application, or implements the steps of a method for generating an action image as described in any one of the embodiments of the present application.

[0020] The method for constructing an action generation model, the method for generating an action image, the device and the storage medium disclosed in the present application can generate more realistic action images by changing the input conditions of the network model, that is, the generated action images include more details, such as more facial information, human body shape, clothing and other details, thereby improving the user experience.

[0021] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 is a schematic flow chart of the steps of a method for constructing an action generation model provided in an embodiment of the present application;

[0024] Figure 2 This is a schematic diagram of the effect of a texture map provided in an embodiment of the present application;

[0025] Figure 3 is a schematic block diagram of an action generation model provided by an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of a first branch network model provided in an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of a first branch network model provided in an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of a second branch network model provided in an embodiment of the present application;

[0029] Figure 7 is a schematic diagram of a backbone network model provided by an embodiment of the present application;

[0030] Figure 8 is a schematic diagram of a corresponding construction scene of an action generation model provided in an embodiment of the present application;

[0031] Fig. 9 is a schematic diagram of the steps of a method for generating an action image provided by an embodiment of the present application;

[0032] Fig.10 It is a schematic block diagram of a computer device provided in an embodiment of the present application.

[0033] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0036] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0037] It should be understood that, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially identical functions and effects. For example, the first image set and the second image set are only used to distinguish between different image sets, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0038] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0039] To facilitate understanding of the embodiments of the present application, some words involved in the embodiments of the present application are briefly explained below.

[0040] 1. Action generation: It refers to generating a series of actions of a character based on a static character image. For example, given a static character image, a motion video is automatically generated based on a series of skeletal joint motion sequences (also called motion driving sources) using a trained deep learning model, so that the character can move according to the given motion pattern, such as dancing, exercising, etc.

[0041] 2. Texture mapping: In computer graphics, texture mapping is a technique that uses images, functions, or other data sources to change the appearance of an object's surface. Texturing is an efficient technique for "modeling" the properties of an object's surface. The pixels in an image texture are usually called texels. By applying the projection function to points in space, a set of parameter space values ​​is obtained. The parameter space values ​​are numerical values ​​about the texture. One or more corresponding functions are used to convert the parameter-space values ​​to the texture space. This process is called mapping, also known as texture mapping. Accurately correspond each point on the 2D projected image to the surface of the 3D model object, and perform smooth image interpolation processing at the gaps between points. This is UV mapping.

[0042] 3. Warp transform: also called image affine transform, Warp transform actually infers the image information at the next moment through optical flow and current image information.

[0043] 4. 3D reconstruction: also known as three-dimensional reconstruction, specifically refers to the establishment of a mathematical model suitable for computer representation and processing of three-dimensional objects, so as to process, operate and analyze their properties in a computer environment. It is also a key technology for establishing virtual reality that expresses the objective world in a computer. Common 3D reconstruction algorithms include HMR, Simplify-x, and TotalCapture. Among them, HMR is an end-to-end Human Mesh Recovery framework that reconstructs a SMPL 3D mesh from an RGB bitmap containing a human body and attempts to reproject it back onto the image; the Simplify-x algorithm is also called the SMPL-X model (SMPL expression), which is based on SMPL and follows the SMPLify method. It uses OpenPose to estimate two-dimensional image features "bottom-up". OpenPose detects the joints of the body, hands, feet, and facial features, and can better capture the body's movements, including gestures and expressions; Total Capture, specifically "A 3D Deformation Model for Tracking Faces, Hands, and Bodies", can simultaneously estimate the postures of the face, body, and hands, and fuse them into a three-dimensional model.

[0044] 5. Centroid coordinates: Also known as barycentric coordinates, they are Barycentric Coordinates in computer graphics and a concept of 3D rendering. The 3D model of the human body can be divided into many triangular facets, and each vertex of the triangular facet has a one-to-one correspondence. When driving an action, in addition to the action, there is also rendering. When there is a 3D action, the rendering method is to find it from the 2D image of the reference image set. At this time, the correspondence of the vertices is required. Each point on the triangular facet has physical meaning. After the vertices are corresponding, the pixels outside the vertices are filled in by Barycentric Coordinates calculation. In the embodiments of the present application, centroid coordinates are mainly used to calculate the flow mapping relationship of pixels between images of two image sets.

[0045] 6. GAN network: GAN network stands for Generative Adversarial Network, which is a machine learning method. There are two networks in the GAN network, namely the generative network and the discriminative network, which can also be called the generative model and the discriminative model. Take the generation of pictures as an example. For example, the ultimate goal is to use the GAN network to generate anime avatars. The generative model is a network that generates pictures. It accepts a random noise z and then generates pictures through this noise. The generated data is recorded as G(z). The discriminative model is a discriminative network that determines whether a picture is "real" (whether it is fabricated). Its input parameter is x, x represents a picture, and the output D(x) represents the probability that x is a real picture. If it is 1, it means that it is definitely a real picture, and if the output is 0, it means that it cannot be a real picture. In the training process, the goal of the generative network is to generate fake pictures to deceive the discriminative network, and the goal of the discriminative network is to be able to distinguish whether a certain picture is generated, which becomes a game process. At the same time, the capabilities of the generative model and the discriminative model are gradually improved during the training process until the training process reaches a balance when the discriminative network can no longer distinguish between real pictures and forged pictures.

[0046] At present, motion generation solutions generally use human key points or posture estimation to achieve human motion generation, but this solution is difficult to generate action images with high-definition details, such as less facial information. In addition, different human bodies have different shapes, such as different heights and fatness, and these details cannot be reflected based on human key points or posture estimation.

[0047] To this end, the embodiments of the present application provide a method for constructing an action generation model, an action image generation method, a computer device, and a storage medium. Among them, the trained action generation model can generate action images with more details when used in the user's action generation scheme, such as more facial information, human body shape, clothing, hair and other details, thereby improving the user experience.

[0048] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0049] See also Figure 1 , Figure 1 A method for constructing an action generation model provided in an embodiment of the present application can be applied to a computer device, specifically a computer device for building a dedicated model, such as a GPU.

[0050] like Figure 1 As shown, the method for constructing the action generation model includes steps S101 to S105.

[0051] S101 . Acquire a first image set and a second image set of a target person, wherein the first image set and the second image set include sequence images of actions of the target person.

[0052] Since multiple images can form a sequence, specifically, a sequence can be formed according to the shooting time, such as video frames of a video, or the target person's regular actions can be shot, and the multiple images obtained form a sequence, each image in the sequence is called a sequence image, and the first image set and the second image set both include multiple sequence images. Of course, a video of the target person's regular actions can also be shot, and the first image set and the second image set are determined according to the video frames in the video, and the first image set and the second image set are used as training sets to construct the action generation model.

[0053] It should be noted that, in the embodiments of the present application, the first image set may also be referred to as a target image set, and the second image set may also be referred to as a reference image set.

[0054] In some embodiments, in order to quickly obtain a training set of the target person to improve the efficiency and accuracy of building the action generation model, an action video of the target person may be obtained, and the video frames of the action video may be divided to obtain a first image set and a second image set. The first image set and the second image set constitute the training set of the target person, which are used to train the action generation model.

[0055] Specifically, a camera or terminal device can be used to capture a video of the target person's regular movements as the action video of the target person, that is, the video frames in the action video need to include the target person's regular movements, and the video frames in the action video are divided, specifically into a target image set (target) and a reference image set (source). The target image set and the reference image set both include multiple video frames. Since the multiple video frames are continuous on the timeline, the video frames can also be called sequence images, that is, the target image set and the reference image set both include multiple sequence images, and the target image set and the reference image set are used as training sets for backup.

[0056] In some embodiments, in order to ensure the accuracy of model construction, the action video may also be limited to include more than a preset number of video frames. For example, the action video may include at least 15,000 video frames. Of course, the video duration may also be limited. For example, the action video duration may be greater than 10 minutes. For the use of video duration, the frame rate of the camera shooting the video may also be limited to be greater than a certain threshold, such as greater than 24 frames per second.

[0057] The corresponding division method for dividing the video frames in the action video can be divided according to the video sequence, and of course, the division can also be performed alternately, such as dividing the 1st to nth frames into the target image set, dividing the n+1th to mth frames into the reference image set, and then dividing the m+1th to ith frames into the target image set, and then dividing the i+1th to jth frames into the reference image set, and so on.

[0058] S102: Perform three-dimensional reconstruction based on the sequence images of the first image set to obtain a plurality of first 3D models.

[0059] Specifically, the 3D reconstruction technology can be used to perform three-dimensional reconstruction on the sequence images of the first image set to obtain multiple first 3D models, which are reconstructed based on the sequence images of the first image set and can be called first 3D models corresponding to the first image set. Specifically, the 3D reconstruction can be performed using a preset 3D reconstruction algorithm, wherein the preset 3D reconstruction algorithm includes at least one of the following: HMR algorithm, Simplify-x algorithm and Total Capture algorithm.

[0060] It should be noted that in the embodiment of the present application, the Simplify-x algorithm is preferably selected, and the Simplify-x algorithm can better capture human body movements including gestures and expressions.

[0061] In some embodiments, three-dimensional reconstruction can also be performed based on the sequential images of the second image set, and multiple first 3D models can also be obtained. The multiple first 3D models are reconstructed based on the sequential images of the second image set, and can therefore be called first 3D models corresponding to the second image set.

[0062] Therefore, based on the first 3D model corresponding to the second image set and the first image set, the flow mapping relationship of pixels between the sequence images of the second image set and the sequence images of the first image set can be calculated using the centroid coordinates, wherein the flow mapping relationship is used to determine the alignment features of the sequence images of the second image set and the sequence images of the first image set.

[0063] Based on the reconstructed first 3D model, the barycentric coordinates (Barycentric Coordinates) can be used to calculate the flow mapping relationship (Appearance Flow) of pixels between the target image set and the sequence images of the reference image set. Since the target image set and the reference image set share the same topological structure when used for 3D model reconstruction, such as both selecting Simpliy-x, the flow mapping relationship of pixels between the target image set and the reference image set can be established based on the reconstructed 3D model. Specifically, if the pixel points on the sequence images (video frames) of the target image set are visible on the sequence images (video frames) of the reference image set, the flow mapping relationship of pixels between the target image set and the reference image set can be calculated based on the barycentric coordinates (Barycentric Coordinates).

[0064] Regarding the centroid coordinates in 3D rendering, the 3D human body will be divided into multiple triangular faces, and each vertex of the multiple triangular faces has a one-to-one correspondence. When performing action driving, in addition to action, there is also rendering. With 3D action, the rendering method can be determined based on the video frame from the reference image set, so the correspondence between vertices is required. Each point on the triangular face has physical meaning. After the vertices are matched, the pixels outside the vertices are filled, that is, BarycentricCoordinates are used for calculation.

[0065] It should be noted that the flow mapping relationship of pixels between the sequence images of the first image set and the second image set can be calculated by other methods besides the centroid coordinates, which are not limited here.

[0066] S103: Acquire a texture map of the target person, and map the multiple first 3D models according to the texture map to obtain multiple texture maps.

[0067] Specifically, a texture map of the target person may be obtained, for example, the texture map of the target person may be stored in a server in advance for use, and the texture map of the target person may be obtained when the model needs to be constructed. Of course, the texture map of the target person may also be determined based on multiple images of the target person. After the texture map of the target person is obtained, the multiple first 3D models may be mapped according to the texture map to obtain multiple texture maps, which are images obtained by mapping the first 3D model according to the texture map.

[0068] Exemplarily, a texture map of a target person is obtained, specifically, multiple captured images of the target person can be obtained, wherein the multiple captured images are images obtained by capturing the target person from different shooting angles; then based on a multi-view 3D reconstruction algorithm, a second 3D model corresponding to the target person is reconstructed according to the multiple captured images; and based on the correspondence between the second 3D model and the captured images, the texture map of the target person is determined.

[0069] It should be noted that the above 3D reconstruction algorithm can of course be used when acquiring the texture map, but in order to process the data quickly, a general multi-view 3D reconstruction algorithm can be used to construct a 3D model.

[0070] In some embodiments, in order to obtain a more comprehensive and accurate texture map, so that the trained motion generation model can generate an image containing more details, when shooting the target person from different shooting angles, the target person can also be allowed to maintain a preset posture, such as maintaining an A-pose posture.

[0071] For example, a camera or terminal device may be used to capture multiple images of a target person from multiple perspectives, such as 100 images. Specifically, the target person may be allowed to save an A-pose posture, and multiple images of the target person in the A-pose posture may be captured from multiple angles.

[0072] After obtaining multiple images corresponding to the multi-perspectives of the target person, the multi-perspective method can be used to estimate the posture and body shape of each image to establish a 3D model, which is called a second 3D model. Specifically, 100 second 3D models can be established based on the 100 multi-perspective images. According to the correspondence between the 100 established second 3D models and the captured images, the texture map of the target person can be completed, and then the texture map of the target person can be obtained. For example, Figure 2 shown.

[0073] It should be noted that due to the errors in the 3D model and the errors between the 3D points and the 2D projection points, the obtained texture map is still relatively rough and even inaccurate.

[0074] S104: Project the multiple first 3D models to obtain multiple 2D projection images.

[0075] Specifically, each 3D point in the second 3D model is placed on a 2D plane to obtain a corresponding 2D projection image, that is, multiple 2D projection images can be obtained.

[0076] It should be noted that, since the sequence of images in the first image set is continuous, the reconstructed multiple first 3D models, texture maps, and the multiple 2D projection images obtained by projection are also continuous.

[0077] S105: construct an action generation model according to the sequence images of the second image set, the 2D projection image and the texture map.

[0078] Specifically, the action generation model to be constructed can be selected, and the sequence images of the second image set, the 2D projection image and the texture map can be input into the action generation model to be constructed, and the output action image can be obtained, until the output action image meets the requirements, and the parameters of the action generation model can be saved to obtain the constructed action generation model. Among them, meeting the requirements can be determined by manual judgment, or by comparing with the sequence images in the second image set. For example, if the difference in pixels at corresponding positions between the sequence images in the second image set and the output action image is within a certain range, it is determined that the requirements are met.

[0079] Due to the model building method of the present application, the input conditions for model building are finely modeled, that is, the sequence images of the second image set, the 2D projection image and the second 3D model are used as input conditions, so that the constructed action generation model can generate action images containing more details.

[0080] It should be noted that the method for constructing the action generation model introduced in the above embodiment is for constructing a model with one target person as the object. Of course, it can also be for multiple target persons, each of which has a corresponding training set and texture image. Of course, it is also possible to construct a model for the same target person by collecting training sets and corresponding texture images multiple times.

[0081] Among them, the action generation model to be constructed may include a preset network, which may specifically be a neural network, exemplarily, a GAN network, and the sequence images, 2D projection images and texture maps of the second image set are input into the GAN network for model construction until the generation network and the discrimination network of the GAN network reach a balance, and the action generation model can be obtained.

[0082] In some embodiments, in order to further improve the details of the generated action images, in addition to improving the model construction conditions, the embodiments of the present application also make fine modifications to the action generation model to be constructed. Figure 3 As shown, the preset network includes a trunk network model, a first branch network model and a second branch network model, the trunk network model includes a generative adversarial network, and the first branch network model and the second branch network model include an Encoder-Decoder network.

[0083] The first branch network model is used to fuse the alignment features of the sequence images of the second image set with the sequence images of the first image set into the backbone network model, the second branch network model is used to fuse the texture features corresponding to the texture map into the backbone network model, and the backbone network model is used to generate action images based on the 2D projection image, alignment features and texture features. By using texture features and alignment features for completion, a more detailed action image can be obtained, which includes features such as facial expressions, hair and clothing.

[0084] Accordingly, when constructing the model, the 2D projection image can be input into the backbone network model, the sequence image of the second image set can be input into the first branch network model, and the texture map can be input into the second branch network model; wherein the first branch network model is used to fuse the alignment features of the sequence images of the second image set with the sequence images of the first image set at the decoding end of the backbone network model, and the second branch network model is used to fuse the texture features corresponding to the texture map at the decoding end of the backbone network model.

[0085] Specifically, the alignment features of the sequence images of the second image set and the sequence images of the first image set, as well as the texture features corresponding to the texture map, are added to the decoding end (Decoder) of the backbone network model in the form of Concat, which can further supplement the generation details and can be generalized to any character action generation, thereby improving the generalization ability of the action generation model.

[0086] The first branch network model can also be called a reference image Encoder-Decoder network model, the second branch network model can also be called a texture image Encoder-Decoder network model, and the backbone network model can also be called a sequence Encoder-Decoder network model. Figure 4 , Figure 6 and Figure 7 shown.

[0087] For example, Figure 4 As shown, the reference graph Encoder-Decoder network model, the reference graph Encoder-Decoder network model takes as input a reference image and reconstructs the reference image, the reference image is a sequence image in the second image set, through the reconstructed reference image, the feature of the reference image can be better extracted and the feature can be fused into the backbone network model.

[0088] In some embodiments, the sequence images of the second image set are input into the first branch network model, and specifically, the affine transformation map corresponding to the sequence images of the second image set can be determined according to the flow mapping relationship; the sequence images of the second image set and the corresponding affine transformation map are input into the first branch network model to reconstruct the sequence images of the second image set; the features of the encoding end of the first branch network model are aligned using the flow mapping relationship to obtain alignment features, and the alignment features are fused into the decoding end of the trunk network model.

[0089] like Figure 5 As shown, the flow mapping relationship Appearance Flow is used to transform the sequence image Source of the second image set to obtain the corresponding affine transformation image Warp, and the sequence image Source and the affine transformation image Warp are input into the reference image Encoder-Decoder network model to reconstruct the sequence image Source, and the flow mapping relationship AppearanceFlow is used to align the features of the encoding end Encoder of the first branch network model to obtain the alignment features, and the alignment features are fused to the decoding end of the trunk network model.

[0090] The use of alignment features can solve the problem of difficulty in reconstructing target characters wearing loose clothes, loose hair, and messy hair. Since the sequence Encoder-Decoder network model further utilizes the features of texture maps and reference images, that is, the texture maps are used to reconstruct real action images at the same time, and their features are added to the sequence features. At the same time, the reference image provides all the details of the affine transformation map Warp, and the features corresponding to the sequence image Source are added according to the Appearance Flow. For example, if the eye position of the sequence image Source is in the first image set, the feature is directly added to the corresponding position. This ensures the high definition and continuity of the generated action image.

[0091] For example, Figure 6 As shown in the figure, the texture image Encoder-Decoder network model, its input is the texture map, and its output corresponding image is the texture image, denoted as P texture Since the texture map already has relatively good textures for the face and hands, but there are no corresponding control points for clothes and hair, etc., and since the input of the texture map is a strong condition, the image generated by the network has clear faces and gestures, but it is difficult for the training network to obtain useful other reconstruction details such as hair and clothes from the input conditions. Therefore, it is necessary to obtain useful features corresponding to hair and clothes from the first branch network model for reconstruction.

[0092] Since the texture map is determined only based on 100 images from different perspectives, the determined texture map may have characteristics such as errors and blurs, that is, the above-mentioned texture map is still relatively rough and even inaccurate. Therefore, a slight update is made to the texture map based on the reconstructed image to make the texture map clearer. Specifically, the reverse transfer gradient of the second branch network model can be used to update the texture map; and the multiple first 3D models can be mapped according to the updated texture map to obtain multiple texture maps. By updating the texture map, the texture map can be clearer, thereby making the details of the generated action image clearer.

[0093] For example, Figure 7 As shown, the sequence Encoder-Decoder network model has a 2D projection image as input and fuses the features of the sequence images of the second image set and the texture features of the texture map in the decoding end of the network, so its output is an action image containing more details.

[0094] In some embodiments, it is necessary not only to ensure that the generated action image includes more details, but also to ensure the continuity and stability of the generated action image, thereby preventing jitter. Since multiple 2D projection images constitute an image sequence, the 2D projection image is input into the backbone network model, and specifically, the previous frame 2D projection image of the current 2D projection image in the image sequence can be obtained, wherein the current 2D projection image is the 2D projection image that needs to be input into the backbone network model at the current moment, the previous frame 2D projection image is the 2D projection image that has been input into the backbone network model, and the previous frame 2D projection image is continuous with the current 2D projection image; obtain the rendering image corresponding to the previous frame 2D projection image, wherein the rendering image is the output image corresponding to the previous frame 2D projection image input into the backbone network model; input the current 2D projection image, the previous frame 2D projection image and the rendering image into the backbone network model.

[0095] Specifically, when the 2D projection image is input into the sequence Encoder-Decoder network model, multiple 2D projection images constitute an image sequence, the continuity between frames of the image sequence is taken into account. Three frames can be input each time. Assuming that the current time is t, the 2D projection image (t-2, t-1, t) and the rendered image (t-2, t-1) that has been rendered are input. The continuity between the generated action image and the current 2D projection image is taken into account. When there is a deviation in the image input at time t, it can be fine-tuned according to the images at time t-2 and t-1. In addition, in order to continue the continuity of the image, the sequence Encoder-Decoder network model is set to have two outputs, one of which is the rendered image and the other is the weighted image, that is, Figure 7In this application, the optical flow between the images at time t-1 and time t is regressed using time t-2 and time t-1, and then some details of the image at time t can be obtained from time t-1 according to the optical flow, as the final supplement to the final generated rendered image, and the generated rendered image is marked as P series It should be noted that the rendered image can be used as the final action image.

[0096] It should be noted that the 2D projection images (t-2, t-1, t) are the 2D projection images at time t-2, the 2D projection images at time t-1, and the 2D projection images at time t, respectively, where the 2D projection image at time t is the current 2D projection image, and the rendered images (t-2, t-1) are the rendered images at time t-2 and the rendered images at time t-1, respectively. Among them, the rendered image at time t-2 and the rendered image at time t-1 are the output images corresponding to the 2D projection images at time t-2 and the 2D projection images at time t-1 input to the backbone network model, respectively. The 2D projection image at time t-2 and the 2D projection image at time t-1 are the 2D projection images of the previous frame.

[0097] like Figure 8 As shown, Figure 8 The structure of the action generation model provided by an embodiment of the present application is shown. Figure 8 In the example, the backbone network model outputs a rendered image and a weight image, and the second branch network model outputs a texture image. The action image output by the preset network is defined as:

[0098] P final =P series *M+P texture *(1-M)

[0099] Among them, P final Indicates action image, P series represents the rendered image, M represents the weight image, P texture Represents a texture image. Using weights and texture images can produce action images with clearer details.

[0100] The model building method provided in the above-mentioned embodiments, by changing the input conditions of the network model and modifying the network, enables the final action generation model to generate more realistic action images, that is, the generated action images include more details, such as more facial information, human body shape, clothing and other details, thereby improving the user experience.

[0101] See also Fig. 9 , Fig. 9It is a schematic flowchart of the steps of an action image generation method provided in an embodiment of the present application. The action image generation method can be applied to a computer device, which can be a server or a terminal device, wherein the computer device can store the action generation model trained in the above embodiment.

[0102] like Fig. 9 As shown, the motion image generation method includes steps S201 to S204.

[0103] S201, acquiring multiple images of a user, and determining a texture map of the user according to the multiple images;

[0104] S202, performing three-dimensional reconstruction according to the multiple images to obtain a 3D model of the user;

[0105] S203, mapping the 3D model according to the texture map to obtain a texture map;

[0106] S204: Generate an action image of the user according to the image and the texture map.

[0107] The multiple images of the user may be multiple images taken continuously, or a video may be taken and multiple images of the user may be extracted from the video frames of the video. Preferably, the images or videos may be taken from multiple different angles so as to obtain a more complete texture map.

[0108] The texture map of the user is determined according to the multiple images. The 3D model provided in the above embodiment can be first established, and the texture map of the user can be determined by using the correspondence between the 3D model and the images used to establish the 3D model.

[0109] Specifically, the action image of the user is generated according to the image and the texture map, and the action image of the user can be generated according to the image and the texture map by using a neural network model. For example, the action image of the user can be generated according to the image and the texture map by using a human body key point or posture estimation method, or based on a pre-built target 3D model, other actions can be migrated to the target 3D model by using a 3D skeleton method expression and action analysis.

[0110] In some embodiments, after obtaining multiple images of the user and corresponding texture maps, the user's images and texture maps can be input into a pre-trained motion generation model to output the user's motion images. The pre-trained motion generation model can be a model constructed by the above-mentioned model construction method.

[0111] In some embodiments, an action driving source can also be obtained. After obtaining multiple images of the user, the action driving source and the corresponding texture map, the user's image, the action driving source and the texture map can be input into a pre-trained action generation model to output the user's action image.

[0112] Among them, the action driving source is used to drive the pre-trained action generation model to generate action images. In the embodiment of the present application, the action driving source can specifically include a series of 3D models, which are 3D models representing different actions, and are used to drive the action generation model to generate action images with similar actions to the 3D model. The action generation model may be accompanied by an action driving source, or the action driving source may be obtained separately. It should be noted that the action driving source includes a 3D model, which may be the 3D model of the user or the 3D model of other users.

[0113] The action image generation method is applied to different objects and is divided into different scenarios, specifically divided into: application scenario 1 and application scenario 2.

[0114] Application scenario 1: The terminal device pre-stores the above-mentioned trained action generation model, and the terminal device executes the action generation method, inputs the user's image and the corresponding texture map into the action generation model, so that the action generation model outputs the corresponding action image.

[0115] Application scenario 2: The terminal device interacts with the server, and the server pre-stores the above-mentioned trained action generation model. The terminal device executes the action generation method, and sends the user's image and the corresponding texture map to the server, so that the server inputs the user's image and the corresponding texture map into the action generation model, so that the action generation model outputs the corresponding action image, and the server sends the output action image to the terminal device.

[0116] It should be noted that the action image generation method of the embodiment of the present application has similar beneficial technical effects as the method for constructing the action generation model provided by the various embodiments described above, so it will not be repeated here.

[0117] See also Fig.10 , Fig.10 is a schematic block diagram of a computer device provided in an embodiment of the present application. Fig.10 As shown, the computer device 300 includes one or more processors 301 and a memory 302. The processor 301 and the memory 302 are connected via a bus 33, and the bus 33 is, for example, an I2C (Inter-integrated Circuit) bus.

[0118] Among them, one or more processors 301 work individually or collectively to execute the steps of the method for constructing the action generation model or the method for generating an action image provided in the above embodiments.

[0119] Specifically, the processor 301 may be a micro-controller unit (MCU), a central processing unit (CPU) or a digital signal processor (DSP).

[0120] Specifically, the memory 302 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0121] The processor 301 is used to run the computer program stored in the memory 302, and implement the steps of the method for constructing the action generation model or the method for generating an action image provided in the above-mentioned embodiment when executing the computer program.

[0122] Exemplarily, the processor 301 is used to run the computer program stored in the memory 302, and implement the following steps when executing the computer program:

[0123] Acquire a first image set and a second image set of a target person, wherein the first image set and the second image set include sequence images about the target person's actions; perform three-dimensional reconstruction based on the sequence images of the first image set to obtain a plurality of first 3D models; acquire a texture map of the target person, map the plurality of first 3D models based on the texture map to obtain a plurality of texture maps; project the plurality of first 3D models to obtain a plurality of 2D projection images; and construct an action generation model based on the sequence images of the second image set, the 2D projection images and the texture maps.

[0124] In some embodiments, the action generation model includes a backbone network model, a first branch network model and a second branch network model; wherein the first branch network model is used to fuse the alignment features of the sequence images of the second image set with the sequence images of the first image set into the backbone network model, and the second branch network model is used to fuse the texture features corresponding to the texture map into the backbone network model, and the backbone network model is used to generate an action image based on the 2D projection image, the alignment features and the texture features.

[0125] In some embodiments, the backbone network model includes a generative adversarial network, and the first branch network model and the second branch network model include an Encoder-Decoder network.

[0126] In some embodiments, the plurality of 2D projection images constitute an image sequence; and the processor specifically implements:

[0127] Obtain a previous frame 2D projection image of a current 2D projection image in the image sequence, wherein the current 2D projection image is a 2D projection image that needs to be input into the backbone network model at the current moment, the previous frame 2D projection image is a 2D projection image that has been input into the backbone network model, and the previous frame 2D projection image is continuous with the current 2D projection image; obtain a rendered image corresponding to the previous frame 2D projection image, wherein the rendered image is an output image obtained by inputting the previous frame 2D projection image into the backbone network model; input the current 2D projection image, the previous frame 2D projection image and the rendered image into the backbone network model to obtain an action image.

[0128] In some embodiments, the processor is configured to implement:

[0129] Perform three-dimensional reconstruction based on the sequence images of the second image set to obtain multiple first 3D models; based on the second image set and the first 3D models corresponding to the first image set, use the center of mass coordinates to calculate the flow mapping relationship of pixels between the sequence images of the second image set and the sequence images of the first image set; wherein the flow mapping relationship is used to determine the alignment features of the sequence images of the second image set and the sequence images of the first image set.

[0130] In some embodiments, the processor is configured to implement:

[0131] According to the flow mapping relationship, the affine transformation map corresponding to the sequence image of the second image set is determined; the sequence image of the second image set and the corresponding affine transformation map are input into the first branch network model to reconstruct the sequence image of the second image set; the features of the encoding end of the first branch network model are aligned using the flow mapping relationship to obtain alignment features, and the alignment features are fused into the decoding end of the trunk network model.

[0132] In some embodiments, the processor is further configured to implement:

[0133] The gradient of the second branch network model is transferred back to the texture map to update the texture map; and the multiple first 3D models are mapped according to the updated texture map to obtain a texture map.

[0134] In some embodiments, the backbone network model outputs a rendering image and a weight image, the second branch network model outputs a texture image, and the action image output by the action generation model is represented as:

[0135] Pfinal =P series *M+P texture *(1-M)

[0136] Among them, P final Indicates action image, P series represents the rendered image, M represents the weight image, P texture Represents a texture image.

[0137] In some embodiments, the processor is configured to implement:

[0138] Acquire an action video of a target person; and divide video frames of the action video to obtain a first image set and a second image set of the target person.

[0139] In some embodiments, when the processor implements the acquiring of the texture map of the target person, the processor is specifically configured to implement:

[0140] Acquire multiple captured images of a target person, wherein the multiple captured images are images obtained by capturing the target person from different shooting angles; reconstruct a second 3D model corresponding to the target person according to the multiple captured images based on a multi-view 3D reconstruction algorithm; and determine a texture map of the target person according to a correspondence between the second 3D model and the captured images.

[0141] In some embodiments, the first 3D model is reconstructed using a preset 3D reconstruction algorithm, and the preset 3D reconstruction algorithm includes at least one of the following: an HMR algorithm, a Simplify-x algorithm, and a Total Capture algorithm.

[0142] Exemplarily, the processor 301 is used to run the computer program stored in the memory 302, and implement the following steps when executing the computer program:

[0143] Acquire multiple images of the user, determine a texture map of the user based on the multiple images; perform three-dimensional reconstruction based on the multiple images to obtain a 3D model of the user; map the 3D model based on the texture map to obtain a texture map; generate an action image of the user based on the multiple images and the texture map.

[0144] In some embodiments, when the processor generates the action image of the user according to the multiple images and the texture map, it is specifically used to implement:

[0145] The multiple images and the texture map are input into a pre-trained action generation model, and the action image of the user is output; wherein the pre-trained action generation model can be any one of the action generation models constructed in the above-mentioned embodiments of the present application.

[0146] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the steps of the method for constructing an action generation model or the method for generating an action image provided in the above embodiment.

[0147] The computer-readable storage medium may be an internal storage unit of the computer device described in any of the above embodiments, such as a hard disk or memory of the terminal device. The computer-readable storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the terminal device.

[0148] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for constructing an action generation model, characterized in that: The method comprises: Acquire a first image set and a second image set of a target person, wherein the first image set and the second image set include sequence images about actions of the target person; Performing three-dimensional reconstruction based on the sequence images of the first image set to obtain a plurality of first 3D models; Acquire a texture map of the target person, and map the multiple first 3D models according to the texture map to obtain multiple texture maps; Projecting the multiple first 3D models to obtain multiple 2D projection images; An action generation model is constructed based on the sequence images of the second image set, the 2D projection image and the texture map; the action generation model includes a backbone network model, a first branch network model and a second branch network model; wherein the first branch network model is used to fuse the alignment features of the sequence images of the second image set and the sequence images of the first image set into the backbone network model, the second branch network model is used to fuse the texture features corresponding to the texture map into the backbone network model, and the backbone network model is used to generate an action image based on the 2D projection image, the alignment features and the texture features.

2. The method according to claim 1, characterized in that The backbone network model includes a generative adversarial network, and the first branch network model and the second branch network model include an Encoder-Decoder network.

3. The method according to claim 1, characterized in that The plurality of 2D projection images constitute an image sequence; the method comprising: Acquire a previous frame 2D projection image of a current 2D projection image in the image sequence, wherein the current 2D projection image is a 2D projection image that needs to be input to the backbone network model at the current moment, the previous frame 2D projection image is a 2D projection image that has been input to the backbone network model, and the previous frame 2D projection image is continuous with the current 2D projection image; Acquire a rendered image corresponding to the previous frame 2D projection image, wherein the rendered image is an output image obtained by inputting the previous frame 2D projection image into the backbone network model; The current 2D projection image, the previous frame 2D projection image and the rendered image are input into the backbone network model to obtain an action image.

4. The method according to claim 1, characterized in that The method further comprises: Performing three-dimensional reconstruction based on the sequence images of the second image set to obtain a plurality of first 3D models; Based on the first 3D model corresponding to the second image set and the first image set, calculating the flow mapping relationship of pixels between the sequence images of the second image set and the sequence images of the first image set using centroid coordinates; The stream mapping relationship is used to determine alignment features between the sequence images of the second image set and the sequence images of the first image set.

5. The method according to claim 4, characterized in that The method comprises: Determining, according to the stream mapping relationship, an affine transformation map corresponding to the sequence images of the second image set; Inputting the sequence images of the second image set and the corresponding affine transformation map into the first branch network model to reconstruct the sequence images of the second image set; The features of the encoding end of the first branch network model are aligned using the stream mapping relationship to obtain alignment features, and the alignment features are fused to the decoding end of the trunk network model.

6. The method according to claim 1, characterized in that The method further comprises: Using the reverse transfer gradient of the second branch network model to the texture map, so as to update the texture map; and Mapping is performed on the multiple first 3D models according to the updated texture map to obtain a texture map.

7. The method according to claim 1, characterized in that The backbone network model outputs a rendering image and a weight image, the second branch network model outputs a texture image, and the action image output by the action generation model is represented as: P final =P series *M+P texture *(1-M) Among them, P final Indicates action image, P series represents the rendered image, M represents the weight image, P texture Represents a texture image.

8. The method according to any one of claims 1 to 7, characterized in that: The method comprises: Get the target person's action video; The video frames of the action video are divided to obtain a first image set and a second image set of the target person.

9. The method according to any one of claims 1 to 7, characterized in that: The step of obtaining the texture map of the target person includes: Acquire multiple photographed images of a target person, wherein the multiple photographed images are images obtained by photographing the target person from different shooting angles; Reconstructing a second 3D model corresponding to the target person according to the multiple captured images based on a multi-view 3D reconstruction algorithm; and A texture map of the target person is determined according to a correspondence between the second 3D model and the captured image.

10. The method according to any one of claims 1 to 7, characterized in that: The first 3D model is reconstructed using a preset 3D reconstruction algorithm, and the preset 3D reconstruction algorithm includes at least one of the following: an HMR algorithm, a Simplify-x algorithm, and a Total Capture algorithm.

11. A method for generating an action image, characterized in that: The method comprises: Acquire multiple images of a user, and determine a texture map of the user based on the multiple images; Perform three-dimensional reconstruction based on the multiple images to obtain a 3D model of the user; Mapping the 3D model according to the texture map to obtain a texture map; generating an action image of the user based on the multiple images and the texture map; The step of generating the action image of the user according to the multiple images and the texture map includes: The multiple images and the texture map are input into a pre-trained action generation model, and the action image of the user is output; wherein the pre-trained action generation model is obtained according to the method for constructing the action generation model according to any one of claims 1-10.

12. A computer device, characterized in that: The computer device comprises: Memory and processor; Wherein, the memory is connected to the processor and is used to store programs; The processor is used to implement the steps of the method for constructing an action generation model as described in any one of claims 1 to 10 by running the program stored in the memory; or, implement the steps of the method for generating an action image as described in claim 11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to implement the steps of the method for constructing an action generation model as described in any one of claims 1 to 10; or, implement the steps of the method for generating an action image as described in claim 11.

Images