A method and apparatus for generating a moving image of a person

Abstract

This invention discloses a method and apparatus for generating motion images of people, relating to the field of computer technology. One specific embodiment of the method includes: acquiring a person image and multiple first action images, wherein the multiple first action images are consecutive frames; obtaining a second action image corresponding to each first action image based on the person image, the multiple first action images, and a basic generation model; determining a frequency domain temporal difference function between the first action images and the second action images; constructing a loss function containing frequency domain constraints based on the frequency domain temporal difference function; and updating the basic generation model based on the loss function containing frequency domain constraints to obtain a motion image generation model. This embodiment reduces computational complexity and storage consumption, avoids additional errors introduced by other complex neural network models, improves the temporal consistency of the generated motion images of people, and improves image quality.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for generating motion images of people. Background Technology

[0002] Human motion image generation refers to generating a series of images of a specified human figure performing actions from a reference video, given a human figure and a given video.

[0003] In existing technologies, a trained generative model is used to generate an image of a specified person in the corresponding reference action for each frame of the action in the reference action video. These images are then used as consecutive frames to synthesize a motion image of the specified person. However, when observing consecutive frames, the motion image will show obvious noise such as inter-frame jitter and artifacts, and the temporal consistency is poor. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and apparatus for generating motion images of people, which can improve the temporal consistency of consecutive frame images, improve the quality of single frame images, reduce computational complexity and storage consumption, and effectively avoid additional errors introduced by other complex neural network models.

[0005] To achieve the above objectives, according to one aspect of the present invention, a method for training a human motion image generation model is provided, comprising:

[0006] A character image and multiple first action images are obtained, wherein the multiple first action images are consecutive frames. Based on the character image, the multiple first action images and the basic generation model, a second action image corresponding to each first action image is obtained.

[0007] Determine the frequency domain temporal difference function of the first action image and the second action image, and construct a loss function containing frequency domain constraints based on the frequency domain temporal difference function;

[0008] The basic generative model is updated based on the loss function containing frequency domain constraints to obtain a human motion image generation model.

[0009] Optionally, determining the frequency domain temporal difference function between the first action image and the second action image includes:

[0010] Determine the first amplitude spectrum and the first phase spectrum of each first action image in the frequency domain; determine the first temporal amplitude difference between two first action images corresponding to adjacent frames based on the first amplitude spectrum; determine the first temporal phase difference between two first action images corresponding to adjacent frames based on the first phase spectrum;

[0011] Determine the second amplitude spectrum and the second phase spectrum of each second action image in the frequency domain; determine the second temporal amplitude difference between two second action images corresponding to adjacent frames based on the second amplitude spectrum; determine the second temporal phase difference between two second action images corresponding to adjacent frames based on the second phase spectrum.

[0012] The timing amplitude difference function is determined based on the first timing amplitude difference and the second timing amplitude difference; the timing difference function is determined based on the first timing phase difference and the second timing phase difference.

[0013] The frequency domain timing difference function is determined based on the timing amplitude difference function and the timing difference function.

[0014] Optionally, determining the first temporal amplitude difference between two first action images corresponding to adjacent frames based on the first amplitude spectrum includes:

[0015] Determine the amplitude value at time t at each coordinate in the first amplitude spectrum of the first motion image corresponding to the current frame, where time t is the time corresponding to the current frame;

[0016] Determine the amplitude value at time t-1 at each coordinate in the first amplitude spectrum of the first motion image corresponding to the previous frame;

[0017] The absolute value or square of the difference between the amplitude value at time t and the amplitude value at time t-1 at each coordinate is taken as the first time-series amplitude difference at time t.

[0018] Optionally, determining the first temporal phase difference between two first action images corresponding to adjacent frames based on the first phase spectrum includes:

[0019] Determine the phase value at time t at each coordinate in the first phase spectrum of the first motion image corresponding to the current frame, where time t is the time corresponding to the current frame;

[0020] Determine the phase value at time t-1 at each coordinate in the first phase spectrum of the first motion image corresponding to the previous frame;

[0021] The absolute value or square of the difference between the phase value at time t and the phase value at time t-1 at each coordinate is taken as the first time-series phase difference at time t.

[0022] Optionally, determining the timing amplitude difference function based on the first timing amplitude difference and the second timing amplitude difference includes:

[0023] The timing amplitude difference function is determined based on the absolute value or square of the difference between the first timing amplitude difference and the second timing amplitude difference.

[0024] Optionally, determining the timing phase difference function based on the first timing phase difference and the second timing phase difference includes:

[0025] The timing phase difference function is determined based on the absolute value or square of the difference between the first timing phase difference and the second timing phase difference.

[0026] Optionally, the loss function further includes pixel-domain constraints. A loss function containing frequency-domain constraints is constructed based on the frequency-domain temporal difference function, including:

[0027] Using the frequency domain temporal difference function as the frequency domain constraint, the pixel domain constraint and the frequency domain constraint are weighted and summed to determine the loss function.

[0028] According to a second aspect of the present invention, a method for generating motion images of a person is provided, comprising:

[0029] Acquire an image of the target person and multiple action images, wherein the multiple action images are consecutive frames;

[0030] The target character image and the multiple motion images are input into a character motion image generation model to obtain a character motion image containing the target character image corresponding to each motion image.

[0031] The human motion image generation model is obtained by the training method of the human motion image generation model according to an embodiment of the present invention.

[0032] According to a third aspect of the present invention, a training apparatus for a human motion image generation model is provided, comprising:

[0033] The first acquisition module acquires a character image and multiple first action images, wherein the multiple first action images are consecutive frames, and obtains a second action image corresponding to each first action image based on the character image, the multiple first action images and the basic generation model.

[0034] The determination module determines the frequency domain temporal difference function of the first action image and the second action image, and constructs a loss function containing frequency domain constraints based on the frequency domain temporal difference function;

[0035] The training module updates the basic generative model according to the loss function containing frequency domain constraints to obtain a human motion image generation model.

[0036] According to a fourth aspect of the present invention, an apparatus for generating motion images of a person is provided, comprising:

[0037] The second acquisition module acquires the target person's image and multiple action images, wherein the multiple action images are consecutive frames;

[0038] The generation module inputs the target character image and the multiple motion images into the character motion image generation model to obtain a character motion image containing the target character image corresponding to each motion image.

[0039] The human motion image generation model is obtained by the training method of the human motion image generation model according to an embodiment of the present invention.

[0040] According to another aspect of the present invention, an electronic device is provided, comprising:

[0041] One or more processors;

[0042] Storage device for storing one or more programs.

[0043] When the one or more programs are executed by the one or more processors, the one or more processors implement the training method for the human motion image generation model or the human motion image generation method provided by the present invention.

[0044] According to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the training method for a human motion image generation model or the method for generating human motion images provided by the present invention.

[0045] One embodiment of the above invention has the following advantages or beneficial effects: A second motion image corresponding to the first motion image is obtained using a human figure image, multiple first motion images, and a basic generation model. Frequency domain temporal change analysis is performed on the first and second motion images to determine their frequency domain temporal difference function. Then, a loss function containing frequency domain constraints is constructed using the frequency domain temporal difference function as a frequency domain constraint. The basic generation model is updated based on the loss function to obtain a human motion image generation model. By adding frequency domain constraints to the loss function to update the basic model, the frequency domain temporal change differences of consecutive frame images are considered, which can improve the temporal consistency of consecutive frames, reduce computational complexity and storage consumption, and avoid additional errors introduced by other models. The human motion image generation model generated using this human motion image generation model can improve the temporal consistency of consecutive frame images and improve the quality of single-frame images.

[0046] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0047] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0048] Figure 1 This is a schematic diagram of the main process of a training method for a human motion image generation model according to an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the main flow of a training method for another human motion image generation model according to an embodiment of the present invention;

[0050] Figure 3 This is a flowchart illustrating a training method for a human motion image generation model according to an embodiment of the present invention.

[0051] Figure 4 This is a schematic diagram of the main flow of a method for generating a human motion image according to an embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram of the main modules of the training device for the human motion image generation model according to an embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram of the main modules of the apparatus for generating motion images of people according to an embodiment of the present invention;

[0054] Figure 7 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0055] Figure 8 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0056] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0057] Temporal consistency in human motion images refers to the consistency of a person's appearance, position, and movements across consecutive frames, without significant inter-frame jitter, artifact noise, or other artifacts. Existing technologies, such as Generative Adversarial Networks (GANs), improve temporal consistency to some extent by constraining consecutive frames in the pixel domain, such as by introducing pixel-domain temporal information like optical flow, when training generative models. However, obtaining pixel-domain temporal information like optical flow relies on other complex neural network models, increasing computation time and storage space. Furthermore, the dependent models introduce additional errors, compromising estimation accuracy. Moreover, pixel-level noise and edge jitter are not significant enough in the pixel domain and are easily ignored by models targeting pixel-domain optical flow, resulting in poor improvement in the temporal consistency of consecutive frames. Therefore, to address these issues, this invention provides a training method for a human motion image generation model and a method for generating human motion images, which can improve the temporal consistency of generated human motion images, reduce computational complexity and storage consumption, and avoid introducing additional errors.

[0058] Figure 1 This is a schematic diagram illustrating the main flow of a training method for a human motion image generation model according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes the following steps:

[0059] Step 101: Obtain the character image and multiple first action images, wherein the multiple first action images are consecutive frames. Based on the character image, multiple first action images and the basic generation model, obtain the second action image corresponding to each first action image.

[0060] Step 102: Determine the frequency domain temporal difference function of the first action image and the second action image, and construct a loss function containing frequency domain constraints based on the frequency domain temporal difference function;

[0061] Step 103: Update the basic generative model based on the loss function containing frequency domain constraints to obtain the human motion image generation model.

[0062] In this embodiment of the invention, the human figure image can be an image containing a human figure, such as a head image, a full-body image, or a half-body image. The first action image is a reference action image. Multiple first action images are consecutive frames; that is, the first action image is a single frame image in the reference action video, and the multiple first action images are consecutive frames in the reference action video, which can form the reference action video. The first action image contains action information. For example, the first action image can be an image containing a human figure in action, or it can be an image containing key points of action information. The key points can be key points of body parts corresponding to the human body. The human figure in the image containing the human figure in action can be the same as or different from the human figure in the human figure image.

[0063] In this embodiment of the invention, a basic generative model is trained based on a character image, multiple first action images, and multiple real images of character actions. A loss function is constructed based on the generated character action images and the multiple real images of character actions. The model is then iteratively trained, and gradient backpropagation is used to optimize the basic generative model. The basic generative model can be any generative model that includes Generative Adversarial Networks (GANs), such as LWGAN (LiquidWarping GAN) or C2F-FWN (Coarse-to-Fine Flow Warping Network). The loss function includes pixel-domain constraints, which can be one or more of image style constraints, feature constraints, and face constraints. After training the basic generative model, the character image and multiple first action images are input into the trained model to generate second action images corresponding to each first action image, i.e., multiple second motion images can be generated, and these multiple second motion images are consecutive frames. The second action images are images of the character performing the actions in the first action images. This embodiment of the invention performs frequency domain temporal analysis on the basic generative model to improve the temporal consistency of consecutive frame images.

[0064] In this embodiment of the invention, after obtaining the second action image corresponding to each first action image, frequency domain temporal change analysis is performed on the first action image and the corresponding second action image to determine the frequency domain temporal difference function between the first action image and the second action image, including:

[0065] Determine the first amplitude spectrum and the first phase spectrum of each first motion image in the frequency domain; determine the first temporal amplitude difference between two first motion images corresponding to adjacent frames based on the first amplitude spectrum; determine the first temporal phase difference between two first motion images corresponding to adjacent frames based on the first phase spectrum;

[0066] Determine the second amplitude spectrum and second phase spectrum of each second motion image in the frequency domain; determine the second temporal amplitude difference between two second motion images corresponding to adjacent frames based on the second amplitude spectrum; determine the second temporal phase difference between two second motion images corresponding to adjacent frames based on the second phase spectrum.

[0067] The timing amplitude difference function is determined based on the first timing amplitude difference and the second timing amplitude difference; the timing difference function is determined based on the first timing phase difference and the second timing phase difference.

[0068] The frequency domain timing difference function is determined based on the timing amplitude difference function and the timing difference function.

[0069] In this embodiment of the invention, the frequency domain temporal changes of consecutive frames of multiple first action images and the frequency domain temporal changes of consecutive frames of multiple second action images are first analyzed. Before determining the first amplitude spectrum and the first phase spectrum of each first action image in the frequency domain, a two-dimensional discrete Fourier transform (DFT) is performed on each first action image, i.e., a single frame image, to obtain the frequency domain representation of the first action image, as shown in equation (1).

[0070]

[0071] In equation (1), f(x,y) is the pixel value of the first motion image at coordinates (x,y), M and N are the length and width of the image, respectively, and F(u,v) represents the value of the first motion image at coordinates (u,v) in the spectrum after transformation. F(u,v) can be expressed as equation (2).

[0072] F(u,v)=R(u,v)+I(u,v), formula (2),

[0073] In equation (2), R(u,v) is the real part of F(u,v), and I(u,v) is the imaginary part of F(u,v);

[0074] According to equation (2), the first amplitude value and the first phase value at the spectrum (u,v) coordinates after the transformation of the first motion image can be obtained;

[0075] First amplitude value The first phase value ∠F(u,v) = arctan(I(u,v) / R(u,v)).

[0076] The first amplitude spectrum of the first motion image in the frequency domain can be obtained from the first amplitude value under all coordinates of the spectrum; the first phase spectrum of the first motion image in the frequency domain can be obtained from the first phase value under all coordinates of the spectrum. Similarly, the second amplitude spectrum and second phase spectrum of the second motion image in the frequency domain can be obtained.

[0077] After obtaining the first amplitude spectrum and the first phase spectrum of each first motion image, the frequency domain temporal changes of the two first motion images corresponding to adjacent frames are analyzed. The first temporal amplitude difference can be determined based on the first amplitude spectrum of the two first motion images corresponding to adjacent frames, and the first temporal phase difference can be determined based on the first phase spectrum of the two first motion images corresponding to adjacent frames.

[0078] In embodiments of the present invention, such as Figure 2 As shown, determining the first temporal amplitude difference between two first action images corresponding to adjacent frames based on the first amplitude spectrum includes:

[0079] Step 201: Determine the amplitude value of the first amplitude spectrum of the first motion image corresponding to the current frame at time t at each coordinate; where time t is the time corresponding to the current frame;

[0080] Step 202: Determine the amplitude value of the first amplitude spectrum of the first motion image corresponding to the previous frame at time t-1 at each coordinate;

[0081] Step 203: Use the absolute value or square of the difference between the amplitude value at time t and the amplitude value at time t-1 at each coordinate as the first time series amplitude difference.

[0082] For example, the amplitude value of the first amplitude spectrum of the first action image corresponding to the current frame at time t at coordinates (u,v) is |F t (u,v)|, where the amplitude of the first amplitude spectrum of the first motion image corresponding to the previous frame at time t-1 is |F. t-1 If (u,v)|, then the first temporal amplitude difference between the two first motion images corresponding to the current frame and the previous frame at coordinates (u,v) can be calculated by equation (3), that is, the absolute value of the difference between the amplitude value at time t and the amplitude value at time t-1.

[0083] TAC t (u,v)=||F t (u,v)|-|F t-1 (u,v)||, Equation (3)

[0084] In equation (3), TAC t (u,v) represents the temporal amplitude difference of the first amplitude spectrum of adjacent frames at coordinates (u,v).

[0085] In this embodiment of the invention, the first temporal amplitude difference between adjacent frames can be obtained in the following way: set a corresponding weight value at time t for the amplitude value at each coordinate, set a corresponding weight value at time t-1 for the amplitude value at each coordinate, determine the product of the amplitude value at time t and the weight value at time t, determine the product of the amplitude value at time t-1 and the weight value at time t-1, and use the absolute value or square of the difference between the two products at each coordinate as the first amplitude difference between adjacent frames.

[0086] In this embodiment of the invention, determining the first temporal phase difference between two first motion images corresponding to adjacent frames based on the first phase spectrum includes:

[0087] Determine the phase value of the first phase spectrum of the first motion image corresponding to the current frame at time t at each coordinate, where time t is the time corresponding to the current frame;

[0088] Determine the phase value of the first phase spectrum of the first motion image corresponding to the previous frame at time t-1 at each coordinate;

[0089] The absolute value or square of the difference between the phase value at time t and the phase value at time t-1 at each coordinate is taken as the first time-series phase difference at time t.

[0090] For example, the phase value of the first phase spectrum of the first action image corresponding to the current frame at time t at coordinates (u,v) is ∠F. t (u,v), the phase value of the first phase spectrum of the first action image corresponding to the previous frame at time t-1 at coordinates (u,v) is ∠F. t-1 If (u,v), then the first temporal phase difference between the two first motion images corresponding to the current frame and the previous frame at coordinates (u,v) can be calculated by equation (4), that is, the absolute value of the difference between the phase value at time t and the phase value at time t-1.

[0091] TPC t (u,v)=|∠F t (u,v)-∠F t-1 (u,v)|, Equation (4)

[0092] In equation (4), TAC t (u,v) represents the temporal phase difference of the first phase spectrum of adjacent frames at coordinates (u,v).

[0093] In this embodiment of the invention, the first temporal phase difference between adjacent frames can be obtained in the following way: set a corresponding weight value at time t for the phase value at each coordinate, set a corresponding weight value at time t-1 for the phase value at each coordinate, determine the product of the phase value at time t and the weight value at time t, determine the product of the phase value at time t-1 and the weight value at time t-1, and use the absolute value or square of the difference between the two products at each coordinate as the first phase difference between adjacent frames.

[0094] Based on the above process for determining the first temporal amplitude difference, the second temporal amplitude difference of two second motion images corresponding to adjacent frames can be obtained similarly. Similarly, based on the above process for determining the first temporal phase difference, the second temporal phase difference of two second motion images corresponding to adjacent frames can be obtained similarly.

[0095] To measure the difference between the second action image and the first action image in the frequency domain time sequence, a consistency measure of frequency domain time sequence change is analyzed.

[0096] In this embodiment of the invention, a timing amplitude difference function can be determined based on a first timing amplitude difference and a second timing amplitude difference. The first timing amplitude difference and the second timing amplitude difference correspond to a first motion image and a second motion image corresponding to the first motion image, respectively. The timing amplitude difference function can be used to measure the amplitude difference between the first motion image and the second motion image in the frequency domain timing. Determining the timing amplitude difference function based on the first timing amplitude difference and the second timing amplitude difference includes: determining the timing amplitude difference function based on the absolute value or square of the difference between the first timing amplitude difference and the second timing amplitude difference. For example, the timing amplitude difference function determined based on the absolute value of the difference is shown in equation (5).

[0097]

[0098] In equation (5), The first action image at time t; The second action image represents time t; Let w(u,v) be the temporal amplitude difference function between the first and second motion images at time t, which can also be called the temporal amplitude difference consistency measure; w(u,v) is the angular frequency at coordinate (u,v); The first temporal amplitude difference at time t; The second temporal amplitude difference at time t; It is the absolute value of the difference between the first time series amplitude difference and the second time series amplitude difference.

[0099] In this embodiment of the invention, the timing amplitude difference function can also be obtained in the following way: set a corresponding first timing amplitude weight for the first timing amplitude difference, set a corresponding second timing amplitude weight for the second timing amplitude difference, determine the product of the first timing amplitude difference and the first timing amplitude weight, determine the product of the second timing amplitude difference and the second timing amplitude weight, and determine the timing amplitude difference function based on the absolute value or square of the difference between the two products.

[0100] In this embodiment of the invention, a timing phase difference function can be determined based on a first timing phase difference and a second timing phase difference. The first timing phase difference and the second timing phase difference correspond to a first motion image and a second motion image corresponding to the first motion image, respectively. The timing phase difference function can be used to measure the phase difference between the first motion image and the second motion image in the frequency domain. Determining the timing phase difference function based on the first timing phase difference and the second timing phase difference includes: determining the timing phase difference function based on the absolute value or square of the difference between the first timing phase difference and the second timing phase difference. For example, the timing phase difference function determined based on the absolute value of the difference is shown in equation (5).

[0101]

[0102] In equation (5), The first action image at time t; The second action image represents time t; Let w(u,v) be the temporal phase difference function between the first and second motion images at time t, also known as the temporal phase difference consistency measure; w(u,v) is the angular frequency at coordinate (u,v); The first temporal phase difference at time t; The second temporal phase difference at time t; It is the absolute value of the difference between the first timing phase difference and the second timing phase difference.

[0103] In this embodiment of the invention, the timing phase difference function can also be obtained in the following way: set a corresponding first timing phase weight for the first timing phase difference, set a corresponding second timing phase weight for the second timing phase difference, determine the product of the first timing phase difference and the first timing phase weight, determine the product of the second timing phase difference and the second timing phase weight, and determine the timing phase difference function based on the absolute value or square of the difference between the two products.

[0104] After determining the timing amplitude difference function and the timing phase difference function, the frequency domain timing difference function, i.e., the frequency domain constraint, can be obtained. The frequency domain timing difference function can be the summation result or weighted summation result of the timing amplitude difference function and the timing phase difference function, as shown in equation (6).

[0105]

[0106] In equation (6), The frequency domain time difference function is the frequency domain constraint, where α and β are the values ​​used to control L. TAC and L TPC The coefficients, representing orders of magnitude, can be adjusted according to the actual situation.

[0107] In this embodiment of the invention, after determining the frequency domain temporal difference function of the first motion image and the second motion image, the third temporal amplitude difference of the two first motion images separated by a preset number of frames can be determined based on the first amplitude spectrum after determining the first amplitude spectrum and the third temporal phase difference of the two first motion images separated by a preset number of frames based on the first phase spectrum.

[0108] After determining the second amplitude spectrum and the second phase spectrum of each second motion image in the frequency domain, the fourth temporal amplitude difference between two second motion images separated by a preset number of frames is determined based on the second amplitude spectrum, and the fourth temporal phase difference between two second motion images separated by a preset number of frames is determined based on the second phase spectrum.

[0109] The timing amplitude difference function is determined based on the third timing amplitude difference and the fourth timing amplitude difference, and the timing difference function is determined based on the third timing phase difference and the fourth timing phase difference.

[0110] The frequency domain difference function is determined based on the timing amplitude difference function and the timing phase difference function.

[0111] The preset frame number N can be a custom number of frames, where N is an integer greater than or equal to 1. For example, the preset frame number N can be 1 frame, 2 frames, etc., meaning that the frames are spaced apart by a preset number of frames, such as the first and third frames, the second and fourth frames, and so on. For instance, the amplitude value at time t can be determined at each coordinate in the first amplitude spectrum of the first motion image corresponding to the current frame, where time t is the time corresponding to the current frame; the amplitude value at time (tN-1) can be determined at each coordinate in the first amplitude spectrum of the first motion image separated by a preset frame number N, and the absolute value or square of the difference between the amplitude value at time t and the amplitude value at time (tN-1) can be used as the third temporal amplitude difference at time t; the phase value at time t can be determined at each coordinate in the first phase spectrum of the first motion image corresponding to the current frame, and the phase value at time (tN-1) can be determined at each coordinate in the first phase spectrum of the first motion image separated by a preset frame number N, and the absolute value or square of the difference between the phase value at time t and the phase value at time (tN-1) can be used as the third temporal phase difference at time t. Similarly, the fourth time-series amplitude difference and the fourth time-series phase difference can be obtained. Then, the time-series amplitude difference function can be determined based on the absolute value or square of the difference between the third and fourth time-series amplitude differences, and the time-series phase difference function can be determined based on the absolute value or square of the difference between the third and fourth time-series phase differences.

[0112] In this embodiment of the invention, after obtaining the frequency domain temporal difference function, i.e., the frequency domain constraint, a loss function is constructed based on the frequency domain constraint. This loss function includes pixel domain constraints and frequency domain constraints. Constructing a loss function containing frequency domain constraints based on the frequency domain temporal difference function includes: using the frequency domain temporal difference function as the frequency domain constraint, performing a weighted summation of the pixel domain constraint and the frequency domain constraint to determine the loss function. Alternatively, the direct summation of the pixel domain constraint and the frequency domain constraint can be used as the loss function. The basic generative model is iteratively trained using the loss function, and the basic generative model is updated using gradient backpropagation until the effect no longer improves, thus obtaining a human motion image generation model. By adding the frequency domain temporal sequence to the loss function to update and optimize the basic generative model, the temporal consistency of consecutive frames can be improved, and the noise of consecutive frames can be reduced.

[0113] like Figure 3The diagram illustrates a flowchart of a training method for a human motion image generation model according to an embodiment of the present invention. First, a human figure image and multiple reference motion images are acquired. The multiple reference motion images are continuous motion images of the human figure. A generation model is trained based on the human figure image, the reference motion images, and a loss function including pixel-domain constraints; that is, a basic generation model based on a generative adversarial network. Based on the human figure image, the multiple reference motion images, and the basic generation model, multiple corresponding generated motion images are obtained. Through forward computation, frequency domain temporal variation analysis is performed on the reference motion images and the generated motion images. First, a two-dimensional Fourier transform (DFT) is performed on a single frame of the reference motion images and the generated motion images to obtain the amplitude spectrum and phase spectrum. Based on the amplitude spectrum and phase spectrum, the frequency domain temporal variation (TPC) is obtained. The temporal amplitude difference (TAC) is obtained based on the amplitude spectrum of consecutive frames, and the temporal phase difference (TPC) is obtained based on the phase spectrum of consecutive frames. Therefore, a temporal amplitude difference consistency measure (L) is obtained based on the temporal amplitude difference between the reference motion images and the generated motion images. TAC The temporal phase difference consistency measure (L) is obtained based on the temporal phase difference between the reference motion image and the generated motion image. TPC According to L TAC and L TPC Obtain the time-series frequency domain difference function (L WTFR The temporal frequency domain difference function is passed to the generative model as a frequency domain constraint. That is, the loss function is the sum of the pixel domain constraint and the frequency domain constraint. The generative model is updated by backpropagation of gradients to obtain the human motion image generation model.

[0114] like Figure 4 As shown, another aspect of this invention provides a method for generating motion images of people, including:

[0115] Step S401: Acquire the target character image and multiple action images, wherein the multiple action images are consecutive frames;

[0116] Step S402: Input the target character image and multiple motion images into the character motion image generation model to obtain a character motion image containing the target character image corresponding to each motion image.

[0117] The human motion image generation model is obtained by the training method of the human motion image generation model according to the embodiment of the present invention.

[0118] By inputting the target character image and multiple consecutive motion images into a character motion image generation model, a character motion image containing the target character image can be obtained. The obtained character motion image has high temporal consistency, resulting in a better-quality character motion image. The character motion image generation method of this invention can be applied to multiple fields such as film animation production, virtual try-on, and virtual digital humans.

[0119] In this embodiment of the invention, two human motion image generation models were used to generate human motion images for experimental simulation. LWGAN and C2F-FWN were used as the basic generation models, and updated LWGAN and updated C2F-FWN were used as the corresponding human motion image generation models. The effects of human motion images generated by LWGAN and the updated LWGAN were compared, as were the effects of human motion images generated by C2F-FWN and the updated C2F-FWN. Table 1 shows the comparison results of the temporal consistency between the human motion images generated by LWGAN and the updated LWGAN, and C2F-FWN and the updated C2F-FWN, and the real images. Table 2 shows the comparison results of the single-frame image quality of the human motion images generated by LWGAN and the updated LWGAN, and C2F-FWN and the updated C2F-FWN.

[0120] Table 1

[0121]

[0122] Table 2

[0123]

[0124] As shown in Table 1, compared to LWGAN and C2F-FWN, the updated LWGAN and C2F-FWN have higher Temporal Consistency Metric (TCM) and Interpolation Peak Signal-to-Noise Ratio (Interp-PSNR), indicating that the updated base generative model, i.e., the human motion image generation model, significantly improves the temporal consistency of consecutive frames. Table 2 shows that compared to LWGAN and C2F-FWN, the updated LWGAN and C2F-FWN have higher Peak Signal-to-Noise Ratio (PSNR), lower Perceptual Similarity Perspective (LPIPS), and higher Structural Similarity (SSIM). Specifically, a higher PSNR value indicates better quality for a single frame, a lower LPIPS value indicates better quality for a single frame, and a higher SSIM value indicates better quality for a single frame. This demonstrates that the updated base generative model, i.e., the human motion image generation model, improves the quality of single frames. Furthermore, training the human motion image generation model did not significantly increase computational complexity.

[0125] The training method for a human motion image generation model provided in this invention updates the base generation model by adding frequency domain constraints to the loss function. The base generation model can be any generative model based on a generative adversarial network, resulting in a human motion image generation model. This improves the temporal consistency of consecutive frames, avoids additional errors caused by introducing other complex neural network models, and has low computational complexity and storage consumption. Furthermore, when the base generation model is iteratively upgraded, the training method of this invention can also be used to obtain an upgraded human motion image generation model. The human motion image generation method of this invention reduces artifacts and jitter, improves the temporal consistency of human motion images, reduces noise in images, and improves the quality of single-frame images. This human motion image generation method can be applied to multiple fields such as film animation production, virtual try-on, and virtual digital humans.

[0126] like Figure 5 As shown, another aspect of the present invention provides a training device 500 for generating a human motion image model, comprising:

[0127] The second acquisition module 501 acquires a character image and multiple first action images, wherein the multiple first action images are consecutive frames, and obtains a second action image corresponding to each first action image based on the character image, the multiple first action images and the basic generation model.

[0128] The determination module 502 determines the frequency domain temporal difference function of the first action image and the second action image, and constructs a loss function containing frequency domain constraints based on the frequency domain temporal difference function.

[0129] Training module 503 updates the basic generative model based on a loss function that includes frequency domain constraints to obtain a human motion image generation model.

[0130] In this embodiment of the invention, the determining module 502 is further configured to:

[0131] Determine the first amplitude spectrum and the first phase spectrum of each first motion image in the frequency domain; determine the first temporal amplitude difference between two first motion images corresponding to adjacent frames based on the first amplitude spectrum; determine the first temporal phase difference between two first motion images corresponding to adjacent frames based on the first phase spectrum;

[0132] Determine the second amplitude spectrum and second phase spectrum of each second motion image in the frequency domain; determine the second temporal amplitude difference between two second motion images corresponding to adjacent frames based on the second amplitude spectrum; determine the second temporal phase difference between two second motion images corresponding to adjacent frames based on the second phase spectrum.

[0133] The timing amplitude difference function is determined based on the first timing amplitude difference and the second timing amplitude difference; the timing difference function is determined based on the first timing phase difference and the second timing phase difference.

[0134] The frequency domain timing difference function is determined based on the timing amplitude difference function and the timing difference function.

[0135] In this embodiment of the invention, the determining module 502 is further configured to: determine the amplitude value at time t at each coordinate in the first amplitude spectrum of the first motion image corresponding to the current frame, wherein time t is the time corresponding to the current frame; determine the amplitude value at time t-1 at each coordinate in the first amplitude spectrum of the first motion image corresponding to the previous frame; and use the absolute value or square of the difference between the amplitude value at time t and the amplitude value at time t-1 at each coordinate as the first temporal amplitude difference at time t.

[0136] In this embodiment of the invention, the determining module 502 is further configured to: determine the phase value at time t at each coordinate in the first phase spectrum of the first motion image corresponding to the current frame, wherein time t is the time corresponding to the current frame; determine the phase value at time t-1 at each coordinate in the first phase spectrum of the first motion image corresponding to the previous frame; and use the absolute value or square of the difference between the phase value at time t and the phase value at time t-1 at each coordinate as the first temporal phase difference at time t.

[0137] In this embodiment of the invention, the determining module 502 is further configured to: determine the timing amplitude difference function based on the absolute value or square of the difference between the first timing amplitude difference and the second timing amplitude difference.

[0138] In this embodiment of the invention, the determining module 502 is further configured to: determine the timing phase difference function based on the absolute value or square of the difference between the first timing phase difference and the second timing phase difference.

[0139] In this embodiment of the invention, the loss function further includes pixel domain constraints. The determination module 502 is further used to: use the frequency domain time-series difference function as the frequency domain constraint, and perform a weighted summation of the pixel domain constraint and the frequency domain constraint to determine the loss function.

[0140] like Figure 6 As shown, another aspect of the present invention provides a device 600 for generating motion images of people, comprising:

[0141] The second acquisition module 601 acquires the target character image and multiple action images, wherein the multiple action images are consecutive frames;

[0142] The generation module 602 inputs the target person image and multiple motion images into the person motion image generation model to obtain a person motion image containing the target person image corresponding to each motion image.

[0143] The human motion image generation model is obtained by the training method of the human motion image generation model according to the embodiment of the present invention.

[0144] Another aspect of the present invention provides an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the human motion image generation model or the human motion image generation method of the present invention.

[0145] Another aspect of the present invention provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the training method for the human motion image generation model or the human motion image generation method of the present invention.

[0146] Figure 7 An exemplary system architecture 700 is shown for a training method or apparatus for a human motion image generation model that can be applied to embodiments of the present invention, as well as a human motion image generation method or apparatus for generating human motion images.

[0147] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, and 703, a network 704, and a server 705. Network 704 serves as the medium for providing communication links between terminal devices 701, 702, and 703 and server 705. Network 704 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0148] Users can use terminal devices 701, 702, and 703 to interact with server 705 via network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0149] Terminal devices 701, 702, and 703 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0150] Server 705 can be a server providing various services, such as a backend management server supporting shopping websites browsed by users using terminal devices 701, 702, and 703 (for example only). The backend management server can analyze and process data such as received product information query requests, and feed back the processing results (such as target push information, product information - for example only) to the terminal devices.

[0151] It should be noted that the training method for the human motion image generation model or the human motion image generation method provided in the embodiments of the present invention are generally executed by the server 705. Correspondingly, the training device for the human motion image generation model or the human motion image generation device are generally set in the server 705.

[0152] It should be understood that Figure 7 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0153] The following is for reference. Figure 8 It shows a schematic diagram of the structure of a computer system 800 suitable for implementing a terminal device of the present invention. Figure 8 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0154] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the system 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0155] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0156] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs the functions defined above in the system of this invention.

[0157] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0158] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0159] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including an acquisition module, a determination module, and a training module. The names of these modules do not necessarily limit the module itself; for example, the training module can also be described as "a module that updates the basic generative model according to a loss function containing frequency domain constraints to obtain a human motion image generation model."

[0160] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include: acquiring a human image and a plurality of first motion images, wherein the plurality of first motion images are consecutive frames; obtaining a second motion image corresponding to each first motion image based on the human image, the plurality of first motion images, and a basic generation model; determining a frequency domain temporal difference function for the first motion images and the second motion images; constructing a loss function containing frequency domain constraints based on the frequency domain temporal difference function; and updating the basic generation model based on the loss function containing frequency domain constraints to obtain a human motion image generation model.

[0161] According to the technical solution of the embodiments of the present invention, the training method for the human motion image generation model provided by the embodiments of the present invention updates the basic generation model by adding frequency domain constraints to the loss function, thereby obtaining the human motion image generation model. This improves the temporal consistency of consecutive frame images, avoids additional errors caused by introducing other complex neural network models, and has low computational complexity and storage consumption. Furthermore, when the basic generation model is iteratively upgraded, the training method for the human motion image generation model of the embodiments of the present invention can also be used to obtain the upgraded human motion image generation model. The human motion image generation method of the embodiments of the present invention reduces artifacts, jitter, and other phenomena, improves the temporal consistency of human motion images, reduces noise in images, and improves the quality of single-frame images. This human motion image generation method can be applied to multiple fields such as film animation production, virtual try-on, and virtual digital humans.

[0162] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A training method for a human motion image generation model, characterized in that, include: A character image and multiple first action images are obtained, wherein the multiple first action images are consecutive frames. Based on the character image, the multiple first action images and the basic generation model, a second action image corresponding to each first action image is obtained. Determine the frequency domain temporal difference function of the first action image and the second action image, and construct a loss function containing frequency domain constraints based on the frequency domain temporal difference function; The basic generative model is updated based on the loss function containing frequency domain constraints to obtain a human motion image generation model; The determination of the frequency domain temporal difference function between the first action image and the second action image includes: Determine the first amplitude spectrum and the first phase spectrum of each first action image in the frequency domain; determine the first temporal amplitude difference between two first action images corresponding to adjacent frames based on the first amplitude spectrum; determine the first temporal phase difference between two first action images corresponding to adjacent frames based on the first phase spectrum; Determine the second amplitude spectrum and the second phase spectrum of each second action image in the frequency domain; determine the second temporal amplitude difference between two second action images corresponding to adjacent frames based on the second amplitude spectrum; determine the second temporal phase difference between two second action images corresponding to adjacent frames based on the second phase spectrum. The timing amplitude difference function is determined based on the first timing amplitude difference and the second timing amplitude difference; the timing difference function is determined based on the first timing phase difference and the second timing phase difference. The frequency domain timing difference function is determined based on the timing amplitude difference function and the timing phase difference function.

2. The training method according to claim 1, characterized in that, Determining the first temporal amplitude difference between two first action images corresponding to adjacent frames based on the first amplitude spectrum includes: Determine the amplitude value at time t at each coordinate in the first amplitude spectrum of the first motion image corresponding to the current frame, where time t is the time corresponding to the current frame; Determine the amplitude value at time t-1 at each coordinate in the first amplitude spectrum of the first motion image corresponding to the previous frame; The absolute value or square of the difference between the amplitude value at time t and the amplitude value at time t-1 at each coordinate is taken as the first time-series amplitude difference at time t.

3. The training method according to claim 1, characterized in that, Determining the first temporal phase difference between two first action images corresponding to adjacent frames based on the first phase spectrum includes: Determine the phase value at time t at each coordinate in the first phase spectrum of the first motion image corresponding to the current frame, where time t is the time corresponding to the current frame; Determine the phase value at time t-1 at each coordinate in the first phase spectrum of the first motion image corresponding to the previous frame; The absolute value or square of the difference between the phase value at time t and the phase value at time t-1 at each coordinate is taken as the first time-series phase difference at time t.

4. The training method according to claim 1, characterized in that, Determining the timing amplitude difference function based on the first timing amplitude difference and the second timing amplitude difference includes: The timing amplitude difference function is determined based on the absolute value or square of the difference between the first timing amplitude difference and the second timing amplitude difference.

5. The training method according to claim 1, characterized in that, Determining the timing phase difference function based on the first timing phase difference and the second timing phase difference includes: The timing phase difference function is determined based on the absolute value or square of the difference between the first timing phase difference and the second timing phase difference.

6. The training method according to claim 1, characterized in that, The loss function also includes pixel-domain constraints. A loss function containing frequency-domain constraints is constructed based on the frequency-domain temporal difference function, including: Using the frequency domain temporal difference function as the frequency domain constraint, the pixel domain constraint and the frequency domain constraint are weighted and summed to determine the loss function.

7. A method for generating a moving image of a person, characterized in that, include: Acquire an image of the target person and multiple action images, wherein the multiple action images are consecutive frames; The target character image and the multiple motion images are input into a character motion image generation model to obtain a character motion image containing the target character image corresponding to each motion image. The human motion image generation model is obtained by the training method according to any one of claims 1-6.

8. A training device for a human motion image generation model, characterized in that, include: The first acquisition module acquires a character image and multiple first action images, wherein the multiple first action images are consecutive frames, and obtains a second action image corresponding to each first action image based on the character image, the multiple first action images and the basic generation model. The determination module determines the frequency domain temporal difference function of the first action image and the second action image, and constructs a loss function containing frequency domain constraints based on the frequency domain temporal difference function; The training module updates the basic generative model according to the loss function containing frequency domain constraints to obtain a human motion image generation model. The determining module is also used for: Determine the first amplitude spectrum and the first phase spectrum of each first action image in the frequency domain; determine the first temporal amplitude difference between two first action images corresponding to adjacent frames based on the first amplitude spectrum; determine the first temporal phase difference between two first action images corresponding to adjacent frames based on the first phase spectrum; Determine the second amplitude spectrum and the second phase spectrum of each second action image in the frequency domain; determine the second temporal amplitude difference between two second action images corresponding to adjacent frames based on the second amplitude spectrum; determine the second temporal phase difference between two second action images corresponding to adjacent frames based on the second phase spectrum. The timing amplitude difference function is determined based on the first timing amplitude difference and the second timing amplitude difference; the timing difference function is determined based on the first timing phase difference and the second timing phase difference. The frequency domain timing difference function is determined based on the timing amplitude difference function and the timing phase difference function.

9. A device for generating motion images of people, characterized in that, include: The second acquisition module acquires the target person's image and multiple action images, wherein the multiple action images are consecutive frames; The generation module inputs the target character image and the multiple motion images into the character motion image generation model to obtain a character motion image containing the target character image corresponding to each motion image. The human motion image generation model is obtained by the training method according to any one of claims 1-6.

10. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

11. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images