Motion image processing method, apparatus, device, and storage medium

Abstract

The present disclosure relates to a motion image processing method, device, equipment and storage medium. The method comprises: acquiring a target motion image frame; inputting the target motion image frame into a pre-trained VSTGAN model for prediction to obtain a target motion video; and outputting the target motion video. The VSTGAN model comprises a T-net module and an S-net module, the T-net module is used for generating a target motion video with coherent content in the time dimension, and the S-net module is used for improving the definition of the target motion video in the space dimension. The T-net module of the present application can generate a target motion video with coherent content, and the S-net can improve the definition of the target motion video.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a motion image processing method, apparatus, device, and storage medium. Background Technology

[0002] In related technologies, 3D CNN networks or RNN networks are generally used to predict target action videos based on the current target action image frames. 3D CNN networks struggle to handle both limited temporal and rich spatial information while simultaneously focusing on local features, resulting in disjointed local actions and limited dynamic content in the predicted target action video. RNN networks, on the other hand, accumulate redundant information from local features, leading to blurry predicted target action videos. Summary of the Invention

[0003] To overcome the problems existing in the related technologies, this disclosure provides a motion image processing method, apparatus, device and storage medium to solve the above-mentioned problems.

[0004] According to a first aspect of the present disclosure, a motion image processing method is provided, comprising:

[0005] Acquire target motion image frames;

[0006] The target action image frame is input into a pre-trained VSTGAN model for prediction to obtain the target action video;

[0007] Output target motion video;

[0008] The VSTGAN model includes a T-net module and an S-net module.

[0009] The T-net module is used to generate a coherent target motion video in the time dimension;

[0010] The S-net module is used to improve the clarity of the target motion video in the spatial dimension.

[0011] In one implementation, the T-net module includes a video generator and a video discriminator;

[0012] The video generator includes multiple video generation recurrent neural networks (G-VRNN);

[0013] The video discriminator includes multiple video discriminative recurrent neural networks (D-VRNN);

[0014] Each video generation recurrent neural network G-VRNN and its corresponding video discrimination recurrent neural network D-VRNN form a group;

[0015] Each video generation recurrent neural network (G-VRNN) includes: multiple sets of alternating deconvolutional layers and add layers; multiple stacked deconvolutional layers; multiple sets of alternating deconvolutional layers and attention units (ConvAU);

[0016] Each video discriminative recurrent neural network (D-VRNN) includes: multiple sets of alternating convolutional layers and attention units (ConvAU); and multiple stacked convolutional layers.

[0017] In one implementation, in each set of video generation recurrent neural network G-VRNN and video discrimination recurrent neural network D-VRNN, the video generation recurrent neural network G-VRNN is used to generate image frames of predicted target actions.

[0018] The video discriminative recurrent neural network (D-VRNN) is used to discriminate the predicted target action image frame based on the actual target action image frame;

[0019] If the judgment fails, the video generation recurrent neural network G-VRNN is updated to adjust the generated predicted target action image frames in order to improve the action coherence in the generated predicted target action image frames.

[0020] In one embodiment, the target motion video includes a plurality of predicted target motion image frames;

[0021] For any two adjacent time points, the predicted target action image frames;

[0022] The T-net module is used to correct and fuse the feature map of the predicted target action image frame at the next moment based on the feature map of the predicted target action image frame at the previous moment, so as to improve the action continuity of the predicted target action image frame at the next moment and obtain the corrected and fused predicted target action image frame at the next moment.

[0023] In one implementation, the predicted target action image frame feature map for the next time step is corrected and fused based on the predicted target action image frame feature map of the previous time step, including:

[0024] The channel attention coefficients are determined based on the feature map of the predicted target action image frame from the previous time step, specifically including:

[0025] Max pooling is performed on the feature map of the predicted target action image frame from the previous moment to obtain the fourth intermediate value;

[0026] The fifth intermediate value is obtained by performing average pooling on the feature map of the predicted target action image frame at the previous moment;

[0027] The sixth intermediate value is obtained by performing convolution and Add operations on the fourth and fifth intermediate values.

[0028] The channel attention coefficient is obtained by processing the sixth intermediate value into the activation function;

[0029] Based on the channel attention coefficient, the feature map f of the predicted target action image frame at the previous moment is... t-1 The adjustment is performed to obtain the adjustment value f′ of the feature map of the predicted target action image frame at the previous time step. t-1 ;

[0030] The adjustment value f is obtained by adjusting the feature map ft of the predicted target action image frame at the next time step based on the channel attention coefficient. t ′;

[0031] According to the adjustment value f′ t-1 Determining the spatial attention coefficient specifically includes:

[0032] For the adjustment value f′ t-1 The first intermediate value is obtained by taking the maximum value.

[0033] For the adjustment value f′ t-1 The second intermediate value is obtained by averaging the values.

[0034] A third intermediate value is obtained by performing a concat operation and convolution on the first and second intermediate values.

[0035] The third intermediate value is input into the activation function for processing to obtain the spatial attention coefficient;

[0036] Based on the spatial attention coefficient and the adjustment value f t ′ and the adjustment value f′ t-1 Determine the feature map of the predicted target action image frame at the next moment after correction and fusion.

[0037] In one implementation, the predicted target action image frame feature map f from the previous time step is analyzed based on the channel attention coefficients. t-1 The adjustment is performed to obtain the adjustment value f′ of the feature map of the predicted target action image frame at the previous time step. t-1 ,include:

[0038] The channel attention coefficients are compared with the feature map f of the predicted target action image frame from the previous time step. t-1 Perform Hadamard multiplication to obtain the adjustment value f′. t-1 ;

[0039] Based on the channel attention coefficients, the feature map f of the predicted target action image frame at the next time step is obtained. t The adjustment is performed to obtain the adjustment value f. t',include:

[0040] Determine the first difference between 1 and the channel attention coefficient;

[0041] The first difference is compared with the predicted target action image frame feature map f at the next time step. t The f is obtained by performing Hadamard multiplication. t ′;

[0042] Based on the spatial attention coefficient and the adjustment value f t ′ and the adjustment value f′ t-1 Determining the modified and fused feature map of the predicted target action image frame at the next time step includes:

[0043] Determine the second difference between 1 and the spatial attention coefficient;

[0044] Determine the second difference and the adjustment value f t The first Hadamard product;

[0045] Determine the spatial attention coefficient and the adjustment value f′ t-1 The second Hadamard product;

[0046] The sum is obtained by summing the first Hadamard product, the second Hadamard product, and the predetermined bias value;

[0047] The sum is processed by an activation function to obtain the modified and fused target action image frame feature map for the next time step.

[0048] In one implementation, the loss function of the T-net module includes the sum of the action loss function, the adversarial loss function, and the resolution loss function;

[0049] The calculation formula for the action loss function is as follows:

[0050]

[0051] T represents the number of moments;

[0052] t represents time t;

[0053] I represents the total number of layers in the G-VRNN;

[0054] i represents the i-th G-VRNN layer;

[0055] N represents the quantity;

[0056] n represents the nth Gram matrix;

[0057] Represents the Gram matrix;

[0058]

[0059] This represents the matrix after transformation and calculation of the feature map;

[0060] V = H × W;

[0061] Where H is the height of the predicted target action feature map; W is the width of the predicted target action feature map;

[0062] C represents the number of channels;

[0063] The formula for calculating the resolution loss function is as follows:

[0064]

[0065] Where V′ represents the predicted target action video;

[0066] Y represents the actual target action video;

[0067]

[0068]

[0069] Where σ is the sigmoid activation function, It is a convolution operation in the G-VRNN layer at time t. V represents a mask that combines foreground and background information at time t. t ′ represents the predicted target action image frame at time t generated by the T-net module;

[0070] The formula for calculating the adversarial loss function is as follows:

[0071]

[0072] Among them, D t This represents a video discriminator composed of multiple D-VRNNs;

[0073] X represents the input target motion image frame;

[0074] G t This represents a video generator composed of multiple G-VRNNs.

[0075] Secondly, this application proposes a motion image processing apparatus, comprising:

[0076] The acquisition module is used to acquire target action image frames;

[0077] The processing module is used to input the target action image frame into a pre-trained VSTGAN model to predict and obtain the target action video;

[0078] The output module is used to output the target motion video;

[0079] The VSTGAN model includes a T-net module and an S-net module.

[0080] The T-net module is used to generate a coherent target action video in the time dimension;

[0081] The S-net module is used to improve the clarity of the target motion video in the spatial dimension.

[0082] Thirdly, this application proposes a motion image processing method apparatus, comprising:

[0083] Processor; memory used to store processor-executable instructions;

[0084] The processor is configured to run the executable instructions to implement the method described in any of the preceding descriptions.

[0085] Fourthly, this application proposes a non-transitory computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of any of the methods described above.

[0086] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: In the above-described technical solutions of this application, the T-net module in the VSGGAN model generates coherent target action videos in the time dimension, and the S-net module improves the clarity of the target action videos in the spatial dimension. For the predicted target action videos, both coherent actions are generated and the clarity is improved.

[0087] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0088] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0089] Figure 1 This is a flowchart illustrating an action image processing method according to an exemplary embodiment;

[0090] Figure 2 This is a schematic diagram of the structure of a VSTGAN model according to an exemplary embodiment;

[0091] Figure 3 This is a schematic diagram of the structure of a T-net according to an exemplary embodiment;

[0092] Figure 4 This is a flowchart illustrating another motion image processing method according to an exemplary embodiment.

[0093] Figure 5 This is a flowchart illustrating an adjustment of weights according to an exemplary embodiment;

[0094] Figure 6 This is an example of a Channel attention module and a Spatial attention module, as illustrated in an exemplary embodiment.

[0095] Figure 7 This is a schematic diagram of the structure of an FRB according to an exemplary embodiment;

[0096] Figure 8 This is a G illustrated according to an exemplary embodiment. s Structural diagram;

[0097] Figure 9 This is an image training set illustrated according to an exemplary embodiment;

[0098] Figure 10 This is another image training set illustrated according to an exemplary embodiment;

[0099] Figure 11 This is a qualitative comparison diagram of multiple models according to an exemplary embodiment;

[0100] Figure 12 This is a qualitative comparison diagram of another set of models illustrated according to an exemplary embodiment;

[0101] Figure 13 This is a qualitative comparison diagram of another set of models illustrated according to an exemplary embodiment;

[0102] Figure 14 This is a comparison chart of prediction results between T-net and S-net according to an exemplary embodiment;

[0103] Figure 15 This is a schematic diagram of the structure of an action image processing apparatus according to an exemplary embodiment;

[0104] Figure 16 This is a schematic diagram of the structure of an action image processing device according to an exemplary embodiment. Detailed Implementation

[0105] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description relating to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements.

[0106] This application proposes a motion image processing method, see appendix. Figure 1 ,include:

[0107] In step S102, the target action image frame is acquired.

[0108] In this embodiment, the number of target action image frames is one. A target action image frame is a frame randomly extracted from the video frames of the action. An action video can be generated using this image frame. The target action can be, for example, blinking or raising a hand.

[0109] In step S104, the target action image frame is input into the pre-trained VSTGAN model to predict and obtain the target action video.

[0110] The VSTGAN model includes a T-net module and an S-net module.

[0111] The T-net module is used to generate a coherent target action video in the time dimension.

[0112] The S-net module is used to improve the clarity of target motion videos in the spatial dimension.

[0113] In this embodiment, the video-spatio-temporal GAN ​​(VSTGAN) model uses a temporal net (T-net) module and a spatial net (S-net) module.

[0114] In step S106, the target motion video is output.

[0115] The technical solution described above in this application involves the T-net module in the VSGGAN model generating target motion video in the temporal dimension. This target motion video can display rich motion content and local features, making adjacent image frames in the video more coherent. The S-net module improves the clarity of the target motion video in the spatial dimension. This solves the problem of poor coherence or unclear target motion video in related technologies.

[0116] In one implementation, see Appendix Figure 2 The diagram shows the structure of the VSTGAN model.

[0117] The T-net module includes a video generator and a video discriminator;

[0118] The video generator includes multiple video generation recurrent neural networks (G-VRNN);

[0119] The video discriminator includes multiple video discriminative recurrent neural networks (D-VRNN);

[0120] Each video generation recurrent neural network G-VRNN and its corresponding video discrimination recurrent neural network D-VRNN form a group;

[0121] Each video generation recurrent neural network G-VRNN includes: multiple sets of alternating deconvolutional layers and add layers; multiple stacked deconvolutional layers, multiple sets of alternating deconvolutional layers and attention units (ConvAU);

[0122] Each video discriminative recurrent neural network (D-VRNN) includes: multiple sets of alternating convolutional layers and attention units (ConvAU); and multiple stacked convolutional layers.

[0123] In this embodiment, the VSGGA model includes two GAN networks: T-net and S-net.

[0124] Among them, T-net is generated by Generator G t With Discriminator D t Composition. Among them, G t It has a two-stream structure, consisting of a background stream and a foreground stream. The encoder in the background stream processes the input static image X∈R. 3×H×W Perform feature extraction to generate the encoding vector f z ∈R C×1×1 The decoder will f z The process involves restoration to generate background information. The foreground stream consists of multiple G-VRNN layers, with f... z The foreground dynamic information of frames at different times is used as input. A mask mechanism is employed to combine background information with foreground dynamic information to generate motion video V′. t It consists of multiple D-VRNN layers, with each D-VRNN layer identifying the corresponding dynamic frame image in V′.

[0125] S-net is generated by Generator G s With Discriminator D s Composition. G sAlso employing a two-stream structure, the encoder and decoder in the background stream process still images to generate background information. The foreground stream, composed of stacked FRB and CBAM modules, reconstructs the resolution of V′ to generate foreground information. A mask mechanism is also used to combine the background and foreground information to generate the final predicted motion video V″. D, composed of stacked 3D convolutions... s Perform a discrimination test on V″.

[0126] In one implementation, in each set of video generation recurrent neural network G-VRNN and video discrimination recurrent neural network D-VRNN, the video generation recurrent neural network G-VRNN is used to generate image frames of predicted target actions.

[0127] The video discriminative recurrent neural network (D-VRNN) is used to discriminate the predicted target action image frame based on the actual target action image frame;

[0128] If the judgment fails, the video generation recurrent neural network G-VRNN is updated to adjust the generated predicted target action image frames in order to improve the action coherence in the generated predicted target action image frames.

[0129] In this embodiment, during the verification of the recurrent neural network (D-VRNN), actual motion image frames and generated motion image frames can be compared. This allows the generated motion image frames to display more motion details, thereby improving the motion continuity between adjacent motion image frames.

[0130] For example, see Appendix Figure 3 D-VRNN consists of two sets of alternating convolutional layers and convolutional attention units (ConvAUs), with a total of 6 stacked convolutional layers.

[0131] In G-VRNN, two sets of alternating deconvolutional layers and add layers are set; the number of stacked deconvolutional layers is 3, and three sets of alternating deconvolutional layers and convolutional attention units (ConvAU) are used.

[0132] In one embodiment, the target motion video includes a plurality of predicted target motion image frames;

[0133] For any two adjacent time points, the predicted target action image frames;

[0134] The T-net module is used to correct and fuse the feature map of the predicted target action image frame at the next moment based on the feature map of the predicted target action image frame at the previous moment, so as to improve the action continuity of the predicted target action image frame at the next moment and obtain the corrected and fused predicted target action image frame at the next moment.

[0135] In this embodiment, the Convolutional Attention Unit (ConvAU) described above can perform the steps of correcting and fusing the feature map of the predicted target action image frame at the next time step based on the feature map of the predicted target action image frame at the previous time step.

[0136] The structure of each (Generate-VideoRNN, G-VRNN) layer is the same as that of the decoder in the background stream, and the input is f. z For f z Upsampling is performed to generate foreground information y at time t. t The features of the frame image at time t are largely influenced by the frames at the preceding time t-1. For the feature maps in the G-VRNN layers at different times... We hope to have feature maps with semantic features and The less information loss between time steps, the better. Therefore, the features from the previous time step and the current time step are added together to calculate the semantic features as follows:

[0137]

[0138] in, Let represent the feature map output by the i-th deconvolution layer in the G-VRNN layer at time t. High-resolution feature maps often contain more details and action content. To enable the foreground flow to focus more on the action changes of local features in the time dimension and reduce the learning of redundant information, ConvAU is used to fuse high-resolution features from G-VRNN layers at different times.

[0139] As shown in the figure, the formula for calculating high-resolution features is defined as follows:

[0140]

[0141] See appendix Figure 3 In D-VRNN, D t The structure of the Discriminate-VideoRNN (D-VRNN) layer is the same as that of the encoder in the background stream. However, semantic features are no longer fused in the D-VRNN layer; only ConvAU is used to fuse high-resolution features. Each D-VRNN layer encodes each frame of the image separately, distinguishing between real and fake predictions for each frame to improve G. t Further study.

[0142] In one implementation, see Appendix Figure 4 The process of correcting and fusing the feature map of the predicted target action image frame at the next moment based on the feature map of the predicted target action image frame at the previous moment can further include the following steps:

[0143] In step S401, determining the channel attention coefficients based on the feature map of the predicted target action image frame from the previous moment can specifically include the following steps:

[0144] Max pooling is performed on the feature map of the predicted target action image frame from the previous moment to obtain the fourth intermediate value;

[0145] The fifth intermediate value is obtained by performing average pooling on the feature map of the predicted target action image frame at the previous moment;

[0146] The sixth intermediate value is obtained by performing convolution and Add operations on the fourth and fifth intermediate values.

[0147] The sixth intermediate value is processed by the activation function to obtain the channel attention coefficient.

[0148] In this embodiment, the channel attention coefficient is calculated using the following formula:

[0149] m c =σ(conv(max(f) t-1 ),avg(f t-1 )));

[0150] m c This represents the channel attention coefficient.

[0151] The above f t-1 This is the feature map of the predicted target action image frame from the previous moment.

[0152] See appendix Figure 6 As shown, the attention module includes a Channel attention module and a Spatial attention module. The Channel attention module is used to learn the channel attention coefficients, and the Spatial attention module is used to learn the spatial attention coefficients.

[0153] As shown in the figure, the channel attention module includes an average pooling layer and a max pooling layer, two convolutional units, a summation unit, and an activation function unit. The feature maps are input to the average pooling layer and the max pooling layer, respectively; then they pass through two convolutional units (CONV), a summation unit (Add), and finally the sigmoid activation function.

[0154] In step S403, the feature map f of the predicted target action image frame at the previous time step is processed according to the channel attention coefficient. t-1 The adjustment is performed to obtain the adjustment value f′ of the feature map of the predicted target action image frame at the previous time step. t-1 ;

[0155] In step S405, the feature map f of the predicted target action image frame at the next time step is calculated based on the channel attention coefficients.t The adjustment is performed to obtain the adjustment value f. t ′.

[0156] In some embodiments, according to the adjustment value f′ t-1 Determining the spatial attention coefficient can be further specified by the following steps:

[0157] For the adjustment value f′ t-1 The first intermediate value is obtained by taking the maximum value.

[0158] For the adjustment value f′ t-1 The second intermediate value is obtained by averaging the values.

[0159] A third intermediate value is obtained by performing a concat operation and convolution on the first and second intermediate values.

[0160] The third intermediate value is input into the activation function for processing to obtain the spatial attention coefficient.

[0161] In this embodiment, the formula for calculating the spatial attention coefficient is as follows:

[0162] m s =σ(conv(max(f′) t-1 ),avg(f′ t-1 )));

[0163] Among them, the above f′ t-1 The adjustment value represents the adjusted feature map of the predicted target action image frame from the previous moment.

[0164] See appendix Figure 6 In the Spatial Attention module, the network structure includes: an average pooling layer and a maximum pooling layer, convolutional units, and activation function units. Feature maps are input into the average pooling layer and the maximum pooling layer, respectively; after concatenation and activation function processing, they are output.

[0165] In step S407, based on the spatial attention coefficient and the adjustment value f t ′ and the adjustment value f′ t-1 Determine the feature map of the predicted target action image frame at the next moment after correction and fusion.

[0166] In one implementation, in step S403, the predicted target action image frame feature map f from the previous time step is analyzed based on the channel attention coefficient. t-1 The adjustment is performed to obtain the adjustment value f′ of the feature map of the predicted target action image frame at the previous time step. t-1 This may include the following steps:

[0167] The channel attention coefficients are compared with the feature map f of the predicted target action image frame from the previous time step. t-1 Perform Hadamard multiplication to obtain the adjustment value f′. t-1 .

[0168] In this embodiment, the calculation formula is as follows:

[0169]

[0170] This represents the Hadamard multiplication operation.

[0171] In some embodiments, in step S405, the predicted target action image frame feature map f at the next time step is determined based on the channel attention coefficient. t The adjustment is performed to obtain the adjustment value f. t ′, may include the following steps:

[0172] Determine the first difference between 1 and the channel attention coefficient;

[0173] The first difference is compared with the predicted target action image frame feature map f at the next time step. t The f is obtained by performing Hadamard multiplication. t ′.

[0174] In this embodiment, the calculation formula is as follows:

[0175]

[0176] Among them, f t Feature map of the predicted target action image frame representing the next moment;

[0177] The above f t ′ represents the feature map f of the predicted target action image frame at the next moment. t Adjustments are made to obtain the adjusted value.

[0178] In some embodiments, in step S407, based on the spatial attention coefficient and the adjustment value f t ′ and the adjustment value f′ t-1 Determining the modified and fused feature map of the predicted target action image frame at the next moment may further include the following steps:

[0179] Determine the second difference between 1 and the spatial attention coefficient;

[0180] Determine the second difference and the adjustment value f t The first Hadamard product;

[0181] Determine the spatial attention coefficient and the adjustment value f′ t-1 The second Hadamard product;

[0182] The sum is obtained by summing the first Hadamard product, the second Hadamard product, and the predetermined bias value;

[0183] The sum is processed through an activation function to obtain the modified and fused target action image frame feature map f″ for the next time step. t .

[0184] In some embodiments, combined with Figure 5 The following formula is used for calculation:

[0185]

[0186] Where, f″ t This represents the feature map of the predicted target action image frame at the next moment after correction and fusion.

[0187] F() represents the activation function.

[0188] b represents the bias value.

[0189] The method described in this application dynamically generates weights using spatial and channel attention coefficients. Important channel and spatial features in the hidden state are selected using these channel and spatial attention coefficients. The attention coefficients are then combined with the input using a Hadamard product. This allows ConvAU to focus on local features and reduce redundancy, while effectively fusing information from feature maps at different time steps and updating the hidden state. Therefore, ConvAU can be better applied to 3D time series data.

[0190] In one implementation, the loss function of the T-net module includes the sum of the action loss function, the adversarial loss function, and the resolution loss function.

[0191] In this embodiment, the loss of the T-net module The definition is as follows:

[0192]

[0193] In order for the model to distinguish between generated dynamic content and real dynamic content, it is based on the generated motion video V. ′ Computation with real motion video and Calculate using L1 norm and The distance between them is used as the action loss function to help the model learn more dynamic content. The calculation of the action loss is defined. The formula is as follows:

[0194]

[0195] T represents the number of moments;

[0196] t represents time t;

[0197] I represents the total number of layers in the G-VRNN;

[0198] i represents the i-th G-VRNN layer;

[0199] N represents the quantity;

[0200] n represents the nth Gram matrix;

[0201] Represents the Gram matrix;

[0202]

[0203] This represents the matrix after transformation and calculation of the feature map;

[0204] V = H × W;

[0205] Where H is the height of the predicted target action feature map; W is the width of the predicted target action feature map;

[0206] C represents the number of channels.

[0207] The aforementioned Gram matrices, numbering N, can be used to synthesize dynamic texture information for operations such as image style transfer. Using Gram matrices can further facilitate the network in generating richer dynamic content. For the G-VRNN layer at time t, the feature maps in this layer... Convert to N

[0208] In some embodiments, the L1 norm is used to calculate the generated motion video V. ′ Resolution loss compared to real motion video Y

[0209] The formula for calculating the resolution loss function is as follows:

[0210]

[0211] Among them, V ′ This indicates a video of the predicted target's actions.

[0212] Y represents the actual target action video;

[0213]

[0214]

[0215] Where σ is the sigmoid activation function, It is a convolution operation in the G-VRNN layer at time t. V′ represents a mask that combines foreground and background information at time t. t This represents the predicted target action image frame at time t generated by the T-net module.

[0216] Foreground information y generated by the G-VRNN layer at time t t This tends to include more action content. To ensure that the frame image contains both foreground dynamic information and some background information, a masking mechanism is used to combine the foreground stream. t Dynamic information and background flow in y b Background information in the text.

[0217] The formula for calculating the adversarial loss function is as follows:

[0218]

[0219] Among them, D t This represents a video discriminator composed of multiple D-VRNNs;

[0220] X represents the input target motion image frame;

[0221] G t This represents a video generator composed of multiple G-VRNNs.

[0222] In one embodiment, the S-net module includes an FRB module, wherein the FRB module includes a first convolution kernel of 1×k×k and a second convolution kernel of k×1×1.

[0223] In this embodiment, FRB is a lightweight 3D convolutional module, such as Figure 7 As shown, two 3D convolutional kernels, 1×k×k and k×1×1, are stacked, and a residual structure is used to add the input and output. This structure can accomplish the super-resolution task of motion videos while reducing the amount of computation.

[0224] The aforementioned FRB, along with CBAM composed of Channel attention and Spatial attention, is introduced into S-net to focus on local features in the spatial dimension and reconstruct resolution information.

[0225] In some embodiments, S-net is a two-stage network, with the network structure as follows: Figure 1 As shown.

[0226] G s Background flow structure and G t Similarly, the u-net encoding-decoding structure is used to generate background information. For example... Figure 8 As shown, the foreground stream stacks multiple FRB modules to ensure the network has sufficient spatial mapping capability to learn and reconstruct resolution information. A CBAM module is inserted into the FRB modules to help the foreground stream better focus on reconstructing local features. For the input V′, the foreground stream downsamples the frame image and then upsamples it using pixel shuffle. A mask formula is used to combine the foreground and background information to generate the reconstructed video V″.

[0227] D s Composed of 3D convolutions, it performs a downsampling operation on V″ and outputs the discrimination probability. Through D... s Judging the truth value of V″ continuously promotes G s We aim to learn in a more realistic direction. The adversarial loss in S-Net is defined as follows:

[0228]

[0229] To ensure that the S-net network can complete the resolution reconstruction task from both semantic and resolution feature levels, feature loss and resolution loss are calculated for V″. For the feature loss, the generated video V″ and the real video Y are fed into the VGG network respectively to obtain the feature maps output by the h-th layer of the VGG network. f h The distance between two feature maps is calculated using MSE.

[0230]

[0231] f h =VGG h (Y);

[0232] Feature loss The definition is as follows:

[0233]

[0234] The distance between V″ and the real video Y is calculated using the L1 paradigm as the resolution loss. The definition is as follows:

[0235]

[0236] The final loss function of the S-net network The definition is as follows:

[0237]

[0238] In some embodiments, the dataset can be selected as a blink dataset.

[0239] Blinking is both a localized action and contains sufficient detail throughout the entire process. Therefore, we chose to predict blinking videos at both small-scale and large-scale localized action levels to verify the model's ability to focus on local features and predict detailed information. Furthermore, we selected data from KTH that also exhibit localized action characteristics to further validate the model's generalizability.

[0240] Two small sample datasets were created: a facial blinking dataset with small-scale local motion features and an eye-cropped blinking dataset with large-scale local motion features. The resolution of each motion video was adjusted to 256×256. Each motion video consists of seven frames and includes complete eye closing and opening movements. The facial blinking dataset contains 352 motion video sets, such as... Figure 9 As shown, the training set consists of 300 groups, and the test set consists of 52 groups. The blinking dataset contains 448 groups of motion video data.

[0241] like Figure 10 As shown, the training set data is divided into 377 groups, and the test set data is divided into 71 groups.

[0242] The KTH dataset is a classic action recognition dataset, with action videos at a resolution of 160×128. It contains 6 actions, each performed by 25 people. Each action video is processed from 7 frames, resized to 128×128. The waving actions of 1 to 16 people are selected as the training set, totaling 300 action video sets, while the actions of 17 to 25 people are selected as the test set, totaling 75 action video sets.

[0243] In T-Net, using the Adam optimizer, l r The β1 is 0.5, β2 is 0.999, the batch size is 2, and T-Net is trained for 150 epochs. Generator G t Each convolutional layer is followed by instance normalization and the ReLU activation function. Discriminator D t Each convolutional layer in G-VRNN is followed by instance normalization and the Leaky_RELU function. This is required in both G-VRNN and D-VRNN. The loss, latent vectors, and added vectors of the convolutional layers are shown in Table 1, and the specific relevant parameters are shown in Table 2.

[0244]

[0245] Table 1

[0246]

[0247]

[0248] Table 2

[0249] In S-Net, using the Adam optimizer, l r The α value is 0.0002, β1 is 0.5, β2 is 0.999, the data batch size is 2, the S-Net is trained for 100 epochs, α = 0.01, γ = 0.06, and the VGG model is used to calculate the feature loss for the first 34 layers. G s Parameter settings for background flow and G t Similarly, the FRB module consists of a Prelude activation function and two 3D convolutional layers with kernels of 1×3×3 and 3×1×1, respectively. Discriminator D s The 3D convolution in the S-net network is followed by instance normalization and Leaky_RELU. s The specific relevant parameters are shown in Table 3.

[0250]

[0251] Table 3

[0252] In experiments on two blink datasets, the pixel values ​​of the images were normalized, mapping the numerical range to the [-1, 1] interval. The model input consisted of static eye crop images or static face images with a resolution of 256×256. The model generated data of size 3×7×256×256 on the blink dataset.

[0253] Compared to the entire motion video frame, the blinking motion within an eye-captured motion video has a relatively large scale. Therefore, the model was trained on an eye-captured dataset to verify its effectiveness in handling large-scale local features.

[0254] Referring to Table 4, which shows the quantitative comparison results of predicting seven frames based on static eye images, the model's prediction of action video frames achieves good results at both the image and pixel levels.

[0255]

[0256]

[0257] Table 4

[0258] See Figure 11The qualitative comparison chart of several models shows that the comparative models can learn changes in eye movements and effectively predict eye movement trends, resulting in relatively coherent action content in the predicted action videos. However, their ability to process eye features is relatively low, leading to poor quality of the predicted action videos, with eye details remaining somewhat blurry. In contrast, the action videos predicted by the model in this application not only have coherent action content but also accurately predict eye details.

[0259] Unlike the eye-truncation blink dataset, the face blink dataset has smaller eye features, which can further verify the model's effectiveness in handling small-scale local features.

[0260]

[0261] Table 5

[0262] See appendix Figure 12 The comparison charts show that the prediction performance of each model has declined to some extent. VGAN, in particular, can hardly learn the feature changes of the eyes and predicts less dynamic content in the action video. However, as shown in Table 5, the model proposed in this application can still focus on small-scale features, predict more coherent action content, and reconstruct facial details better.

[0263] In some embodiments, the pixel values ​​of the images were also normalized in the experiments on the KTH dataset. The model input was a static waving image with a resolution of 128×128. The model generated data of size 3×7×128×128 on the kth dataset.

[0264] We selected waving data from the KTH dataset to further validate the robustness of the model.

[0265] See Figure 13 The diagram shows a comparison of the model prediction results, and Table 6 shows a comparison of the prediction results of each model.

[0266]

[0267]

[0268] Table 6

[0269] For classic action recognition data, comparative models can predict the movement trend of waving hands, but the detailed features of the arm are blurred and missing during the movement. However, the model in this application can also focus on the local features of the waving hand, predict a continuous waving motion, and predict the detailed information of the arm with relatively high accuracy.

[0270] Experiments on three datasets demonstrate that ConvAU can be effectively applied to 3D temporal data, focusing on local features at different scales. T-net fully leverages the advantages of RNNs and GANs in the temporal dimension; stacked G-VRNN layers using ConvAU can predict relatively realistic local dynamic content, while stacked D-VRNN layers using ConvAU can effectively discriminate local dynamic content. S-net, on the other hand, fully leverages the advantages of CNNs and GANs in the spatial dimension. Using stacked 3DCNNs and attention modules effectively reconstructs the resolution of the action videos predicted by T-net, reducing blurring. When using 3DCNNs for discrimination, the quality of the predicted action videos is further improved. Incorporating the attention mechanism into ConvAU and the S-net foreground flow allows the model to effectively focus on important features, exhibiting good performance on small sample datasets.

[0271] See Figure 14 The image shows a comparison between T-Net and S-Net. T-Net can focus on local features in the time dimension and generate local dynamic information. However, since ConvAU cannot completely eliminate redundant information, this redundant information will continue to accumulate as the feature vector is passed. Therefore, the detailed information predicted by T-Net in motion videos will also be blurred to some extent.

[0272] S-Net can reconstruct the resolution of motion videos predicted by T-Net. The blurring of eye details in blinking images and arm details in waving images is effectively mitigated.

[0273] This application proposes a two-stage model, VSTGAN, comprising two GAN networks: T-net and S-net. It achieves action prediction. T-net predicts local dynamic information in the temporal dimension, while S-net reconstructs the resolution of the dynamic information in the spatial dimension. Based on action image frames, the model can sequentially model from both temporal and spatial dimensions. Furthermore, ConvAU is proposed to dynamically generate weights in the RNN, making it better suited for 3D temporal data. Experiments show that the model can effectively predict local detail information and maintains good performance on small sample datasets.

[0274] Secondly, this application proposes a motion image processing apparatus, see appendix. Figure 15 The diagram shows a structural schematic of a motion image processing device, which includes:

[0275] Acquisition module 121 is used to acquire target motion image frames;

[0276] Processing module 122 is used to input the target action image frame into a pre-trained VSTGAN model to predict and obtain the target action video;

[0277] Output module 123 is used to output the target motion video;

[0278] The VSTGAN model includes a T-net module and an S-net module.

[0279] The T-net module is used to generate a coherent target action video in the time dimension;

[0280] The S-net module is used to improve the clarity of the target motion video in the spatial dimension.

[0281] Thirdly, this application proposes a motion image processing device, see appendix. Figure 16 ,include:

[0282] Processor 31; Memory 132 for storing instructions executable by processor 131;

[0283] The processor 131 is configured to run the executable instructions to implement the method described in any of the above-mentioned embodiments.

[0284] Fourthly, this application proposes a non-transitory computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of any of the methods described above.

[0285] In this embodiment of the invention, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0286] The various methods, steps, and logic diagrams disclosed in the embodiments of this invention can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor reads information from the storage medium and, in conjunction with its hardware, completes the steps of the above methods.

[0287] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0288] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A motion image processing method, characterized in that, include: Acquire target motion image frames; The target action image frame is input into a pre-trained VSTGAN model for prediction to obtain the target action video; Output the target motion video; The VSTGAN model includes a T-net module and an S-net module. The T-net module is used to generate coherent target action videos in the time dimension; The S-net module is used to improve the clarity of the target motion video in the spatial dimension; The T-net module includes a video generator and a video discriminator; The video generator includes multiple video generation recurrent neural networks (G-VRNN); The video discriminator includes multiple video discriminative recurrent neural networks (D-VRNN); Each video generation recurrent neural network G-VRNN and its corresponding video discrimination recurrent neural network D-VRNN form a group; Each video generation recurrent neural network (G-VRNN) includes: multiple sets of alternating deconvolutional layers and add layers; multiple stacked deconvolutional layers; multiple sets of alternating deconvolutional layers and attention units (ConvAU); Each video discriminative recurrent neural network (D-VRNN) includes: multiple sets of alternating convolutional layers and attention units (ConvAU); multiple stacked convolutional layers; The S-net module includes a generator and a discriminator; The generator has a dual-stream structure, where the encoder and decoder in the background stream process static images to generate background information. The foreground stream includes stacked FRB and CBAM modules, which reconstruct the resolution of the video generated by the T-net module to generate foreground information; The predicted motion video is generated by combining the mask mechanism with the background and foreground information. The discriminator discriminates the predicted action video; The S-net module includes an FRB module, which has a first convolution kernel of 1×k×k and a second convolution kernel of k×1×1.

2. The motion image processing method according to claim 1, characterized in that, In each set of video generation recurrent neural network G-VRNN and video discrimination recurrent neural network D-VRNN, the video generation recurrent neural network G-VRNN is used to generate image frames of predicted target actions; The video discriminative recurrent neural network (D-VRNN) is used to discriminate the predicted target action image frame based on the actual target action image frame; If the judgment fails, the video generation recurrent neural network G-VRNN is updated to adjust the generated predicted target action image frames in order to improve the action coherence in the generated predicted target action image frames.

3. The motion image processing method according to claim 1, characterized in that, The target action video includes multiple predicted target action image frames; For any two adjacent time points, the predicted target action image frames; The T-net module is used to correct and fuse the feature map of the predicted target action image frame at the next moment based on the feature map of the predicted target action image frame at the previous moment, so as to improve the action continuity of the predicted target action image frame at the next moment and obtain the corrected and fused predicted target action image frame at the next moment.

4. The motion image processing method according to claim 3, characterized in that, The predicted target action image frame feature map for the next time step is corrected and fused based on the predicted target action image frame feature map from the previous time step, including: The channel attention coefficients are determined based on the feature map of the predicted target action image frame from the previous time step, specifically including: Max pooling is performed on the feature map of the predicted target action image frame from the previous moment to obtain the fourth intermediate value; The fifth intermediate value is obtained by performing average pooling on the feature map of the predicted target action image frame at the previous moment; The sixth intermediate value is obtained by performing convolution and Add operations on the fourth and fifth intermediate values. The channel attention coefficient is obtained by processing the sixth intermediate value into the activation function; Based on the channel attention coefficient, the feature map f of the predicted target action image frame at the previous moment is... t-1 The adjustment is performed to obtain the adjustment value f′ of the feature map of the predicted target action image frame at the previous time step. t-1 ; Based on the channel attention coefficients, the feature map f of the predicted target action image frame at the next time step is obtained. t The adjustment is performed to obtain the adjustment value f. t ′; According to the adjustment value f′ t-1 Determining the spatial attention coefficient specifically includes: For the adjustment value f′ t-1 The first intermediate value is obtained by taking the maximum value. For the adjustment value f′ t-1 The second intermediate value is obtained by averaging the values. A third intermediate value is obtained by performing a concat operation and convolution on the first and second intermediate values. The third intermediate value is input into the activation function for processing to obtain the spatial attention coefficient; Based on the spatial attention coefficient and the adjustment value f t ′ and the adjustment value f′ t-1 Determine the feature map of the predicted target action image frame at the next moment after correction and fusion.

5. The motion image processing method according to claim 4, characterized in that, Based on the channel attention coefficient, the feature map f of the predicted target action image frame at the previous moment is... t-1 The adjustment is performed to obtain the adjustment value f′ of the feature map of the predicted target action image frame at the previous time step. t-1 ,include: The channel attention coefficients are compared with the feature map f of the predicted target action image frame from the previous time step. t-1 Perform Hadamard multiplication to obtain the adjustment value f′. t-1 ; Based on the channel attention coefficients, the feature map f of the predicted target action image frame at the next time step is obtained. t The adjustment is performed to obtain the adjustment value f′. t ,include: Determine the first difference between 1 and the channel attention coefficient; The first difference is compared with the predicted target action image frame feature map f at the next time step. t The f′ is obtained by performing Hadamard multiplication. t ; Based on the spatial attention coefficient and the adjustment value f′ t and the adjustment value f′ t-1 Determining the modified and fused feature map of the predicted target action image frame at the next time step includes: Determine the second difference between 1 and the spatial attention coefficient; Determine the second difference and the adjustment value f′ t The first Hadamard product; Determine the spatial attention coefficient and the adjustment value f′ t-1 The second Hadamard product; The sum is obtained by summing the first Hadamard product, the second Hadamard product, and the predetermined bias value; The sum is processed by an activation function to obtain the modified and fused target action image frame feature map for the next time step.

6. The motion image processing method according to claim 1, characterized in that, The loss function of the T-net module includes the sum of the action loss function, the adversarial loss function, and the resolution loss function; The calculation formula for the action loss function is as follows: T represents the number of moments; t represents time t; I represents the total number of layers in the G-VRNN; i represents the i-th G-VRNN layer; N represents the quantity; n represents the nth Gram matrix; Represents the Gram matrix; This represents the matrix after transformation and calculation of the feature map; V = H × W; Where H is the height of the predicted target action feature map; W is the width of the predicted target action feature map; C represents the number of channels; The formula for calculating the resolution loss function is as follows: Where V′ represents the predicted target action video; Y represents the actual target action video; Where σ is the sigmoid activation function, It is a convolution operation in the G-VRNN layer at time t. V′ represents a mask that combines foreground and background information at time t. t This represents the predicted target action image frame at time t generated by the T-net module; The formula for calculating the adversarial loss function is as follows: Among them, D t This represents a video discriminator composed of multiple D-VRNNs; X represents the input target motion image frame; G t This represents a video generator composed of multiple G-VRNNs.

7. A motion image processing device, characterized in that, include: The acquisition module is used to acquire target action image frames; The processing module is used to input the target action image frame into a pre-trained VSTGAN model to predict and obtain the target action video; The output module is used to output the target motion video; The VSTGAN model includes a T-net module and an S-net module. The T-net module is used to generate coherent target action videos in the time dimension; The S-net module is used to improve the clarity of target motion video in the spatial dimension; The T-net module includes a video generator and a video discriminator; The video generator includes multiple video generation recurrent neural networks (G-VRNN); The video discriminator includes multiple video discriminative recurrent neural networks (D-VRNN); Each video generation recurrent neural network G-VRNN and its corresponding video discrimination recurrent neural network D-VRNN form a group; Each video generation recurrent neural network (G-VRNN) includes: multiple sets of alternating deconvolutional layers and add layers; multiple stacked deconvolutional layers; multiple sets of alternating deconvolutional layers and attention units (ConvAU); Each video discriminative recurrent neural network (D-VRNN) includes: multiple sets of alternating convolutional layers and attention units (ConvAU); multiple stacked convolutional layers; The S-net module includes a generator and a discriminator; The generator has a dual-stream structure, where the encoder and decoder in the background stream process static images to generate background information. The foreground stream includes stacked FRB and CBAM modules, which reconstruct the resolution of the video generated by the T-net module to generate foreground information; The predicted motion video is generated by combining the mask mechanism with the background and foreground information. The discriminator discriminates the predicted action video; The S-net module includes an FRB module, which has a first convolution kernel of 1×k×k and a second convolution kernel of k×1×1.

8. A motion image processing method device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to run the executable instructions to implement the method of any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having computer program instructions stored thereon, characterized in that, When executed by a processor, the program instructions implement the steps of the method according to any one of claims 1 to 6.

Images