An image animation construction method and system based on an attention mechanism
Patent Information
- Application Number
- CN202211458319.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-17
- Publication Date
- 2026-10-23
- Estimated Expiration
- 2042-11-17
AI Technical Summary
[0005]有鉴于此，本发明的目的在于提供一种基于注意力机制的图像动画构建方法和系统，以解决现有技术中FOMM模型仅通过一层卷积网络就得到预测的光流场，使得光流场的预测不准确，从而导致模型无法生成更真实的图像的问题
本发明的技术方案，首先需要获取图像数据，并对所述图像数据进行预处理，将预处理后的图像数据输入至改进模型FOMM-CBAM 中进行训练，并根据损失函数优化模型参数，得到训练后的目标模型，并将图像数据输入至目标模型中，得到增强的特征图像，最后将增强的特征图像合成图像动画，本发明的方案在FOMM模型中加入了一层卷积注意力模型，并通过损失函数优化模型参数，得到目标模型，使得图像数据输入至目标模型后，得到更精细的运动光流场和遮挡图，确保网络生成增强的特征图像，最后在生成器合成图像时，考虑了整个图像的信息，增强了图像特征的表达能力，从而得到更真实的图像动画。
Smart Images
Figure CN115861490B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more specifically to an image animation and construction method based on an attention mechanism. Background Technology
[0002] Image animation is the process of animates static objects in a source image based on the motion that drives the video. For example, animates objects in a still image to generate a video. It has wide applications in many fields, such as filmmaking, virtual reality, photography, and e-commerce.
[0003] With the development of the internet and multimedia technology, image animation has increasingly found applications and has gained widespread attention and research in recent years. Several successful models have been proposed, such as the First-Order Construction Method (FOMM). For more details, please participate... Figure 1 , Figure 1 This is a network structure diagram of the FOMM model illustrated according to an exemplary embodiment. The model uses a keypoint detection module to adaptively learn the keypoint coordinates and their local affine transformations corresponding to various rigid modules (e.g., arms, heads, etc.) of foreground objects in an image. It extracts the optical flow field and occlusion map from the source frame to the driving frame using keypoints from both the source and driving frames. Finally, after distorting the appearance information of the source frame using the optical flow field and refining the image using the occlusion map, the generator module generates the final predicted image.
[0004] Currently, the FOMM model obtains the predicted optical flow field through only one convolutional network, which makes the prediction of the optical flow field inaccurate, thus preventing the model from generating more realistic images. Summary of the Invention
[0005] In view of this, the purpose of this invention is to provide an image animation construction method and system based on an attention mechanism to solve the problem that in the prior art, the FOMM model obtains the predicted optical flow field through only one layer of convolutional network, which makes the prediction of the optical flow field inaccurate, thus causing the model to be unable to generate more realistic images.
[0006] According to a first aspect of the present invention, an image animation construction method based on an attention mechanism is provided, comprising:
[0007] Acquire image data; The image data is preprocessed; The preprocessed image data is input into the improved model FOMM-CBAM for training, and the model parameters are optimized according to the loss function to obtain the trained target model. The image data is input into the target model to obtain an enhanced feature image; The enhanced feature images are then synthesized into an image animation.
[0008] Preferably, the improved model FOMM-CBAM includes: FOMM model and CBAM attention model; The CBAM attention model is used to be inserted into the FOMM model, and the FOMM model includes at least: a key point detection module, a dense network prediction module, and a generator module; An attention model includes at least two modules: a channel attention module and a spatial attention module.
[0009] Preferably, the CBAM attention model is used to insert into the FOMM model, specifically as follows: A convolutional attention model (CBAM) layer is inserted after the dense network prediction module and before the generator module, so that the data from the intermediate feature layers are processed by the convolutional attention model to obtain an enhanced feature image.
[0010] Preferably, the data from the intermediate feature layer, after passing through a convolutional attention model, yields an enhanced feature image, including: The intermediate feature layer image is input into the channel attention module; The maximum value in each channel of the feature image is obtained by channel max pooling. The maximum value on the channel is input into the multilayer perceptron to obtain the channel maximum value weight; The average value of each channel in the feature image is obtained by channel average pooling. The average value on the channel is input into the multilayer perceptron to obtain the channel average value weight; The sum of the channel maximum weight and the channel average weight is input into the sigmoid function to obtain the channel attention weight; Channel attention weights are applied to the channels of the original feature image to obtain a channel-refined feature map; Input the refined feature map of the channel into the spatial attention module; Based on spatial max pooling, the spatial maximum value in the channel refined feature image is obtained; Spatial mean pooling is used to obtain the spatial mean of the channel refined feature image; The maximum value and the mean value in the space are input into the convolution function, and then into the sigmoid function to obtain the spatial attention weights. The spatial attention weights are added to the channel refined feature map to obtain an enhanced feature image.
[0011] Preferably, it further includes: Given intermediate feature layers As input, according to the formula:
[0012]
[0013] We obtain a one-dimensional channel attention map. A two-dimensional spatial attention graph ; in, This represents pixel multiplication; This represents the final output feature; get:
[0014] Optical flow field of FOMM-CBAM The formula is:
[0015] in, This represents an intermediate feature map used by the original FOMM to predict the optical flow field. The optical flow field Input the generator module to obtain the enhanced feature image.
[0016] Preferably, acquiring image data includes: Using web crawler software, image data is scraped from publicly available datasets on video websites.
[0017] Preferably, the preprocessing of the image data specifically includes: The face and body regions are cropped based on the location coordinates of the image data in the dataset, and the resolution of the image data is set to 256x256.
[0018] Preferably, the step of inputting the preprocessed image data into the improved model FOMM-CBAM for training, and optimizing the model parameters according to the loss function to obtain the trained target model includes: 1) Using a pre-trained VGG-19 network as the perceptual loss, the driving frame D and the corresponding reconstructed frame are processed. The perceptual loss function is:
[0019] in, The feature is the i-th channel feature extracted from a specific VGG-19 layer, where i is the number of feature channels in that layer; downsampling is achieved by applying reconstruction loss at multiple resolutions. The image pyramids obtained by D have resolutions of 256×256, 128×128, 64×64 and 32×32, respectively; 2) Equal variance loss:
[0020] in, It is an image After affine transformation matrix The transformed image, where, The value is random; 3) Generator loss:
[0021] 4) Discriminator loss:
[0022] Where Dis represents the discriminator, which is used to distinguish between reconstructed frames and driving frames; D is the reconstructed frame; D is the driving frame. The discriminator loss and generator loss are summed on the image pyramids with resolutions of 256×256, 128×128, 64×64 and 32×32, respectively. Using perceived loss Equal variance loss The model is trained with a weight of 10. After the model parameters are fitted, the key point detection module is frozen. The model is then trained again based on the perceptual loss, equal variance loss, discriminator loss, and generator loss.
[0023] Preferably, the method further includes: The target model is evaluated as follows: Ensure that the pose of the driving object in the first frame is close to the pose of the source frame object; according to Motion pose change driving source frame ; If the pose of the first frame is similar to the pose of the object in the source frame, the motion information is transferred to the source frame through key point changes between driving frames:
[0024] in, This indicates that key point changes between driving frames are transferred to motion information in the source frame; This represents the key points of the source frame; This indicates the key points of the first frame of the driving video; and This represents the key point in the t-th frame of the driving video; Denotes the affine transformation matrix. Represents pixel coordinates.
[0025] According to a second aspect of the present invention, an image animation construction system based on an attention mechanism is provided, comprising: The acquisition module is used to acquire image data; The preprocessing module is used to preprocess the image data; The training module is used to input the preprocessed image data into the improved model FOMM-CBAM for training, and optimize the model parameters according to the loss function to obtain the trained target model; The input module is used to input the image data into the target model to obtain an enhanced feature image; A synthesis module is used to synthesize the enhanced feature images into image animations.
[0026] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects: The technical solution of this invention first requires acquiring image data and preprocessing it. The preprocessed image data is then input into an improved model, FOMM-CBAM, for training. The model parameters are optimized based on a loss function to obtain a trained target model. The image data is then input into the target model to obtain enhanced feature images. Finally, the enhanced feature images are synthesized into an image animation. This invention adds a convolutional attention model layer to the FOMM model and optimizes the model parameters through a loss function to obtain the target model. This results in a more refined motion optical flow field and occlusion map after the image data is input into the target model, ensuring that the network generates enhanced feature images. Finally, when the generator synthesizes the image, it considers the information of the entire image, enhancing the expressive power of the image features, thereby obtaining a more realistic image animation.
[0027] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description
[0028] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0029] Figure 1 This is a network structure diagram of a FOMM model illustrated according to an exemplary embodiment; Figure 2 This is a flowchart illustrating an attention-based image animation construction method according to an exemplary embodiment; Figure 3 This is an improved model FOMM-CBAM illustrated according to an exemplary embodiment; Figure 4 This is a diagram of a channel attention module according to an exemplary embodiment; Figure 5 This is a spatial attention module diagram illustrated according to an exemplary embodiment; Figure 6 This is a diagram of the VGG-19 network structure with a perceptual loss function, illustrated according to an exemplary embodiment. Figure 7 This is a network structure diagram of a discriminator according to an exemplary embodiment; Figure 8 This is a block diagram illustrating an attention-based image animation construction system according to an exemplary embodiment. Detailed Implementation
[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.
[0031] Example 1 Please see Figure 2 , Figure 2 This is a flowchart illustrating an attention-based image animation construction method according to an exemplary embodiment, such as... Figure 2 The flowchart of this method is shown below, and includes: Step S1: Acquire image data; Step S2: Preprocess the image data; Step S3: Input the preprocessed image data into the improved model FOMM-CBAM for training, and optimize the model parameters according to the loss function to obtain the trained target model; Step S4: Input the image data into the target model to obtain an enhanced feature image; Step S5: Synthesize the enhanced feature images into an image animation.
[0032] It should be noted that the application scenarios of this embodiment are film production, virtual reality, photography, e-commerce, Tai Chi image processing, face dialogue, and other application scenarios.
[0033] Understandably, the process begins with acquiring image data and preprocessing it. The preprocessed image data is then input into the improved FOMM-CBAM model for training. The model parameters are optimized using a loss function to obtain the trained target model. The image data is then input into the target model to obtain enhanced feature images. Finally, these enhanced feature images are synthesized into an image animation. This embodiment adds a convolutional attention layer to the FOMM model and optimizes the model parameters using a loss function to obtain the target model. This results in a more refined motion flow field and occlusion map after the image data is input into the target model, ensuring that the network generates enhanced feature images. Finally, when the generator synthesizes the image, it considers the information of the entire image, enhancing the expressive power of the image features and thus obtaining a more realistic image animation.
[0034] Please refer to the details. Figure 3 , Figure 3 An improved model FOMM-CBAM diagram illustrated according to an exemplary embodiment includes: FOMM model 1 and attention model 2, wherein FOMM model 1 includes at least: key point detection module 11, dense network prediction module 12 and generator module 13; Attention model 2 includes at least: channel attention module 21 and spatial attention module 22; It should be noted that the channel attention module 21 and the spatial attention module 22 are located between the dense network prediction module 12 and the generator module 13, so that the image passes through the key point detection module 11, the dense network prediction module 12, the channel attention module 21 and the spatial attention module 22 in sequence to generate an enhanced feature image, and the enhanced feature image is synthesized in the generator module 23 to obtain the image animation.
[0035] Please refer to the details. Figure 4 and Figure 5 , Figure 4 This is a diagram of a channel attention module according to an exemplary embodiment. Figure 5 This is a spatial attention module diagram illustrated according to an exemplary embodiment.
[0036] In practice, the channel attention module 21 and the spatial attention module 22 are specifically as follows: The channel attention module 21 includes: channel max pooling and channel average pooling; The spatial attention module 22 includes: spatial max pooling and spatial mean pooling.
[0037] It should be noted that CBAM represents the attention mechanism, which is represented by FOMM-CBAM based on a first-order motion model. CBAM includes a channel attention module 21 (CAM) and a spatial attention module 22 (SAM). CBAM first reconstructs the feature image through the channel attention module 21, and then inputs it into the spatial attention module 22 to obtain an enhanced feature image.
[0038] In practice, the attention model is used to insert into the FOMM model, specifically as follows: A convolutional attention model (CBAM) layer is inserted after the dense network prediction module and before the generator module, so that the data from the intermediate feature layers are processed by the convolutional attention model to obtain an enhanced feature image.
[0039] In practice, the data from the intermediate feature layer, after passing through a convolutional attention model, yields an enhanced feature image, including: The intermediate feature layer image is input into the channel attention module; The maximum value in each channel of the feature image is obtained by channel max pooling. The maximum value on the channel is input into the multilayer perceptron to obtain the channel maximum value weight; The average value of each channel in the feature image is obtained by channel average pooling. The average value on the channel is input into the multilayer perceptron to obtain the channel average value weight; The sum of the channel maximum weight and the channel average weight is input into the sigmoid function to obtain the channel attention weight; Channel attention weights are applied to the channels of the original feature image to obtain a channel-refined feature map; Input the refined feature map of the channel into the spatial attention module; Based on spatial max pooling, the spatial maximum value in the channel refined feature image is obtained; Spatial mean pooling is used to obtain the spatial mean of the channel refined feature image; The maximum value and the mean value in the space are input into the convolution function, and then into the sigmoid function to obtain the spatial attention weights. The spatial attention weights are added to the channel refined feature map to obtain an enhanced feature image.
[0040] It should be noted that CBAM represents the attention mechanism, which is represented by FOMM-CBAM based on a first-order motion model. CBAM includes a channel attention module (CAM) and a spatial attention module (SAM). CBAM first reconstructs the feature image through the channel attention module, and then inputs it into the spatial attention module to obtain an enhanced feature image.
[0041] Specifically, the channel attention module 21 uses channel max pooling and channel average pooling operations to obtain the maximum value and average value of the channels in the feature map, respectively. Then, it is input into the multilayer perceptron (MLP) to obtain the channel maximum value weight and channel average value weight. After being added together, the channel attention weight is obtained through the sigmoid function. The channel attention weight is then applied to the channels of the original feature image to obtain the channel refined feature map.
[0042] The spatial attention module 22 uses spatial max pooling and spatial mean pooling to obtain the maximum and mean values in the spatial representation of the channel refined feature map after passing through the channel attention module 21. Then, after convolution and sigmoid function operations, spatial attention weights are obtained. Finally, the spatial attention weights are weighted onto the channel refined feature map to obtain an enhanced feature image. The enhanced feature image is then synthesized in the generator module to obtain an image animation. The image animation obtained using this embodiment is more realistic.
[0043] In practice, this also includes: Input intermediate feature layer According to the formula:
[0044]
[0045] We obtain a one-dimensional channel attention map. A two-dimensional spatial attention graph ; in, This represents pixel multiplication; This represents the final output feature; get:
[0046] Optical flow field of FOMM-CBAM The formula is:
[0047] in, The intermediate feature map representing the original FOMM used to predict the optical flow field; the optical flow field Input the generator module to obtain the enhanced feature image.
[0048] in, This represents the intermediate feature layer. Dimensions representing features The dimensions representing the features are (C, H, W). This represents pixel multiplication; specifically, Dimensions may be ,and The output dimension is Based on Python's broadcast mechanism, The output is expanded in dimensions, specifically in the second and third dimensions, to obtain a... The matrix is obtained by copying the values in the first dimension, ultimately resulting in a... A matrix and a matrix that is also Multiplying the matrix by pixels yields a matrix that is also... Matrix output; This represents the refined feature map of the channels after passing through the channel attention module. This indicates the channel attention module. express Internal calculation operations within the module This represents the enhanced feature image after passing through the spatial attention module. This indicates a spatial attention module. express Internal calculation operations within the module; Indicates will Feature map input The output obtained from the module Similarly, these represent dimensions. The values above are the values for the current dimension, for example: here... It can be roughly understood as a cube with a length of 1, a height of H, and a width of W, with a number placed in each position; This represents the coordinates of key points on the driving frame detected by the key point detection module. This indicates the coordinates of key points in the source frame detected by the key point detection module; This represents the affine transformation matrix predicted by the keypoint detection module. Represents the pixel coordinates of the image; This indicates the number of key points detected by the key point detection module. This represents the intermediate feature corresponding to the heatmap of the kth key point; A heatmap representing the background pixels.
[0049] In practice, acquiring image data includes: using web crawler software to crawl image data from publicly available datasets on video websites.
[0050] It's important to note that there are multiple ways to acquire image data. One approach is to use web scraping software to collect video data from publicly available datasets on YouTube, such as the Voxceleb1 dataset for face dialogue and the Taichi' dataset for Tai Chi movements. Alternatively, web scraping software can be used to collect datasets from other videos. Another approach is to obtain the corresponding image data from existing databases. The Taichi' dataset represents a dataset of human movement, while the Voxceleb1 dataset represents a dataset of face speech. For example, if we train using the Voxceleb1 dataset, we obtain a face-related model, which can then be used as input for different faces to create different image animations. Specifically, for the same type of data, such as a face-trained model that can only input face data and a body-trained model that can only input body data, the model will be limited to the same data type.
[0051] In practice, the preprocessing of the image data specifically includes: The face and body regions are cropped based on the location coordinates of the image data in the dataset, and the resolution of the image data is set to 256x256.
[0052] It should be noted that the faces and human body regions were cropped using the location coordinates of the dataset, and the resolution was modified to 256x256.
[0053] It should be noted that all video data is input into the FOMM model with an attention mechanism for training. The training is set to a total of 150 epochs, of which 100 epochs train all network structures except the discriminator, and the parameters of the keypoint detector are frozen in the last 50 epochs while the discriminator module is trained simultaneously.
[0054] In practice, the preprocessed image data is input into the improved model FOMM-CBAM for training, and the model parameters are optimized according to the loss function to obtain the trained target model, including: 1) Use a pre-trained VGG-19 network as the perceptual loss; for details, please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is a diagram of the VGG-19 network architecture with a perceptual loss function, for the driving frame D and the corresponding reconstructed frame. The perceptual loss function is:
[0055] in, The feature is the i-th channel feature extracted from layer VGG-19, where i is the number of feature channels in that layer; downsampling is achieved by applying reconstruction loss at multiple resolutions. The image pyramids obtained by D have resolutions of 256×256, 128×128, 64×64 and 32×32, respectively; It should be noted that VGG-19 is a model of existing technology with a fixed network structure, which will not be described in detail in this embodiment. Among other things, the reconstructed frame... This indicates that the FOMM-CBAM model is driven by input driving frames. and source frame Reconstructed frames output later ; Expressing the request and The perceptual loss (called the perceptual loss function in deep learning); This indicates the total number of feature layers in the VGG-19 network. VGG-19 has 19 layers, so this is 19. Indicates the input reconstruction frame In the Vgg-19 model, its first What are the characteristics of the layer output? Indicates the input driving frame In the Vgg-19 model, its first What are the characteristics of the layer output?
[0056] 2) Equal variance loss:
[0057] in, It is an image After affine transformation matrix The transformed image, where, The value is random; 3) Generator loss:
[0058] 4) Discriminator loss:
[0059] Where Dis represents the discriminator, which is used to distinguish between reconstructed frames and driving frames; D is the reconstructed frame; D is the driving frame. The discriminator loss and generator loss are summed on the image pyramids with resolutions of 256×256, 128×128, 64×64 and 32×32, respectively. It should be noted that, specifically, please refer to the network structure diagram of the discriminator. Figure 7 The discriminator is existing technology and will not be described in detail in this embodiment.
[0060] Using perceived loss Equal variance loss The model is trained with a weight of 10. After the model parameters are fitted, the key point detection module is frozen. The model is then trained again based on the perceptual loss, equal variance loss, discriminator loss, and generator loss.
[0061] Finally, in the first 100 epochs, we employ perceptual loss. Equal variance loss The network was trained with a weight of 10. For the last 50 epochs, the keypoint detection module was frozen, and all loss functions were used to optimize the image generation results.
[0062] In practice, the method further includes: The target model is evaluated as follows: We need to ensure that the pose of the driving object in the first frame is close to the pose of the source frame object. Specifically, our goal is to use... Motion pose changes to drive the source frame If the pose of the first frame is close to the pose of the object in the source frame, we can approximate the motion information transferred to the source frame by the changes in keypoints between driving frames. This is illustrated in the following equation:
[0063] in, This indicates that key point changes between driving frames are transferred to motion information in the source frame; This represents the key points of the source frame; This indicates the key points of the first frame of the driving video; and This represents the key point in the t-th frame of the driving video; Denotes the affine transformation matrix. Represents pixel coordinates.
[0064] The solution of this invention optimizes the optical flow field generation of the network structure by introducing CBAM (convolutional attention mechanism) into the FOMM network, thereby improving the motion transmission capability of the network. The resulting frames have better visual effects and are more vivid and natural.
[0065] The results of the image animation generation are shown in Table 1. In each dataset, the first column is the source frame, the second column is the driving frame, the third column is a frame from the FOMM-CBAM generated video, and the fourth column is a frame from the FOMM generated video. It can be seen that in face pose transfer on the VoxCeleb1 dataset, the FOMM-CBAM model significantly outperforms FOMM. In the third row of Table 1 for the VoxCeleb1 dataset, the facial expressions generated by FOMM-CBAM are closer to the expressions in the driving frames, demonstrating superior expression transfer. In TaiChiHD, when the human pose changes significantly, FOMM almost loses its motion transfer capability, losing much important information in the generated frames, failing to distinguish whether the person has turned around, and failing to accurately redraw head pixel information. In contrast, FOMM-CBAM significantly enhances these key and important aspects. For example, FOMM-CBAM can recognize the front and back of a person, and when the person turns 180 degrees, FOMM-CBAM can correctly redraw the head information, as shown in the first and third rows of Table 1 for the TaiChiHD dataset. Furthermore, rows two, four, and five of the TaiChiHD dataset in Table 1 demonstrate that FOMM-CBAM has a stronger motion transfer capability than FOMM, and the redrawn actions and poses are closer to the driving frames. Objective evaluation results are presented in Table 2 by comparing the reconstructed videos from both algorithms.
[0066] Table 1 Examples of image animation generation and comparison of subjective visual effects
[0067] Table 2 Comparison of video reconstruction evaluation metrics, with both FOMM and FOMM-CBAM (Ours) setting the number of keypoints to 10 (best results are shown in bold).
[0068] Where AKD represents the distance between keypoints in the generated graph and the driving graph; MKR represents the ratio of keypoints present in the driving graph but not in the generated graph. As shown in Table 2, our method achieves better results than FOMM on almost all metrics across all datasets. On the MGif dataset, our method shows comparable results to FOMM. When the video content is more complex and the pose variations of animated objects are more diverse, a comparison of results on VoxCeleb1 and TaiChiHD shows that our proposed improved optical flow prediction algorithm reduces all metrics except for AED. The results in Tables 1 and 2 demonstrate that the FOMM-CBAM algorithm proposed in this application, by incorporating an attention mechanism into the CBAM module, does indeed optimize the final generated video. On the VoxCeleb1 dataset, most of the generated face results are more detailed and realistic than those generated by FOMM. For the TaichiHD dataset, when the human body poses with large movements, the network has a certain ability to track the pose change. In particular, when the human body moves from the front to the side or back, FOMM-CBAM can notice the important information in the intermediate feature map, obtain a more refined optical flow field and occlusion map, and thus estimate the pose transfer, giving the correct human pose in the generated frame. FOMM, on the other hand, cannot estimate the current pose change process, which leads to the generation of an incorrect human pose in the generated frame.
[0069] Example 2 Please see Figure 8 , Figure 8 This is a block diagram of an attention-based image animation construction system 600 according to an exemplary embodiment, such as... Figure 8 As shown, the system includes: Acquisition module 601 is used to acquire image data; Preprocessing module 602 is used to preprocess the image data; The training module 603 is used to input the preprocessed image data into the improved model FOMM-CBAM for training, and optimize the model parameters according to the loss function to obtain the trained target model; Input module 604 is used to input the image data into the target model to obtain an enhanced feature image; The synthesis module 605 is used to synthesize the enhanced feature image into an image animation.
[0070] It should be noted that the application scenarios of this embodiment are film production, virtual reality, photography, e-commerce, Tai Chi image processing, face dialogue, and other application scenarios.
[0071] Understandably, in this embodiment, the technical solution includes an acquisition module 601 for acquiring image data, a preprocessing module 602 for preprocessing the image data, a training module 603 for inputting the preprocessed image data into the improved model FOMM-CBAM for training and optimizing the model parameters according to the loss function to obtain the trained target model, an input module 604 for inputting the image data into the target model to obtain an enhanced feature image, and a synthesis module 605 for synthesizing the enhanced feature image into an image animation. This embodiment adds a convolutional attention model layer to the FOMM model and optimizes the model parameters through a loss function to obtain the target model. This results in a more refined motion optical flow field and occlusion map after the image data is input into the target model, ensuring that the network generates enhanced feature images. Finally, when the generator synthesizes the image, it considers the information of the entire image, enhancing the expressive power of the image features and thus obtaining a more realistic image animation.
[0072] The computer-readable storage media disclosed in this embodiment include, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0073] It should be noted that since the implementation methods and beneficial effects of each module in this embodiment can be found in the relevant description in Embodiment 1, this embodiment will not repeat them here.
[0074] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.
[0075] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.
[0076] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0077] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0078] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0079] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.
[0080] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0081] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. A method for constructing image animations based on an attention mechanism, characterized in that, include: Acquire image data; The image data is preprocessed; The preprocessed image data is input into the improved model FOMM-CBAM for training, and the model parameters are optimized according to the loss function to obtain the trained target model. The image data is input into the target model to obtain an enhanced feature image; The enhanced feature images are then synthesized into an image animation. The improved model FOMM-CBAM includes: FOMM model and CBAM attention model; The CBAM attention model is used to be inserted into the FOMM model, and the FOMM model includes at least: a key point detection module, a dense network prediction module, and a generator module; Attention models should include at least: a channel attention module and a spatial attention module; The CBAM attention model is used for insertion into the FOMM model, specifically as follows: A convolutional attention model (CBAM) layer is inserted after the dense network prediction module and before the generator module, so that the data from the intermediate feature layers are processed by the convolutional attention model to obtain an enhanced feature image.
2. The method according to claim 1, characterized in that, The data from the intermediate feature layer, after passing through a convolutional attention model, yields an enhanced feature image, including: The intermediate feature layer image is input into the channel attention module; The maximum value in each channel of the feature image is obtained by channel max pooling. The maximum value on the channel is input into the multilayer perceptron to obtain the channel maximum value weight; The average value of each channel in the feature image is obtained by channel average pooling. The average value on the channel is input into the multilayer perceptron to obtain the channel average value weight; The sum of the channel maximum weight and the channel average weight is input into the sigmoid function to obtain the channel attention weight; Channel attention weights are applied to the channels of the original feature image to obtain a channel-refined feature map; Input the refined feature map of the channel into the spatial attention module; Based on spatial max pooling, the spatial maximum value in the channel refined feature image is obtained; Spatial mean pooling is used to obtain the spatial mean of the channel refined feature image; The maximum value and the mean value in the space are input into the convolution function, and then into the sigmoid function to obtain the spatial attention weights. The spatial attention weights are added to the channel refined feature map to obtain an enhanced feature image.
3. The method according to claim 2, characterized in that, Also includes: Given intermediate feature layers As input, according to the formula: We obtain a one-dimensional channel attention map. A two-dimensional spatial attention map ; in, This represents pixel multiplication; This represents the final output feature; get: Optical flow field of FOMM-CBAM The formula is: in, This represents an intermediate feature map used by the original FOMM to predict the optical flow field. The optical flow field Input the generator module to obtain the enhanced feature image.
4. The method according to claim 1, characterized in that, The acquisition of image data includes: Using web crawler software, image data is scraped from publicly available datasets on video websites.
5. The method according to claim 1, characterized in that, The preprocessing of the image data specifically includes: The face and body regions are cropped based on the location coordinates of the image data in the dataset, and the resolution of the image data is set to 256x256.
6. The method according to claim 1, characterized in that, The process of inputting the preprocessed image data into the improved model FOMM-CBAM for training, and optimizing the model parameters according to the loss function to obtain the trained target model includes: 1) Using a pre-trained VGG-19 network as the perceptual loss, the driving frame D and the corresponding reconstructed frame are processed. The perceptual loss function is: in, The features are extracted from the i-th layer output of VGG-19; I is the total number of feature layers in VGG-19; downsampling is obtained by applying reconstruction loss at multiple resolutions. The image pyramids obtained by D have resolutions of 256×256, 128×128, 64×64 and 32×32, respectively; 2) Equal variance loss: in, It is an image After affine transformation matrix The transformed image, where, The value is random; 3) Generator loss: 4) Discriminator loss: Where Dis represents the discriminator, which is used to distinguish between reconstructed frames and driving frames; D is the reconstructed frame; D is the driving frame. The discriminator loss and generator loss are summed on the image pyramids with resolutions of 256×256, 128×128, 64×64 and 32×32, respectively. Using perceived loss Equal variance loss The model is trained with a weight of 10. After the model parameters are fitted, the key point detection module is frozen. The model is then trained again based on the perceptual loss, equal variance loss, discriminator loss, and generator loss.
7. The method according to claim 6, characterized in that, Also includes: The target model is evaluated as follows: Ensure that the pose of the driving object in the first frame is close to the pose of the source frame object; according to Motion pose change driving source frame ; If the pose of the first frame is similar to the pose of the object in the source frame, the motion information is transferred to the source frame through key point changes between driving frames: ； in, This indicates that key point changes between driving frames are transferred to motion information in the source frame; This represents the key points of the source frame; This indicates the key points of the first frame of the driving video; and This represents the key point in the t-th frame of the driving video; Denotes the affine transformation matrix. Represents pixel coordinates.
8. An image animation construction system based on an attention mechanism, characterized in that, include: The acquisition module is used to acquire image data; The preprocessing module is used to preprocess the image data; The training module is used to input the preprocessed image data into the improved model FOMM-CBAM for training, and optimize the model parameters according to the loss function to obtain the trained target model; The input module is used to input the image data into the target model to obtain an enhanced feature image; A synthesis module is used to synthesize the enhanced feature images into image animations; The improved model FOMM-CBAM includes: FOMM model and CBAM attention model; The CBAM attention model is used to be inserted into the FOMM model, and the FOMM model includes at least: a key point detection module, a dense network prediction module, and a generator module; Attention models should include at least: a channel attention module and a spatial attention module; The CBAM attention model is used for insertion into the FOMM model, specifically as follows: A convolutional attention model (CBAM) layer is inserted after the dense network prediction module and before the generator module, so that the data from the intermediate feature layers are processed by the convolutional attention model to obtain an enhanced feature image.