Dynamic image generation method and device, electronic equipment, storage medium and program product

Abstract

The invention relates to a dynamic image generation method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: acquiring a single-frame image and image features of the single-frame image; combining the image feature with a preset noise feature to obtain a first combined feature; combining the image features with preset mask features to obtain second combined features; combining the noise features with the mask features to obtain third combined features; based on the first combined feature, the second combined feature and the third combined feature, performing diffusion processing by using a preset image frame diffusion model to obtain an image sequence after single-frame image diffusion; wherein the image frame diffusion model is obtained based on deep learning network training; and generating a dynamic image based on the image sequence. The single-frame image is utilized, and the noise image features and the mask features are utilized to perform frame expansion, so that the dynamic image is generated, and the hardware demand and loss for generating the dynamic image from the multi-frame image are reduced.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a method and apparatus for generating dynamic images, electronic devices, storage media, and program products. Background Technology

[0002] With the continuous development of technology, more and more researchers are beginning to focus on the field of image-to-motion image generation. Motion images can be applied to various application scenarios, such as creating realistic virtual product models and scenes in advertising, or creating realistic virtual fitting rooms and scenes in marketing applications to help users shop and experience; or they can be applied to applications such as live wallpapers and themes, news and weather applications, to provide a better user experience.

[0003] However, current methods for generating dynamic images suffer from high power consumption and demanding hardware requirements. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a method and apparatus for generating dynamic images, an electronic device, a storage medium, and a program product. This aims to improve the problems of high power consumption and demanding hardware requirements in dynamic image generation.

[0005] According to a first aspect of the present disclosure, a method for generating dynamic images is provided, comprising:

[0006] Obtain a single frame image and its image features;

[0007] The image features are combined with preset noise features to obtain the first combined features;

[0008] The image features are combined with the preset mask features to obtain the second combined features;

[0009] The noise features are combined with the mask features to obtain the third combined feature;

[0010] Based on the first combined feature, the second combined feature, and the third combined feature, a preset image frame diffusion model is used to perform diffusion processing to obtain an image sequence after diffusion of a single frame image; wherein, the image frame diffusion model is obtained based on deep learning network training;

[0011] Dynamic images are generated based on image sequences.

[0012] In some embodiments, obtaining the image sequence after diffusion of a single frame image using a preset image frame diffusion model based on the first combined feature, the second combined feature, and the third combined feature includes:

[0013] The first feature processing layer of the image frame diffusion model is used to perform feature fusion diffusion processing on the first combined feature, the second combined feature, and the third combined feature to obtain a preset number of fused first feature maps; wherein, the preset number is an integer multiple of the predetermined target diffusion number, and the number of image frames included in the image sequence is the predetermined target diffusion number.

[0014] The image frame diffusion model uses convolutional layers to perform convolution processing on a preset number of first feature maps to obtain a preset number of second feature maps.

[0015] The second feature processing layer of the image frame diffusion model is used to perform spatiotemporal feature modeling on a preset number of second feature maps to obtain a third feature map with a predetermined target diffusion number.

[0016] Based on the third feature map of the predetermined target diffusion number, the image sequence after diffusion of a single frame image is obtained.

[0017] In some embodiments, the first feature processing layer using the image frame diffusion model performs feature fusion diffusion processing on the first combined features, the second combined features, and the third combined features to obtain a preset number of fused first feature maps, including:

[0018] The first feature processing layer performs the same normalization process on the first combined feature, the second combined feature, and the third combined feature to obtain the first normalized combined feature, the second normalized combined feature, and the third normalized combined feature; the feature points in the first normalized combined feature, the second normalized combined feature, and the third normalized combined feature are fused to obtain multiple fused feature maps.

[0019] For each fused feature map, feature dimension mapping is performed to obtain a preset number of fused first feature maps.

[0020] In some embodiments, the first combined feature includes multiple sets of first combined sub-features formed by combining image features with different noise features respectively; the second combined feature includes multiple sets of second combined sub-features formed by combining image features with different mask features respectively; the third combined feature includes multiple sets of third combined sub-features formed by combining each noise feature and each mask feature; the first normalized combined feature includes a first normalized combined sub-feature after normalizing each first set of sub-features; the second normalized combined feature includes a second normalized combined sub-feature after normalizing each second set of sub-features; and the third normalized combined feature includes a third normalized combined sub-feature after normalizing each third set of sub-features.

[0021] Based on the feature points from the first normalized combined feature, the second normalized combined feature, and the third normalized combined feature, multiple fused feature maps are obtained, including:

[0022] The feature points in each associated first normalized combined sub-feature, second normalized combined sub-feature, and third normalized combined sub-feature are fused to obtain the fused feature map corresponding to each associated combined sub-feature; wherein the associated first normalized combined sub-feature, second normalized combined sub-feature, and third normalized combined sub-feature include the same noise feature and mask feature.

[0023] In some embodiments, fusing the feature points in each associated first normalized combined sub-feature, second normalized combined sub-feature, and third normalized combined sub-feature among the first normalized combined feature, second normalized combined feature, and third normalized combined feature to obtain a fused feature map corresponding to each associated combined sub-feature includes:

[0024] Determine the feature mean corresponding to each associated first normalized combinatorial sub-feature, second normalized combinatorial sub-feature, and third normalized combinatorial sub-feature, and determine the weighting coefficient corresponding to the feature mean;

[0025] Based on the feature values ​​at the same position in each associated first normalized combinatorial feature, second normalized combinatorial feature, and third normalized combinatorial feature, a weighted fusion is performed using each weighting coefficient to obtain the fused feature map corresponding to each associated combinatorial feature.

[0026] In some embodiments, among the multiple sets of first combined sub-features included in the first combined feature, the variation amplitude of the noise feature is negatively correlated with the predetermined target diffusion quantity.

[0027] In some embodiments, the second feature processing layer includes a plurality of connected feature processing groups, each feature processing group consisting of a first sub-feature processing layer and a second sub-feature processing layer connected together.

[0028] The second feature processing layer of the image frame diffusion model is used to perform spatiotemporal feature modeling on a predetermined number of second feature maps to obtain a third feature map with a predetermined target diffusion number, including:

[0029] For each feature processing group in the second feature processing layer, the first sub-feature processing layer in the feature processing group is used to construct the correlation between the input feature maps to obtain multiple time-modeling feature maps; wherein, the feature map input to the first sub-feature processing layer of the first feature processing group is a preset number of second feature maps; the feature map input to the first sub-feature processing layer of the feature processing group after the first one is the feature map output of the adjacent previous feature processing group and image features.

[0030] For each feature processing group in the second feature processing layer, the second sub-feature processing layer in the feature processing group is used to transpose each time modeling feature map and construct the correlation between feature points in the transposed time modeling feature map to obtain multiple spatial modeling feature maps. Based on the multiple spatial modeling feature maps output by the second sub-feature processing layer of the last feature processing group, a third feature map of the predetermined target diffusion number is obtained.

[0031] In some embodiments, the multiple connected feature processing groups are divided into a first part and a second part, the first part and the second part include the same number of feature processing groups, and the second part is connected after the first part; wherein, the feature processing groups in the first part are used to encode the input feature map in sequence; the feature processing groups in the second part are used to decode the input feature map in sequence.

[0032] In some embodiments, the method further includes:

[0033] The feature map output by the last feature processing group in the first part is regularized to obtain the regularized feature map.

[0034] Based on the third feature map with a predetermined target diffusion amount, the image sequence after diffusion of a single frame image is obtained, including:

[0035] The third feature map based on the predetermined target diffusion number and the regularized feature map are used for decoding to obtain the image sequence after diffusion of a single frame image.

[0036] In some embodiments, the image frame diffusion model includes multiple second feature processing layers;

[0037] The second feature processing layer of the image frame diffusion model is used to perform spatiotemporal feature modeling on a predetermined number of second feature maps to obtain a third feature map with a predetermined target diffusion number, including:

[0038] Spatiotemporal feature modeling is performed on a preset number of second feature maps using multiple second feature processing layers to obtain a third feature map with a predetermined target diffusion number. The input of the first second feature processing layer is the preset number of second feature maps, and the input of the second feature processing layer after the first one is the output of the adjacent second feature processing layer. The third feature map with the predetermined target diffusion number is obtained based on the output of the last second feature processing layer.

[0039] In some embodiments, the method further includes:

[0040] Obtain the style features corresponding to the preset style;

[0041] Based on the first combination feature, the second combination feature, and the third combination feature, a preset image frame diffusion model is used for diffusion processing to obtain the image sequence after diffusion of a single frame image, including:

[0042] Based on the first combination feature, the second combination feature, the third combination feature, and the style feature, a preset image frame diffusion model is used to perform diffusion processing to obtain an image sequence with a preset style after diffusion of a single frame image.

[0043] In some embodiments, acquiring a single-frame image and its image features includes:

[0044] A single-frame image is acquired, and the single-frame image is encoded and compressed to obtain the image features of the single-frame image.

[0045] In some embodiments, acquiring a single-frame image and its image features includes:

[0046] In response to the detection of a photo capture command, a single-frame image and its image features are acquired.

[0047] Based on image sequences, dynamic images are generated, including:

[0048] The image sequence is rendered and then displayed as a dynamic image.

[0049] In some embodiments, the method further includes:

[0050] The settings interface based on dynamic images retrieves the settings parameters associated with the dynamic images; these settings parameters include: parameters indicating the frame rate, parameters indicating the rendering method, and parameters indicating the image quality.

[0051] The process involves rendering an image sequence and generating a dynamic image for display, including:

[0052] The system renders each image frame in the image sequence according to the rendering method and image quality indicated by the settings parameters, and then combines the rendered image frames into a dynamic image for display according to the indicated frame rate.

[0053] According to a second aspect of the present disclosure, a dynamic image generation apparatus is provided, comprising:

[0054] The acquisition unit is configured to acquire a single-frame image and the image features of the single-frame image;

[0055] The first combination unit is configured to combine image features with preset noise features to obtain a first combined feature;

[0056] The second combination unit is configured to combine image features with preset mask features to obtain a second combined feature;

[0057] The third combination unit is configured to combine noise features with mask features to obtain a third combined feature;

[0058] The diffusion unit is configured to perform diffusion processing based on a first combination feature, a second combination feature, and a third combination feature using a preset image frame diffusion model to obtain an image sequence after diffusion of a single frame image; wherein, the image frame diffusion model is obtained based on deep learning network training;

[0059] The generation unit is configured to generate dynamic images based on image sequences.

[0060] In some embodiments, the apparatus further includes:

[0061] The first feature processing layer of the image frame diffusion model is used to perform feature fusion diffusion processing on the first combined feature, the second combined feature, and the third combined feature to obtain a preset number of fused first feature maps; wherein, the preset number is an integer multiple of the predetermined target diffusion number, and the number of image frames included in the image sequence is the predetermined target diffusion number.

[0062] The image frame diffusion model uses convolutional layers to perform convolution processing on a preset number of first feature maps to obtain a preset number of second feature maps.

[0063] The second feature processing layer of the image frame diffusion model is used to perform spatiotemporal feature modeling on a preset number of second feature maps to obtain a third feature map with a predetermined target diffusion number.

[0064] Based on the third feature map of the predetermined target diffusion number, the image sequence after diffusion of a single frame image is obtained.

[0065] In some embodiments, the apparatus further includes:

[0066] The first feature processing layer performs the same normalization process on the first combined feature, the second combined feature, and the third combined feature to obtain the first normalized combined feature, the second normalized combined feature, and the third normalized combined feature; the feature points in the first normalized combined feature, the second normalized combined feature, and the third normalized combined feature are fused to obtain multiple fused feature maps.

[0067] For each fused feature map, feature dimension mapping is performed to obtain a preset number of fused first feature maps.

[0068] In some embodiments, the first combined feature includes multiple sets of first combined sub-features formed by combining image features with different noise features respectively; the second combined feature includes multiple sets of second combined sub-features formed by combining image features with different mask features respectively; the third combined feature includes multiple sets of third combined sub-features formed by combining each noise feature and each mask feature; the first normalized combined feature includes a first normalized combined sub-feature after normalizing each first set of sub-features; the second normalized combined feature includes a second normalized combined sub-feature after normalizing each second set of sub-features; and the third normalized combined feature includes a third normalized combined sub-feature after normalizing each third set of sub-features.

[0069] Based on the feature points from the first normalized combined feature, the second normalized combined feature, and the third normalized combined feature, multiple fused feature maps are obtained, including:

[0070] The feature points in each associated first normalized combined sub-feature, second normalized combined sub-feature, and third normalized combined sub-feature are fused to obtain the fused feature map corresponding to each associated combined sub-feature; wherein the associated first normalized combined sub-feature, second normalized combined sub-feature, and third normalized combined sub-feature include the same noise feature and mask feature.

[0071] In some embodiments, the apparatus further includes:

[0072] Determine the feature mean corresponding to each associated first normalized combinatorial sub-feature, second normalized combinatorial sub-feature, and third normalized combinatorial sub-feature, and determine the weighting coefficient corresponding to the feature mean;

[0073] Based on the feature values ​​at the same position in each associated first normalized combinatorial feature, second normalized combinatorial feature, and third normalized combinatorial feature, a weighted fusion is performed using each weighting coefficient to obtain the fused feature map corresponding to each associated combinatorial feature.

[0074] In some embodiments, among the multiple sets of first combined sub-features included in the first combined feature, the variation amplitude of the noise feature is negatively correlated with the predetermined target diffusion quantity.

[0075] In some embodiments, the apparatus further includes:

[0076] The second feature processing layer of the image frame diffusion model is used to perform spatiotemporal feature modeling on a predetermined number of second feature maps to obtain a third feature map with a predetermined target diffusion number, including:

[0077] For each feature processing group in the second feature processing layer, the first sub-feature processing layer in the feature processing group is used to construct the correlation between the input feature maps to obtain multiple time-modeling feature maps; wherein, the feature map input to the first sub-feature processing layer of the first feature processing group is a preset number of second feature maps; the feature map input to the first sub-feature processing layer of the feature processing group after the first one is the feature map output of the adjacent previous feature processing group and image features.

[0078] For each feature processing group in the second feature processing layer, the second sub-feature processing layer in the feature processing group is used to transpose each time modeling feature map and construct the correlation between feature points in the transposed time modeling feature map to obtain multiple spatial modeling feature maps. Based on the multiple spatial modeling feature maps output by the second sub-feature processing layer of the last feature processing group, a third feature map of the predetermined target diffusion number is obtained.

[0079] In some embodiments, the multiple connected feature processing groups are divided into a first part and a second part, the first part and the second part include the same number of feature processing groups, and the second part is connected after the first part; wherein, the feature processing groups in the first part are used to encode the input feature map in sequence; the feature processing groups in the second part are used to decode the input feature map in sequence.

[0080] In some embodiments, the apparatus further includes:

[0081] The feature map output by the last feature processing group in the first part is regularized to obtain the regularized feature map.

[0082] Based on the third feature map with a predetermined target diffusion amount, the image sequence after diffusion of a single frame image is obtained, including:

[0083] The third feature map based on the predetermined target diffusion number and the regularized feature map are used for decoding to obtain the image sequence after diffusion of a single frame image.

[0084] In some embodiments, the apparatus further includes using multiple second feature processing layers to iteratively model a preset number of second feature maps to obtain a third feature map with a predetermined target diffusion number; wherein the input of the first second feature processing layer is the preset number of second feature maps, the input of the second feature processing layer after the first one is the output of the adjacent preceding second feature processing layer, and the third feature map with the predetermined target diffusion number is obtained based on the output of the last second feature processing layer.

[0085] In some embodiments, the apparatus further includes:

[0086] Obtain the style features corresponding to the preset style;

[0087] Based on the first combined feature, the second combined feature, and the third combined feature, a preset image frame diffusion model is used for diffusion processing to obtain the image sequence after diffusion of a single frame image, including:

[0088] Based on the first combination feature, the second combination feature, the third combination feature, and the style feature, a preset image frame diffusion model is used to perform diffusion processing to obtain an image sequence with a preset style after diffusion of a single frame image.

[0089] In some embodiments, the apparatus further includes:

[0090] A single-frame image is acquired, and the single-frame image is encoded and compressed to obtain the image features of the single-frame image.

[0091] In some embodiments, the apparatus further includes:

[0092] In response to the detection of a photo capture command, a single-frame image and its image features are acquired.

[0093] Based on image sequences, dynamic images are generated, including:

[0094] The image sequence is rendered and then displayed as a dynamic image.

[0095] In some embodiments, the apparatus further includes:

[0096] The settings interface based on dynamic images retrieves the settings parameters associated with the dynamic images; these settings parameters include: parameters indicating the frame rate, parameters indicating the rendering method, and parameters indicating the image quality.

[0097] The process involves rendering an image sequence and generating a dynamic image for display, including:

[0098] The system renders each image frame in the image sequence according to the rendering method and image quality indicated by the settings parameters, and then combines the rendered image frames into a dynamic image for display according to the indicated frame rate.

[0099] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0100] processor;

[0101] Memory used to store computer programs or instructions;

[0102] The processor executes the computer program or instructions to implement the steps of the method described in any one of the first aspects above.

[0103] According to a fourth aspect of the embodiments of this disclosure, a non-transitory computer-readable storage medium is provided.

[0104] The storage medium stores a computer program or instructions, which, when executed by a processor, implement the steps of the method described in any one of the first aspects. According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program or instructions, which, when executed by a processor, implement the steps of the method described in any one of the first aspects.

[0105] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0106] In the dynamic image generation method proposed in this embodiment, the electronic device generates a dynamic image by using only a single frame image and expanding the frame using noise image features and mask features, thereby reducing the hardware requirements and losses for generating dynamic images from multiple frames.

[0107] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0108] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0109] Figure 1 This is a flowchart illustrating a dynamic image generation method according to an exemplary embodiment.

[0110] Figure 2 This is a schematic diagram of a dynamic image generation method according to an embodiment of the present disclosure.

[0111] Figure 3 This is a flowchart illustrating a process for generating a first feature map in an embodiment of this disclosure.

[0112] Figure 4 This is an example diagram illustrating an application scenario of dynamic image generation in an embodiment of this disclosure.

[0113] Figure 5 This is a block diagram of a motion image generation apparatus according to an exemplary embodiment.

[0114] Figure 6 This is a frame of an apparatus shown according to an exemplary embodiment. Figure 1 .

[0115] Figure 7 This is a frame of an apparatus shown according to an exemplary embodiment. Figure 2 . Detailed Implementation

[0116] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0117] Several dynamic image generation methods exist in related technologies. For example, in Scheme 1, redundant image frames are identified based on the similarity between adjacent image frames in the image frame sequence of the dynamic image to be generated, and then these redundant frames are removed from the image frame sequence to generate the dynamic image. The drawback of this scheme is that the generated dynamic image has few dynamic regions and limited dynamic amplitude, failing to meet practical needs. Its final effect is similar to a perturbation of a static image, and may also cause noise and distortion. Scheme 2, for instance, acquires the target image and dynamic effect description information, and generates semantic weight information based on the dynamic effect description information. Using a pre-trained model, the static and dynamic layers of the target image are separated based on the semantic weight information, and semantic category information is obtained. Then, using the dynamic layer and semantic category information, a dynamic image optical flow map is generated. Finally, based on the target image, the dynamic image optical flow map, the static layer, the dynamic layer, and the semantic category information, a dynamic image of the target image is generated. The disadvantages of this scheme are that it is not suitable for real-time processing of camera images, the connection between different pre-trained models may lead to poor results, it does not include rendering effects, it cannot perform real-time image processing, and it is difficult to integrate on the edge. For example, in Scheme 3, a first still image and an image sequence within a predetermined time period are acquired at the start of the shooting. The size of the first still image is adjusted to match the second still images in the image sequence. Then, the adjusted first still image is inserted into the new image sequence to finally generate a dynamic image. The disadvantages of this scheme are that it requires multiple still images, resulting in high power consumption and high hardware requirements. The generated effect is monotonous, lacks any dynamic rendering technology, and has poor short-term dynamic effects.

[0118] In this regard, this disclosure presents a method for generating dynamic images.

[0119] Figure 1 This is a flowchart illustrating a dynamic image generation method according to an exemplary embodiment. For example... Figure 1 As shown, the method mainly includes the following steps:

[0120] In step 101, a single-frame image is acquired, and the image features of the single-frame image are extracted;

[0121] In step 102, the image features are combined with preset noise features to obtain the first combined features;

[0122] In step 103, the image features are combined with preset mask features to obtain the second combined features;

[0123] In step 104, the noise features are combined with the mask features to obtain the third combined features;

[0124] In step 105, based on the first combined features, the second combined features, and the third combined features, a preset image frame diffusion model is used for diffusion processing to obtain an image sequence after diffusion of a single frame image; wherein, the image frame diffusion model is obtained based on deep learning network training;

[0125] In step 106, a dynamic image is generated based on the image sequence.

[0126] Here, a single frame image can be a high-resolution static image. Compared to low-resolution images, high-resolution images can provide more pixel information, which helps to generate clear and detailed dynamic images, thereby improving the dynamic effect. Furthermore, high-fidelity dynamic images can be generated through high-resolution images.

[0127] It should be noted that the number of channels in a single frame of an image can be different values.

[0128] For example, a single frame image has one channel, which is a grayscale image; when a single frame image has three channels, it is an RGB image.

[0129] In related techniques, to improve consistency with a given image, some methods encode the image as high-level image cues and inject them into the model via cross-attention. However, due to the use of only high-order semantics and resulting in excessive detail, such methods struggle to achieve fine-grained consistency. Furthermore, concatenating additional image information into the input image, while providing finer-grained low-level information, makes convergence more difficult and produces less stable dynamic images.

[0130] In this embodiment of the disclosure, in order to inject image information, noise features (noise feature maps) and mask features (mask feature maps) are added to the image features. The number of image features and noise features, or image features and mask features, or noise features and mask features is related to the frame rate of the generated dynamic image; the higher the frame rate of the generated dynamic image, the more feature maps are generated.

[0131] For example, the image feature maps of 3 single-frame images and 3 noise feature maps are concatenated into 6 feature images, or the image feature maps of 3 single-frame images and 3 noise feature maps are concatenated into 6 feature images, or the noise feature maps of 3 single-frame images and 3 noise feature maps are concatenated into 6 feature images. In order to increase redundancy, the 6 feature images are mapped to 9 feature images. Of course, the 9 feature images here are not fixed. For example, when generating dynamic images at a high frame rate, in order to make the picture more detailed, the 9 feature images may be mapped to values ​​greater than 9.

[0132] Here, the preset noise features are not learned automatically during training, but are generated by adding noise in stages to a single frame of the image. The number of stages for the preset noise features is determined by the number of video frames generated for the dynamic image; the more frames, the fewer stages.

[0133] Here, image features are combined with preset noise features through sequential concatenation to obtain the first combined feature. The noise features are used to control various attributes of the generated image, such as color, shape, and texture. By randomly adjusting the noise level of the noise features, images with different attributes can be generated to ensure the original image's control over dynamic video frames. For example, increasing the noise level of the noise features may generate images with more vibrant colors and more complex shapes; decreasing the noise level of the noise features may generate images with softer colors and simpler shapes.

[0134] Here, the preset mask features are obtained by performing a binary masking process on a single frame image.

[0135] For example, the binary masking process achieves a masking range of 25% of the entire image by randomly masking 3-5 adjacent feature points.

[0136] It should be noted that the preset mask features can be adjusted as needed. Since the binary masking process uses specific non-zero values ​​to represent regions of interest and zero values ​​to represent regions of no interest, effectively segmenting the image into visible and invisible regions, the desired preset mask features can be obtained by adjusting the regions corresponding to specific non-zero and zero values. This allows for the extraction, masking, or modification of specific regions within a single frame of an image.

[0137] Here, the image features and the preset mask features are combined sequentially to obtain the second combined features, which can realize the dynamic controllable range of dynamic images.

[0138] Here, noise features and mask features are combined sequentially to obtain a third combined feature. Since mask features cause loss in a single frame image, noise features are added to make the missing parts no longer 0, but have a reference value dependency. This ensures both the space for generating dynamic image content and the controllability of differences from the original image.

[0139] In this embodiment, a single-frame image is acquired, and image features of the single-frame image are extracted; the image features are combined with preset noise features to obtain a first combined feature; the image features are combined with preset mask features to obtain a second combined feature; the noise features are combined with mask features to obtain a third combined feature; based on the first combined feature, the second combined feature, and the third combined feature, a preset image frame diffusion model is used for diffusion processing to obtain an image sequence after diffusion of the single-frame image; wherein, the image frame diffusion model is obtained based on deep learning network training; and a dynamic image is generated based on the image sequence.

[0140] In other words, by using a single frame image and expanding the frame using noise image features and mask features, dynamic images can be generated, reducing the hardware requirements and overhead of generating dynamic images from multiple frames.

[0141] Figure 2 This is a schematic diagram illustrating the principle of a dynamic image generation method according to an embodiment of this disclosure. Figure 3 This is a flowchart illustrating a process for generating a first feature map according to an embodiment of this disclosure. Figure 4 This is an example diagram illustrating an application scenario of dynamic image generation according to an embodiment of this disclosure. The following embodiments will be combined with... Figure 2 , Figure 3 , Figure 4 Please provide an explanation.

[0142] In some embodiments, based on the first combined feature, the second combined feature, and the third combined feature, a preset image frame diffusion model is used to obtain an image sequence after diffusion of a single frame image, including:

[0143] The first feature processing layer of the image frame diffusion model is used to perform feature fusion diffusion processing on the first combined feature, the second combined feature, and the third combined feature to obtain a preset number of fused first feature maps; wherein, the preset number is an integer multiple of the predetermined target diffusion number, and the number of image frames included in the image sequence is the predetermined target diffusion number.

[0144] The image frame diffusion model uses convolutional layers to perform convolution processing on a preset number of first feature maps to obtain a preset number of second feature maps.

[0145] The second feature processing layer of the image frame diffusion model is used to perform spatiotemporal feature modeling on a preset number of second feature maps to obtain a third feature map with a predetermined target diffusion number.

[0146] Based on the third feature map of the predetermined target diffusion number, the image sequence after diffusion of a single frame image is obtained.

[0147] Here, the first feature processing layer of the image frame diffusion model is Figure 2 The masking fusion operator shown is used to expand a single frame image into multiple feature maps.

[0148] For example, the first combination of features is Figure 2 The F shown c The second combination of features is Figure 2 The F shown i And the third combination of features is Figure 2 The F shown m The convolutional layer of the image frame diffusion model is input into the masking fusion operator through three paths. Figure 2 The two-dimensional convolution (Conv2d) shown is the second feature processing layer of the image frame diffusion model. Figure 2 A1 is shown.

[0149] Here, the predetermined target diffusion number is the number of frames of the final dynamic image to be generated, which can be 16 frames, 24 frames, 32 frames, and 48 frames, etc. The more predetermined target diffusion number, the smoother and more continuous the visual effect of the generated dynamic image.

[0150] For example, if the predetermined target diffusion number is 24 frames, and each pair of feature maps (e.g., an image feature and a noise feature, an image feature and a mask feature, or a noise feature and a complete mask feature) is combined to map one frame, then 48 feature maps are needed to achieve 24 frames. Therefore, the mask fusion operator ultimately outputs feature maps that are integer multiples of 24 frames, i.e., feature maps that are integer multiples of the predetermined target diffusion number. This integer multiple relationship ensures that the feature maps between frames of the generated dynamic image can be accurately aligned and combined, thereby producing a smooth inter-frame transition and improving the stability of dynamic image generation. If the number is not an integer multiple, an additional neural network is required to perform the operation, which increases computational complexity and reduces computational speed.

[0151] Here, convolutional layers of the image frame diffusion model are used for convolution processing to enhance local feature effects and make features uniform.

[0152] Here, the second feature processing layer of the image frame diffusion model can be a space-time dependent Unet structure generator network.

[0153] For example, the second feature processing layer of the image frame diffusion model is Figure 2 A1 is shown.

[0154] In this embodiment, a preset image frame diffusion model is constructed by a first feature processing layer, a convolutional layer, and a second feature processing layer, achieving finer-grained modeling. The first, second, and third combined features of the input are processed by fusion diffusion, convolution, and spatiotemporal feature modeling respectively through each layer to achieve dynamic image generation. The image frame diffusion model can represent features of multiple frames of images, enabling fast feature calculation output and feature compression, thus accelerating computational efficiency while ensuring feature validity and achieving low-power deployment on the edge.

[0155] In some embodiments, the first feature processing layer of the image frame diffusion model performs feature fusion diffusion processing on the first combined features, the second combined features, and the third combined features to obtain a preset number of fused first feature maps, including:

[0156] The first feature processing layer performs the same normalization process on the first combined feature, the second combined feature, and the third combined feature to obtain the first normalized combined feature, the second normalized combined feature, and the third normalized combined feature; the feature points in the first normalized combined feature, the second normalized combined feature, and the third normalized combined feature are fused to obtain multiple fused feature maps.

[0157] For each fused feature map, feature dimension mapping is performed to obtain a preset number of fused first feature maps.

[0158] Here, normalization means standardizing the feature map. The same normalization process means that the first, second, and third normalized combined features have the same feature dimensions and feature value ranges. Having the same feature dimensions and feature value ranges means that the three combined features have been scaled and adjusted to the same scale.

[0159] For example, the feature value range can be [0,1] or [-1,1].

[0160] For example, such as Figure 3 As shown, the normalization process is the channel normalization in the figure, that is, the normalization of the three different sets of features F c F i and F m Normalization is applied to the same basis by calculating the mean and variance of each set of features and mapping them to a Gaussian-distributed feature space, ensuring the stability of low-rank pixel features. Furthermore, the normalized features are more easily processed by subsequent convolutional layers or other modules, improving the model's performance and accuracy.

[0161] Understandably, by fusing feature points from the first, second, and third normalized combined features, multiple fused feature maps are obtained. These maps can combine different feature information to enhance the features. The fused feature maps can better describe the essential features of the image, thereby improving the model's performance and accuracy.

[0162] For example, the fusion of three normalized combined features corresponds to Figure 3 The channel fusion part in the middle.

[0163] Here, a linear layer maps the feature dimensions of each fused feature map. The linear layer is a simple fully connected layer that multiplies the input feature map by a weight matrix and then adds a bias vector, which can be used to increase or decrease the feature dimension.

[0164] For example, such as Figure 3 As shown, each fused feature map is multiplied by a weight matrix through a linear layer, and then a bias vector is added to obtain a preset number of fused first feature maps.

[0165] In this embodiment, by performing the same normalization operation on the three combined features and fusing the three normalized combined features after the normalization operation, and performing feature dimension mapping, the stability of low-rank pixel features is ensured, the features are enhanced, the channel dimension is increased, and the performance and accuracy of the model are improved. Moreover, since the three combined features contain the image features of a single frame image, fine-grained image detail information can be recovered, which is beneficial to improving the fidelity of the generated dynamic image.

[0166] In some embodiments, the first combined feature includes multiple sets of first combined sub-features formed by combining image features with different noise features respectively; the second combined feature includes multiple sets of second combined sub-features formed by combining image features with different mask features respectively; the third combined feature includes multiple sets of third combined sub-features formed by combining each noise feature and each mask feature; the first normalized combined feature includes a first normalized combined sub-feature after normalizing each first set of sub-features; the second normalized combined feature includes a second normalized combined sub-feature after normalizing each second set of sub-features; and the third normalized combined feature includes a third normalized combined sub-feature after normalizing each third set of sub-features.

[0167] Based on the feature points from the first normalized combined feature, the second normalized combined feature, and the third normalized combined feature, multiple fused feature maps are obtained, including:

[0168] The feature points in each associated first normalized combined sub-feature, second normalized combined sub-feature, and third normalized combined sub-feature are fused to obtain the fused feature map corresponding to each associated combined sub-feature; wherein the associated first normalized combined sub-feature, second normalized combined sub-feature, and third normalized combined sub-feature include the same noise feature and mask feature.

[0169] Here, the first set of sub-features consists of multiple sets, each set consisting of an image feature and a noise feature; the second set of sub-features consists of multiple sets, each set consisting of an image feature and a mask feature; the third set of sub-features consists of multiple sets, each set consisting of a noise feature and a mask feature.

[0170] For example, such as Figure 2 As shown, the first combined feature F c Composed of two image features and two noise features, the image features and noise features form two sets of first combined sub-features; the second combined feature F i Composed of two image features and two mask features, the image features and mask features form two sets of second-group sub-features; the third-group feature F m Composed of two noise features and two mask features, the noise features and mask features form two sets of third sub-features.

[0171] Here, since there are multiple sets of the first, second, and third combined features, the normalized combined features, the second, and third normalized combined features after normalization processing each include multiple first, second, and third normalized combined features.

[0172] For example, if there are two sets of each of the first, second, and third combined sub-features, then the first normalized combined feature includes two first normalized combined sub-features, the second normalized combined feature includes two second normalized combined sub-features, and the third normalized combined feature includes two second normalized combined sub-features.

[0173] It is understandable that feature fusion can be achieved based on the same noise features and mask features of the first normalized combined sub-feature, the second normalized combined sub-feature, and the third normalized combined sub-feature, since the first normalized combined sub-feature contains noise features, the second normalized combined sub-feature contains mask features, and the third normalized combined sub-feature contains both noise features and mask features.

[0174] For example, we select the noise feature and mask feature from each combined sub-feature as associated sub-feature pairs. For each associated sub-feature pair, we can choose the following fusion methods: Method 1: Concatenation: Directly concatenate the two sub-feature points together to form a longer feature vector. Method 2: Average: Calculate the average of the two sub-feature points as the fused feature points. Method 3: Weighted Average: Assign weights to the two sub-feature points according to their importance, and then perform a weighted average. The fused feature points are combined into a new feature map, i.e., the fused feature map.

[0175] In this embodiment of the disclosure, by associating combined sub-features with noise features and mask features, image quality can be improved, dynamic range and color performance can be enhanced, noise processing capabilities and generalization capabilities can be improved, thereby improving the quality and performance of dynamic images.

[0176] In some embodiments, feature points in each associated first normalized combined sub-feature, second normalized combined sub-feature, and third normalized combined sub-feature among the first normalized combined feature, second normalized combined feature, and third normalized combined feature are fused to obtain a fused feature map corresponding to each associated combined sub-feature, including:

[0177] Determine the feature mean corresponding to each associated first normalized combinatorial sub-feature, second normalized combinatorial sub-feature, and third normalized combinatorial sub-feature, and determine the weighting coefficient corresponding to the feature mean;

[0178] Based on the feature values ​​at the same position in each associated first normalized combinatorial feature, second normalized combinatorial feature, and third normalized combinatorial feature, a weighted fusion is performed using each weighting coefficient to obtain the fused feature map corresponding to each associated combinatorial feature.

[0179] Here, the weighting coefficient can be the reciprocal of the characteristic mean.

[0180] The following formula (1) is an example of a weighted fusion method:

[0181]

[0182] Here, M*N refers to the feature dimension, k and j represent the row index and column index of the feature point, and X t It is the final output of taking the average of each feature point. It should be noted that S in formula (1) is only used to represent weighting and is not equivalent to the weighting coefficient of the present disclosure embodiment.

[0183] In this embodiment of the disclosure, the characteristic mean refers to F. c First normalized combinatorial feature, F i The second normalized combinatorial feature, F mThe feature mean S corresponding to the third normalized combinatorial features c S i and S m The weighting coefficients corresponding to the feature mean refer to 1 / S c 1 / S i and 1 / S m Moreover, due to F c F m F i Each feature has multiple first normalized combinatorial features, second normalized combinatorial features, and third normalized combinatorial features. Therefore, the feature values ​​at the same position in each associated first normalized combinatorial feature, second normalized combinatorial feature, and third normalized combinatorial feature are weighted and fused using each weighting coefficient.

[0184] In this embodiment of the disclosure, the fusion of three sets of features is achieved by first taking the average value within each set of combined features, and then taking the average value among the combined features.

[0185] In some embodiments, among the multiple sets of first combined sub-features included in the first combined feature, the variation amplitude of the noise feature is negatively correlated with the predetermined target diffusion quantity.

[0186] It's understandable that the more frames an image has, the smaller the time interval between images and the smaller the content variation. Therefore, only noise with small amplitude variation needs to be added. Excessive noise variation might blur image details and degrade video quality. Conversely, the fewer frames an image has, the larger the time interval between each image and the greater the variation between each image. To smooth these jumps and simulate continuous motion, it might be necessary to add noise with large amplitude variation between images. Therefore, the amplitude of noise feature variation is negatively correlated with the amount of diffusion of the intended target.

[0187] For example, if a 1-second video consists of 5 images, noise features with large variations need to be added between the images. However, if a 1-second video consists of 100 images, noise features with small variations need to be added.

[0188] In this embodiment of the disclosure, by adding noise features with varying amplitudes at different predetermined target diffusion amounts, the quality of the generated dynamic image can be more precisely controlled. The amplitude and diffusion amount of the noise features can be adjusted according to actual needs to achieve the best visual effect.

[0189] In some embodiments, the second feature processing layer includes a plurality of connected feature processing groups, each feature processing group consisting of a first sub-feature processing layer and a second sub-feature processing layer connected together.

[0190] The second feature processing layer of the image frame diffusion model is used to perform spatiotemporal feature modeling on a predetermined number of second feature maps to obtain a third feature map with a predetermined target diffusion number, including:

[0191] For each feature processing group in the second feature processing layer, the first sub-feature processing layer in the feature processing group is used to construct the correlation between the input feature maps to obtain multiple time-modeling feature maps; wherein, the feature map input to the first sub-feature processing layer of the first feature processing group is a preset number of second feature maps; the feature map input to the first sub-feature processing layer of the feature processing group after the first one is the feature map output of the adjacent previous feature processing group and image features.

[0192] For each feature processing group in the second feature processing layer, the second sub-feature processing layer in the feature processing group is used to transpose each time modeling feature map and construct the correlation between feature points in the transposed time modeling feature map to obtain multiple spatial modeling feature maps. Based on the multiple spatial modeling feature maps output by the second sub-feature processing layer of the last feature processing group, a third feature map of the predetermined target diffusion number is obtained.

[0193] Here, the feature processing group consists of a spatiotemporal encoder and a spatiotemporal decoder for spatiotemporal feature modeling. The first sub-feature processing layer can be a temporal encoder and a spatial encoder to ensure the smoothness of the temporal dimension and the continuity of time to prevent misalignment, thus minimizing the differences between adjacent frames and making the transition more natural. The second sub-feature processing layer, adjacent to the first sub-feature processing layer, can be a temporal decoder and a spatial decoder to focus on more differentiated features.

[0194] For example, such as Figure 2 As shown, the feature processing group includes: a first feature processing group composed of B1 and B2, a second feature processing group composed of B3 and B4, a first feature processing group composed of C1 and C2, and a fourth feature processing group composed of C3 and C4. Among them, B1, B3, C1, and C3 are the first sub-feature processing layers; B2, B4, C2, and C4 are the second sub-feature processing layers.

[0195] Understandably, since there are multiple feature processing groups, there are multiple first sub-feature processing layers, and therefore, multiple time modeling plots can be obtained.

[0196] For example, such as Figure 2 As shown, a preset number of second feature maps after convolution are input into the time encoder B1 of the first sub-feature processing layer to obtain a time-modeling feature map. Then, the map is input into the time encoder B3 of the second sub-feature processing layer to perform time modeling and obtain a time-modeling feature map.

[0197] For example, a preset number of second feature maps are input into the temporal encoder. This can be achieved by performing Transformer computation in the time dimension, transposing the time dimension, and then performing the same network computation to map it into a temporal modeling feature map. The feature processing for the spatial encoder is similar to that of the temporal encoder; the spatial modeling feature map can be obtained by mapping the transposed feature map.

[0198] Here, the feature map of the first feature processing group, which is used for spatiotemporal modeling, is concatenated with the image features and then input into the next feature processing group.

[0199] For example, such as Figure 2 As shown, the second feature map, obtained through convolution operations and having a predetermined number of features, is input into the temporal encoder B1 of the first sub-feature processing layer for processing to obtain a temporal modeling feature map. The temporal modeling feature map is then input into the spatial encoder B2 of the first sub-feature processing layer for processing to obtain a spatial modeling feature map. This spatial modeling feature map is then concatenated with the image features encoded by the encoder and input together into the... Figure 2 The second feature processing group (B3 and B4) in the model is then used for spatiotemporal modeling.

[0200] Here, image features are extracted by the encoder from a single frame of image.

[0201] For example, the encoder is Figure 2 The image encoder shown can be a VAE-Encoder, or it can inject high-level image features in the form of cross-attention.

[0202] Understandably, since the image features are extracted from single-frame images, it's crucial to incorporate more image features from individual frames to improve the fidelity of the generated image. Here, the Structural Similarity Index Measure (SSIM) is used to evaluate fidelity.

[0203] Understandably, since there are multiple feature processing groups, it will undergo multiple spatiotemporal modeling processes. The multiple spatial modeling feature maps obtained from the second sub-feature processing layer of the last feature processing group will be used as outputs, and a third feature map with a predetermined target diffusion number will be obtained.

[0204] For example, such as Figure 2 As shown, the last feature processing group of multiple feature processing groups is the fourth feature processing layer (C3 and C4), which takes the spatial decoder C4 of the second sub-feature processing layer of the fourth feature processing group as the output and spreads it to the third feature map of the predetermined target number.

[0205] Understandably, due to multiple spatiotemporal modeling iterations, the generated dynamic images exhibit greater temporal coherence and consistency, resulting in higher dynamic intensity. Here, temporal coherence and consistency are evaluated using Contrastive Language-Image Pre-training (CLIP) scores, which measure temporal coherence and consistency. Dynamic intensity is represented by the Recurrent All-Pairs Field Transforms (RAFT) score, which indicates the flow between adjacent frames of the dynamic image.

[0206] In this embodiment of the disclosure, by inputting a predetermined number of second feature maps into a second feature processing layer composed of multiple connected feature processing groups to perform multiple temporal and spatial modeling, the dynamic amplitude effect of the generated dynamic image is enhanced, the fidelity and the temporal coherence and consistency of the generated image are improved, and low-power deployment on the edge is achieved.

[0207] In some embodiments, the multiple connected feature processing groups are divided into a first part and a second part, the first part and the second part include the same number of feature processing groups, and the second part is connected after the first part; wherein, the feature processing groups in the first part are used to encode the input feature map in sequence; the feature processing groups in the second part are used to decode the input feature map in sequence.

[0208] Here, the encoding process in the first part can reduce the dimensionality of the feature map, improve the computational efficiency and generalization ability of the model, and remove redundant information and noise from the feature map, allowing the model to focus more on learning key features, thereby improving the model's accuracy and robustness. The encoding process is often accompanied by feature transformation and enhancement, making features of different categories more distinguishable in the encoded space, which helps the model to classify and recognize better.

[0209] The second part of the decoding process can restore the dimension of the first part of the encoding process, thereby increasing the dimension of the feature map and obtaining a third feature map with the predetermined target diffusion number.

[0210] Here, the first and second parts can be symmetrically funnel-shaped.

[0211] For example, the first part is Figure 2 As shown in A11, the first part is Figure 2 As shown in A12. The first part A11 consists of the first feature processing group (B1, B2) and the second feature processing group (B3, B4). The first part A12 consists of the third feature processing group (C1, C2) and the fourth feature processing group (C3, C4).

[0212] In this embodiment of the disclosure, a third feature map with a predetermined target diffusion number is obtained through the first part of encoding processing and the second part of decoding processing, which improves the feature extraction capability of the model and realizes low-power deployment on the edge.

[0213] In some embodiments, the method further includes:

[0214] The feature map output by the last feature processing group in the first part is regularized to obtain the regularized feature map.

[0215] Based on the third feature map with a predetermined target diffusion amount, the image sequence after diffusion of a single frame image is obtained, including:

[0216] The third feature map based on the predetermined target diffusion number and the regularized feature map are used for decoding to obtain the image sequence after diffusion of a single frame image.

[0217] Here, regularization is mainly used to reduce overfitting in neural networks. It is achieved by randomly dropping a portion of neurons (including their connections) out of the network temporarily (i.e., setting them to 0 or "deactivated").

[0218] For example, such as Figure 2 As shown, dropout, a regularization technique with a random number of 0.3, is used to regularize the feature maps output by the second feature processing group (B3, B4). However, inactive neurons do not participate in the computation at all, which may lead to losses.

[0219] It is understandable that the feature map output by the last feature processing group in the first part is regularized because the feature processing layer has a funnel-shaped structure that makes the network compression smaller and smaller, making it the part with the least compression. Therefore, the amount of computation can be reduced, and the effectiveness of the features is also high due to multiple compressions at this position.

[0220] Here, the loss from regularization can be compensated by decoding the third feature map of the predetermined target diffusion number and the regularized feature map.

[0221] Understandably, while regularization can address overfitting, it can lead to a loss of accuracy or relevant content. The third feature map representing the predetermined target diffusion number is obtained through a multi-layer neural network, which can also result in loss due to feature distortion. However, combining the third feature map representing the predetermined target diffusion number with the regularized feature map can compensate for each other's losses, resulting in better dynamic image generation.

[0222] For example, the feature map after compressing the features and dropping out by 0.3 is as follows: Figure 2The compression regularization (0.3) shown and the third feature map with the predetermined target diffusion number are input together into the decoder for decoding, which can be a Fu-Decoder.

[0223] It should be noted that, in order to further improve computational efficiency, the last two layers of computation in the original pre-trained decoder were removed.

[0224] In this embodiment of the disclosure, by decoding the third feature map with a predetermined target diffusion number and the feature map after regularization, the computational efficiency is accelerated, real-time dynamic image generation is achieved, and the dynamic effect of the generated dynamic image is improved.

[0225] In some embodiments, the image frame diffusion model includes multiple second feature processing layers;

[0226] The second feature processing layer of the image frame diffusion model is used to perform spatiotemporal feature modeling on a predetermined number of second feature maps to obtain a third feature map with a predetermined target diffusion number, including:

[0227] Spatiotemporal feature modeling is performed on a preset number of second feature maps using multiple second feature processing layers to obtain a third feature map with a predetermined target diffusion number. The input of the first second feature processing layer is the preset number of second feature maps, and the input of the second feature processing layer after the first one is the output of the adjacent second feature processing layer. The third feature map with the predetermined target diffusion number is obtained based on the output of the last second feature processing layer.

[0228] Understandably, because noisy features and mask features are incorporated during the dynamic generation process, generating content-rich and meaningful images from these features requires one or more denoising processes. This means iterative denoising is performed multiple times, so the second feature processing layer can include multiple layers. The more second feature processing layers there are, the better the denoising effect and the better the generated dynamic image.

[0229] Furthermore, due to the reference of a single-frame image, the generated dynamic images have high fidelity. Even if multiple second feature processing layers are required, the generation speed of dynamic images is faster due to fewer processing steps.

[0230] For example, the feature maps of the first combined features, the second combined features, and the third combined features after masking and fusion and convolution are input into the Unet generation network for processing, and then the feature maps output by the Unet generation network are input into the Unet generation network again.

[0231] According to the embodiments disclosed herein, denoising can be gradually achieved and long dynamic range images can be generated through an iterative approach.

[0232] In some embodiments, the method further includes:

[0233] Obtain the style features corresponding to the preset style;

[0234] Based on the first combined feature, the second combined feature, and the third combined feature, a preset image frame diffusion model is used for diffusion processing to obtain the image sequence after diffusion of a single frame image, including:

[0235] Based on the first combination feature, the second combination feature, the third combination feature, and the style feature, a preset image frame diffusion model is used to perform diffusion processing to obtain an image sequence with a preset style after diffusion of a single frame image.

[0236] Here, style refers to a series of visual elements and expressive techniques that collectively determine the overall look and feel of an image. Preset styles can include at least one of the following: animation style, visual art style, color style, lighting style, time and rhythm style, and sound and music style. Different preset styles correspond to different stylistic characteristics.

[0237] For example, if the preset style is selected as animation style, then the animation style features corresponding to the animation style are obtained. The first combination feature, the second combination feature, the third combination feature and the animation style feature are input together into the preset image frame diffusion model for diffusion processing to obtain an image sequence with animation style after single frame image diffusion. Then, a stylized dynamic image is generated based on the image sequence with animation style.

[0238] It should be noted that a pre-trained stylization model is used to extract image features, and style weights are then incorporated into the feature processing group.

[0239] For example, such as Figure 2 As shown, style features are extracted using a preset anime style and directly input into the transformer parts of B1, B2, B3, B4, C1, C2, C3 and C4 in the Unet generator network to achieve stylistic injection.

[0240] Understandably, by simply adjusting the parameters of the added time layer and input layer, it is possible to achieve integration with stylization, which is obviously a simpler approach.

[0241] In this embodiment of the disclosure, stylization processing can be incorporated to achieve stylized dynamic image generation from realistic images.

[0242] In some embodiments, obtaining a single-frame image and its image features includes:

[0243] A single-frame image is acquired, and the single-frame image is encoded and compressed to obtain the image features of the single-frame image.

[0244] Here, encoding compression refers to using a pre-trained encoder to extract features from a single frame of an image to obtain image features.

[0245] For example, a single-frame image is input into an encoder consisting of 8 residual convolutional layers. The encoder compresses and encodes the single-frame image to obtain image features. An 8-layer residual convolutional encoder means that the model has 8 residual blocks, each of which may contain multiple convolutional layers and activation functions to extract features at different levels. The 8-layer residual convolutional encoder compresses the high-resolution image into a low-dimensional space with an aspect ratio of 6:1. The minimum resolution of the long side of the single-frame image is 256, and the minimum resolution of the short side is 32. That is, after compression and encoding, the single-frame image yields a feature map of size 3x32x256, which represents the image features. Here, 3 represents the RGB channels of the image.

[0246] Understandably, high-resolution images contain a large number of pixels and thus a large amount of data, requiring more computational resources to process. Compression coding can significantly reduce image size, decrease computational load, and speed up processing. Furthermore, compression helps remove redundant information from images, allowing models to focus more on key features and improving their generalization ability and robustness.

[0247] It's important to note that INT8 quantization is performed during deployment. INT8 quantization is a numerical quantization technique that converts the weights and activation values ​​in the model from floating-point numbers (such as FP32) to integers (such as INT8), i.e., 8-bit integers. Since INT8 data occupies significantly less storage space and has a much faster processing speed than FP32, quantization can significantly reduce the model size and computational cost.

[0248] In this embodiment of the disclosure, by compressing the acquired image, the amount of computation can be reduced, the processing speed can be accelerated, and the generalization ability and robustness of the model can be improved.

[0249] In some embodiments, obtaining a single-frame image and its image features includes:

[0250] In response to the detection of a photo capture command, a single-frame image and its image features are acquired.

[0251] Based on image sequences, dynamic images are generated, including:

[0252] The image sequence is rendered and then displayed as a dynamic image.

[0253] Here, a single-frame image can be captured in real time by detecting the image acquisition module's photo capture command, allowing the user to obtain the single-frame image they want.

[0254] It should be noted that single-frame images can also be obtained through methods such as downloading from the internet or storing images in the terminal device's image library. This image acquisition module is suitable for cameras, video recording cameras, and terminal devices, which can include: mobile phones, tablets, laptops, wearable electronic devices, etc.

[0255] Here, dynamic images are generated by performing different rendering processes on image sequences to achieve different application scenarios.

[0256] For example, in advertising and marketing applications: creating realistic virtual product models and scenes in advertisements, or realistic virtual fitting rooms and scenes in marketing applications to help users shop and experience products, and used to create engaging advertising content, such as dynamically showcasing different features and functions of a product. In film and television production applications: users can use motion graphics generation to create and edit video content, such as adding special effects and transitions, or creating animations. In live wallpaper and theme applications: users can use motion graphics generation to personalize their devices, including live wallpapers, lock screens, and themes. In news and weather applications: motion graphics generation can be used to display dynamic visualizations of real-time weather changes, storm paths, or other important events. In virtual fitting room applications: through motion graphics generation, users can try on different clothes or accessories on their phones and see how they look in motion.

[0257] This embodiment of the disclosure achieves real-time rendering based on real-time acquired single-frame images, ensuring that the generated dynamic images can achieve different visual rendering effects in different application scenarios. The rendering effect can be adjusted according to specific needs and scenarios. Real-time rendering allows users to instantly see the effects of operations or environmental changes, enhancing interactivity and immersion, and improving the user experience.

[0258] In some embodiments, the method further includes:

[0259] The settings interface based on dynamic images retrieves the settings parameters associated with the dynamic images; these settings parameters include: parameters indicating the frame rate, parameters indicating the rendering method, and parameters indicating the image quality.

[0260] The process involves rendering an image sequence and generating a dynamic image for display, including:

[0261] The system renders each image frame in the image sequence according to the rendering method and image quality indicated by the settings parameters, and then combines the rendered image frames into a dynamic image for display according to the indicated frame rate.

[0262] Here, frame rate is the number of image frames displayed per second. Parameters indicating frame rate include: standard frame rate, high frame rate, variable frame rate, etc.; rendering is the process of converting a 3D model or scene into a 2D image. Parameters indicating rendering mode include real-time rendering, pre-rendering, etc.; image quality refers to the degree of quality of the image. Parameters indicating image quality include: low quality, medium quality, high quality, ultra-high quality, etc.

[0263] For example, such as Figure 4 As shown, users can input or select desired settings parameters through the settings interface. In the settings interface, users can find frame rate adjustment options, such as 24fps, 30fps, or 60fps. Users can also select the rendering method, such as real-time rendering or pre-rendering. Image quality options include low, medium, high, and ultra-high, which users can choose according to their needs. The user selects 24fps as the frame rate, real-time rendering as the rendering method, and ultra-high image quality. By processing each image frame of the image sequence according to the real-time rendering method to achieve ultra-high image quality, the processed image frames are composited at a frame rate of 24fps. During the compositing process, smooth transitions between each frame are ensured to maintain the smoothness of the generated dynamic image. The composited dynamic image is displayed on the screen at a frame rate of 24fps.

[0264] The embodiments of this disclosure obtain the setting parameters of the associated dynamic image, which can perform real-time rendering processing on the image sequence generated from a single frame image, ensuring that the generated dynamic image has a better visual rendering effect.

[0265] The following is about Figure 2 The complete description process mainly includes the following steps:

[0266] By inputting a single-frame image, an encoder extracts the image features from the single-frame image. Then, by adding noise features and mask features, these features are combined with the image features to obtain the first combined feature F. c Second combination feature F i and the third combination feature F m The first combination of features F c Second combination feature F i and the third combination feature F iThe first feature map is obtained by fusion diffusion processing after being input into the masked fusion operator. Then, it is input into the two-dimensional convolution (Conv2D) for convolution processing. The second feature map is then input into the Unet generator network (composed of the first and second parts A12, B1 is Time Transformer-Reverse, B2 is WH Transformer-Reverse, B3 is Time Transformer, B4 is WH Transformer, C1 is Time Transformer, C2 is WH Transformer, C3 is Time Transformer-Reverse, and C4 is WH Transformer-Reverse). It is combined with the image features obtained by the image encoder for spatiotemporal modeling processing to obtain the third feature map with the predetermined target diffusion number. The feature map is then combined with the feature map processed by compression regularization (Compress Dropout (0.3)) and stacked into each convolutional layer. The feature maps are then input into the decoder (Fu-Decoder) for decoding to generate a dynamic image.

[0267] During training, we used an internal dataset of 20 million images and 3,000 independent videos. Each video was approximately 1-6 seconds long. Multiple frames were extracted from the training videos and stitched together to increase the reference content during training. These frames were then input into the encoder for subsequent operations. A preset result was obtained from the training videos, and this preset result was compared with the original images. Training was considered complete when the difference between the preset result and the original images was small. Zero signal-to-noise ratio (SNR) was used as the loss function during training, with a default resolution of 2048 and 24 frames. Randomly sampling frames from 1-2 seconds of video data for model training significantly improved the stability of the model's dynamic image generation.

[0268] The following is Figure 3 A complete description process:

[0269] In step 301, the first combined feature F c Second combination feature F i and the third combination feature F m Perform channel normalization to give it zero mean and unit variance, resulting in a normalized feature map;

[0270] In step 302, the corresponding channels of the three normalized features are fused using ChannelFusion to increase the number of channels in the third step.

[0271] In step 303, the number of channels for the feature is increased by the expansion encoder, which may reduce the spatial resolution.

[0272] In step 304, the channel dimension is mapped to an integer multiple of the frame number through a single-layer linear layer;

[0273] In step 305, the interaction between different features is emphasized through the cross attention mechanism, highlighting important feature interactions;

[0274] In step 306, the features output by the linear layer are inverted and input into the cross attention, which allows the model to focus on the difference between the reconstructed features and the original features.

[0275] In step 307, the spatial relationships within the features are emphasized through a self-attention mechanism;

[0276] In step 308, the two feature inversions are input into the self-attention mechanism for feature reconstruction at different levels;

[0277] In step 309, the features are further processed using a feed-forward neural network;

[0278] In step 310, the two feature inversions are processed by a linear layer and then input into the feedforward neural network. The linear layer ensures that the feature dimensions after feature inversion are consistent with the requirements of the feedforward network.

[0279] In step 311, the number of feature channels is reduced by a compression decoder while increasing the spatial resolution to obtain features with high resolution.

[0280] Figure 5 This is a block diagram illustrating a control device according to an exemplary embodiment. Figure 5 As shown, the device mainly includes:

[0281] The acquisition unit 501 is configured to acquire a single-frame image and the image features of the single-frame image;

[0282] The first combination unit 502 is configured to combine image features with preset noise features to obtain a first combined feature;

[0283] The second combination unit 503 is configured to combine image features with preset mask features to obtain a second combination feature;

[0284] The third combination unit 504 is configured to combine noise features with mask features to obtain a third combined feature;

[0285] The diffusion unit 505 is configured to perform diffusion processing based on a first combined feature, a second combined feature, and a third combined feature using a preset image frame diffusion model to obtain an image sequence after diffusion of a single frame image; wherein, the image frame diffusion model is obtained based on training of a deep learning network;

[0286] The generation unit 506 is configured to generate dynamic images based on image sequences.

[0287] In some embodiments, the apparatus further includes:

[0288] The first feature processing layer of the image frame diffusion model is used to perform feature fusion diffusion processing on the first combined feature, the second combined feature, and the third combined feature to obtain a preset number of fused first feature maps; wherein, the preset number is an integer multiple of the predetermined target diffusion number, and the number of image frames included in the image sequence is the predetermined target diffusion number.

[0289] The image frame diffusion model uses convolutional layers to perform convolution processing on a preset number of first feature maps to obtain a preset number of second feature maps.

[0290] The second feature processing layer of the image frame diffusion model is used to perform spatiotemporal feature modeling on a preset number of second feature maps to obtain a third feature map with a predetermined target diffusion number.

[0291] Based on the third feature map of the predetermined target diffusion number, the image sequence after diffusion of a single frame image is obtained.

[0292] In some embodiments, the device 500 further includes:

[0293] The first feature processing layer performs the same normalization process on the first combined feature, the second combined feature, and the third combined feature to obtain the first normalized combined feature, the second normalized combined feature, and the third normalized combined feature; the feature points in the first normalized combined feature, the second normalized combined feature, and the third normalized combined feature are fused to obtain multiple fused feature maps.

[0294] For each fused feature map, feature dimension mapping is performed to obtain a preset number of fused first feature maps.

[0295] In some embodiments, the first combined feature includes multiple sets of first combined sub-features formed by combining image features with different noise features respectively; the second combined feature includes multiple sets of second combined sub-features formed by combining image features with different mask features respectively; the third combined feature includes multiple sets of third combined sub-features formed by combining each noise feature and each mask feature; the first normalized combined feature includes a first normalized combined sub-feature after normalizing each first set of sub-features; the second normalized combined feature includes a second normalized combined sub-feature after normalizing each second set of sub-features; and the third normalized combined feature includes a third normalized combined sub-feature after normalizing each third set of sub-features.

[0296] Based on the feature points from the first normalized combined feature, the second normalized combined feature, and the third normalized combined feature, multiple fused feature maps are obtained, including:

[0297] The feature points in each associated first normalized combined sub-feature, second normalized combined sub-feature, and third normalized combined sub-feature are fused to obtain the fused feature map corresponding to each associated combined sub-feature; wherein the associated first normalized combined sub-feature, second normalized combined sub-feature, and third normalized combined sub-feature include the same noise feature and mask feature.

[0298] In some embodiments, the device 500 further includes:

[0299] Determine the feature mean corresponding to each associated first normalized combinatorial sub-feature, second normalized combinatorial sub-feature, and third normalized combinatorial sub-feature, and determine the weighting coefficient corresponding to the feature mean;

[0300] Based on the feature values ​​at the same position in each associated first normalized combinatorial feature, second normalized combinatorial feature, and third normalized combinatorial feature, a weighted fusion is performed using each weighting coefficient to obtain the fused feature map corresponding to each associated combinatorial feature.

[0301] In some embodiments, among the multiple sets of first combined sub-features included in the first combined feature, the variation amplitude of the noise feature is negatively correlated with the predetermined target diffusion quantity.

[0302] In some embodiments, the device 500 further includes:

[0303] The second feature processing layer of the image frame diffusion model is used to perform spatiotemporal feature modeling on a predetermined number of second feature maps to obtain a third feature map with a predetermined target diffusion number, including:

[0304] For each feature processing group in the second feature processing layer, the first sub-feature processing layer in the feature processing group is used to construct the correlation between the input feature maps to obtain multiple time-modeling feature maps; wherein, the feature map input to the first sub-feature processing layer of the first feature processing group is a preset number of second feature maps; the feature map input to the first sub-feature processing layer of the feature processing group after the first one is the feature map output of the adjacent previous feature processing group and image features.

[0305] For each feature processing group in the second feature processing layer, the second sub-feature processing layer in the feature processing group is used to transpose each time modeling feature map and construct the correlation between feature points in the transposed time modeling feature map to obtain multiple spatial modeling feature maps. Based on the multiple spatial modeling feature maps output by the second sub-feature processing layer of the last feature processing group, a third feature map of the predetermined target diffusion number is obtained.

[0306] In some embodiments, the multiple connected feature processing groups are divided into a first part and a second part, the first part and the second part include the same number of feature processing groups, and the second part is connected after the first part; wherein, the feature processing groups in the first part are used to encode the input feature map in sequence; the feature processing groups in the second part are used to decode the input feature map in sequence.

[0307] In some embodiments, the device 500 further includes:

[0308] The feature map output by the last feature processing group in the first part is regularized to obtain the regularized feature map.

[0309] Based on the third feature map with a predetermined target diffusion amount, the image sequence after diffusion of a single frame image is obtained, including:

[0310] The third feature map based on the predetermined target diffusion number and the regularized feature map are used for decoding to obtain the image sequence after diffusion of a single frame image.

[0311] In some embodiments, the device 500 further includes:

[0312] Spatiotemporal feature modeling is performed on a preset number of second feature maps using multiple second feature processing layers to obtain a third feature map with a predetermined target diffusion number. The input of the first second feature processing layer is the preset number of second feature maps, and the input of the second feature processing layer after the first one is the output of the adjacent second feature processing layer. The third feature map with the predetermined target diffusion number is obtained based on the output of the last second feature processing layer.

[0313] In some embodiments, the device 500 further includes:

[0314] Obtain the style features corresponding to the preset style;

[0315] Based on the first combined feature, the second combined feature, and the third combined feature, a preset image frame diffusion model is used for diffusion processing to obtain the image sequence after diffusion of a single frame image, including:

[0316] Based on the first combination feature, the second combination feature, the third combination feature, and the style feature, a preset image frame diffusion model is used to perform diffusion processing to obtain an image sequence with a preset style after diffusion of a single frame image.

[0317] In some embodiments, the device 500 further includes:

[0318] A single-frame image is acquired, and the single-frame image is encoded and compressed to obtain the image features of the single-frame image.

[0319] In some embodiments, the device 500 further includes:

[0320] In response to the detection of a photo capture command, a single-frame image and its image features are acquired.

[0321] Based on image sequences, dynamic images are generated, including:

[0322] The image sequence is rendered and then displayed as a dynamic image.

[0323] In some embodiments, the device 500 further includes:

[0324] The settings interface based on dynamic images retrieves the settings parameters associated with the dynamic images; these settings parameters include: parameters indicating the frame rate, parameters indicating the rendering method, and parameters indicating the image quality.

[0325] The process involves rendering an image sequence and generating a dynamic image for display, including:

[0326] The system renders each image frame in the image sequence according to the rendering method and image quality indicated by the settings parameters, and then combines the rendered image frames into a dynamic image for display according to the indicated frame rate.

[0327] Figure 6 This is a structural block diagram of a device 600 according to an exemplary embodiment. Figure 1 For example, device 600 can be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0328] Reference Figure 6The device 600 may include one or more of the following components: processing component 602, memory 604, power supply component 606, multimedia component 608, audio component 610, input / output (I / O) interface 612, sensor component 614, and communication component 616.

[0329] Processing component 602 typically controls the overall operation of device 600, such as operations associated with at least one of display, telephone call, data communication, camera operation, and recording operation. Processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0330] Memory 604 is configured to store various types of data to support operation on device 600. Examples of such data include at least one of the following: instructions for any application or method operating on device 600, contact data, phonebook data, messages, pictures, and videos. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0331] Power supply component 606 provides power to various components of device 600. Power supply component 606 may include at least one of the following: a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to device 600.

[0332] Multimedia component 608 includes a screen that provides an output interface between device 600 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When device 600 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0333] Audio component 610 is configured to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) configured to receive external audio signals when device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0334] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, and buttons. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0335] Sensor assembly 614 includes one or more sensors for providing state assessments of various aspects of device 600. For example, sensor assembly 614 may detect the on / off state of device 600, the relative positioning of components such as the display and keypad of device 600, changes in the position of device 600 or one of its components, the presence or absence of user contact with device 600, the orientation or acceleration / deceleration of device 600, and temperature changes of device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include, but is not limited to, at least one of the following: an accelerometer, a gyroscope, a magnetometer, a pressure sensor, and a temperature sensor.

[0336] Communication component 616 is configured to facilitate wired or wireless communication between device 600 and other devices. Device 600 can access wireless networks based on communication standards, such as Wi-Fi, 4G, 5G, or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth (BT), and other technologies.

[0337] In an exemplary embodiment, device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components.

[0338] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including executable instructions or a computer program, which can be executed by the processor 620 of the device 600 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0339] A non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by the processor of a mobile terminal, enables the mobile terminal to perform any of the dynamic image generation methods described in the embodiments of this disclosure. For example, the method includes:

[0340] A single-frame image and its image features are acquired; the image features are combined with preset noise features to obtain a first combined feature; the image features are combined with preset mask features to obtain a second combined feature; the noise features are combined with mask features to obtain a third combined feature; based on the first combined feature, the second combined feature, and the third combined feature, a preset image frame diffusion model is used for diffusion processing to obtain an image sequence after diffusion of the single-frame image; wherein, the image frame diffusion model is obtained based on deep learning network training; and a dynamic image is generated based on the image sequence.

[0341] This disclosure provides a computer program product comprising a computer program or executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer program or executable instructions from the computer-readable storage medium and executes the computer program or executable instructions, causing the computer device to perform any of the dynamic image generation methods described above in this disclosure.

[0342] Figure 7 This is a block diagram of an apparatus 700 for generating moving images according to an exemplary embodiment. Figure 2 For example, device 700 can be provided as a server. (See reference...) Figure 7 The apparatus 700 includes a processing component 722, which further includes one or more processors, and memory resources represented by memory 732 for storing instructions, such as application programs, that can be executed by the processing component 722. The application programs stored in memory 732 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 722 is configured to execute instructions to perform any of the aforementioned motion image generation methods. For example, the method includes:

[0343] A single-frame image and its image features are acquired; the image features are combined with preset noise features to obtain a first combined feature; the image features are combined with preset mask features to obtain a second combined feature; the noise features are combined with mask features to obtain a third combined feature; based on the first combined feature, the second combined feature, and the third combined feature, a preset image frame diffusion model is used for diffusion processing to obtain an image sequence after diffusion of the single-frame image; wherein, the image frame diffusion model is obtained based on deep learning network training; and a dynamic image is generated based on the image sequence.

[0344] Device 700 may also include a power supply component 726 configured to perform power management of device 700, a wired or wireless network interface 750 configured to connect device 700 to a network, and an input / output (I / O) interface 758. Device 700 can operate an operating system stored in memory 732, such as Windows Server™, Mac OS X™, Unix™ Linux™, FreeBSD™, or similar.

[0345] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0346] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for generating dynamic images, characterized in that, include: Acquire a single-frame image and the image features of the single-frame image; The image features are combined with preset noise features to obtain the first combined features; The image features are combined with preset mask features to obtain a second combined feature; The noise feature is combined with the mask feature to obtain a third combined feature; Based on the first combined feature, the second combined feature, and the third combined feature, a preset image frame diffusion model is used to perform diffusion processing to obtain the image sequence after diffusion of the single frame image; wherein, the image frame diffusion model is obtained based on deep learning network training; Based on the image sequence, a dynamic image is generated.

2. The method according to claim 1, characterized in that, The process of obtaining the image sequence after diffusion of the single-frame image using a preset image frame diffusion model based on the first combined feature, the second combined feature, and the third combined feature includes: The first feature processing layer of the image frame diffusion model is used to perform feature fusion diffusion processing on the first combined feature, the second combined feature, and the third combined feature to obtain a preset number of fused first feature maps; wherein, the preset number is an integer multiple of the predetermined target diffusion number, and the number of image frames included in the image sequence is the predetermined target diffusion number; The image frame diffusion model is used to perform convolution processing on the preset number of first feature maps to obtain the preset number of second feature maps; The second feature processing layer of the image frame diffusion model is used to perform spatiotemporal feature modeling on the preset number of second feature maps to obtain the third feature map of the predetermined target diffusion number. Based on the third feature map of the predetermined target diffusion quantity, the image sequence after diffusion of the single frame image is obtained.

3. The method according to claim 2, characterized in that, The first feature processing layer of the image frame diffusion model performs feature fusion diffusion processing on the first combined feature, the second combined feature, and the third combined feature to obtain a preset number of fused first feature maps, including: The first feature processing layer is used to perform the same normalization process on the first combined feature, the second combined feature, and the third combined feature to obtain the first normalized combined feature, the second normalized combined feature, and the third normalized combined feature; the feature points in the first normalized combined feature, the second normalized combined feature, and the third normalized combined feature are fused to obtain multiple fused feature maps. For each fused feature map, feature dimension mapping is performed to obtain the preset number of fused first feature maps.

4. The method according to claim 3, characterized in that, The first combined feature includes multiple sets of first combined sub-features formed by combining the image features with different noise features respectively; the second combined feature includes multiple sets of second combined sub-features formed by combining the image features with different mask features respectively; the third combined feature includes multiple sets of third combined sub-features formed by combining each noise feature and each mask feature; the first normalized combined feature includes a first normalized combined sub-feature after normalizing each first set of sub-features; the second normalized combined feature includes a second normalized combined sub-feature after normalizing each second set of sub-features; the third normalized combined feature includes a third normalized combined sub-feature after normalizing each third set of sub-features. The feature points from the first normalized combined feature, the second normalized combined feature, and the third normalized combined feature are fused to obtain multiple fused feature maps, including: The feature points in each associated first normalized combined sub-feature, second normalized combined sub-feature, and third normalized combined sub-feature of the first normalized combined feature, the second normalized combined feature, and the third normalized combined feature are fused to obtain a fused feature map corresponding to each associated combined sub-feature; wherein the associated first normalized combined sub-feature, second normalized combined sub-feature, and third normalized combined sub-feature include the same noise feature and mask feature.

5. The method according to claim 4, characterized in that, The step of fusing feature points in each associated first normalized combined sub-feature, second normalized combined sub-feature, and third normalized combined sub-feature from the first normalized combined feature, the second normalized combined feature, and the third normalized combined feature to obtain a fused feature map corresponding to each associated combined sub-feature includes: Determine the feature mean corresponding to each associated first normalized combinatorial sub-feature, second normalized combinatorial sub-feature, and third normalized combinatorial sub-feature, and determine the weighting coefficient corresponding to the feature mean; Based on the feature values ​​at the same position in each associated first normalized combinatorial feature, second normalized combinatorial feature, and third normalized combinatorial feature, a weighted fusion is performed using each weighting coefficient to obtain the fused feature map corresponding to each associated combinatorial feature.

6. The method according to claim 4, characterized in that, Among the multiple sets of first combined sub-features included in the first combined feature, the variation amplitude of the noise feature is negatively correlated with the diffusion quantity of the predetermined target.

7. The method according to claim 2, characterized in that, The second feature processing layer includes multiple connected feature processing groups, each feature processing group consisting of a first sub-feature processing layer and a second sub-feature processing layer connected together; The second feature processing layer using the image frame diffusion model performs spatiotemporal feature modeling on the preset number of second feature maps to obtain the predetermined target number of third feature maps, including: For each feature processing group in the second feature processing layer, the first sub-feature processing layer in the feature processing group is used to construct the correlation between the input feature maps to obtain multiple time-modeling feature maps; wherein, the feature map input to the first sub-feature processing layer of the first feature processing group is the preset number of second feature maps; the feature map input to the first sub-feature processing layer of the feature processing group after the first one is the feature map output by the adjacent previous feature processing group and the image features. For each feature processing group in the second feature processing layer, the second sub-feature processing layer in the feature processing group is used to transpose each time modeling feature map and construct the correlation between feature points in the transposed time modeling feature map to obtain multiple spatial modeling feature maps. Based on the multiple spatial modeling feature maps output by the second sub-feature processing layer of the last feature processing group, a third feature map of the predetermined target diffusion number is obtained.

8. The method according to claim 7, characterized in that, The multiple connected feature processing groups are divided into a first part and a second part, and the first part and the second part include the same number of feature processing groups. The second part is connected after the first part. The feature processing groups in the first part are used to encode the input feature map in sequence. The feature processing groups in the second part are used to decode the input feature map in sequence.

9. The method according to claim 8, characterized in that, The method further includes: The feature map output by the last feature processing group in the first part is regularized to obtain the regularized feature map. The third feature map based on the predetermined target diffusion number is used to obtain the image sequence after diffusion of the single frame image, including: Based on the third feature map of the predetermined target diffusion number and the regularized feature map, decoding is performed to obtain the image sequence after diffusion of the single frame image.

10. The method according to claim 2, characterized in that, The image frame diffusion model includes multiple second feature processing layers; The second feature processing layer using the image frame diffusion model performs spatiotemporal feature modeling on the preset number of second feature maps to obtain the predetermined target number of third feature maps, including: The predetermined number of second feature maps are spatiotemporally modeled using the multiple second feature processing layers to obtain the predetermined number of third feature maps for diffusion. The input of the first second feature processing layer is the predetermined number of second feature maps, and the input of each subsequent second feature processing layer is the output of the adjacent second feature processing layer. The predetermined number of third feature maps for diffusion is obtained based on the output of the last second feature processing layer.

11. The method according to any one of claims 1-10, characterized in that, The method further includes: Obtain the style features corresponding to the preset style; The process of performing diffusion processing based on the first combined feature, the second combined feature, and the third combined feature using a preset image frame diffusion model to obtain the image sequence after diffusion of the single frame image includes: Based on the first combined feature, the second combined feature, the third combined feature, and the style feature, a preset image frame diffusion model is used to perform diffusion processing to obtain an image sequence with the preset style after diffusion of the single frame image.

12. The method according to claim 1, characterized in that, The acquisition of a single-frame image and its image features includes: The single-frame image is acquired, and the single-frame image is encoded and compressed to obtain the image features of the single-frame image.

13. The method according to claim 1, characterized in that, The acquisition of a single-frame image and its image features includes: In response to the detection of a photo capture command, a single-frame image and the image features of the single-frame image are acquired; The step of generating a dynamic image based on the image sequence includes: The image sequence is rendered and the resulting dynamic image is then displayed.

14. The method according to claim 13, characterized in that, The method further includes: Based on the settings interface of the dynamic image, the settings parameters associated with the dynamic image are obtained; wherein, the settings parameters include: parameters indicating the frame rate, parameters indicating the rendering method, and parameters indicating the image quality. The process of rendering the image sequence and generating the dynamic image for display includes: The rendering process is performed on each image frame in the image sequence based on the rendering method and image quality indicated by the setting parameters, and the rendered image frames are combined into the dynamic image and displayed according to the indicated frame rate.

15. A dynamic image generation device, characterized in that, include: The acquisition unit is configured to acquire a single-frame image and the image features of the single-frame image; The first combining unit is configured to combine the image features with preset noise features to obtain a first combined feature; The second combining unit is configured to combine the image features with preset mask features to obtain a second combined feature; The third combination unit is configured to combine the noise feature with the mask feature to obtain a third combined feature; The diffusion unit is configured to perform diffusion processing based on the first combined feature, the second combined feature, and the third combined feature using a preset image frame diffusion model to obtain the image sequence after diffusion of the single frame image; wherein, the image frame diffusion model is obtained based on deep learning network training; The generation unit is configured to generate dynamic images based on the image sequence.

16. An electronic device, characterized in that, include: processor; Memory used to store computer programs or instructions; The processor executes the computer program or instructions to implement the steps of the method according to any one of claims 1 to 14.

17. A non-transitory computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer program or instructions in the storage medium are executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.

18. A program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 14.

Images