Training of dynamic image generation model, dynamic image display method, and electronic device

Abstract

This application discloses a training method for a dynamic image generation model, a dynamic image display method, an electronic device, and a storage medium. The training method includes: determining sample input information based on sample prompts; extracting features from the sample input information using a feature extraction module to obtain sample image features corresponding to the predicted dynamic image; performing semantic interaction on the features corresponding to different predicted sample images in the sample image features through a spatiotemporal interaction layer to obtain semantic interaction features; determining the target sample features corresponding to the predicted dynamic image based on the sample image features and the semantic interaction features through the spatiotemporal interaction layer; and training the dynamic image generation model based on the target sample features. In this application, the accuracy of the dynamic images generated by the trained dynamic image generation model is high.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a training method for a dynamic image generation model, a dynamic image display method, an electronic device, and a storage medium. Background Technology

[0002] Currently, animated images can be generated based on guide images or guide text using trained animated image generation models. However, the accuracy of the generated animated images using existing methods is low. Summary of the Invention

[0003] In view of this, embodiments of this application propose a training method for a dynamic image generation model, a dynamic image display method, an electronic device, and a storage medium.

[0004] In a first aspect, embodiments of this application provide a training method for a dynamic image generation model. The dynamic image generation model includes a feature extraction block and a spatiotemporal interaction layer. The method includes: determining sample input information based on sample prompt information; the sample prompt information is used to indicate the content included in the predicted sample dynamic image to be generated; the predicted sample dynamic image includes multiple predicted sample images; performing feature extraction on the sample input information through a feature extraction module to obtain sample image features corresponding to the predicted sample dynamic image; the sample image features include the features of each of the multiple predicted sample images; performing semantic interaction on the features corresponding to different predicted sample images in the sample image features through the spatiotemporal interaction layer to obtain semantic interaction features; determining the target sample features corresponding to the predicted sample dynamic image through the spatiotemporal interaction layer based on the sample image features and the semantic interaction features; and training the dynamic image generation model based on the target sample features.

[0005] Secondly, embodiments of this application provide a dynamic image display method, the method comprising: displaying a target dynamic image in response to obtaining target text; the target dynamic image is generated based on target prompt information by a dynamic image generation model, the target prompt information is determined based on the target text, and the dynamic image generation model is trained according to the aforementioned method.

[0006] Thirdly, embodiments of this application provide a training apparatus for a dynamic image generation model. The dynamic image generation model includes a feature extraction block and a spatiotemporal interaction layer. The apparatus includes: an information determination module, used to determine sample input information based on sample prompt information; the sample prompt information is used to indicate the content included in the predicted sample dynamic image to be generated; the predicted sample dynamic image includes multiple predicted sample images; a feature extraction module, used to extract features from the sample input information through the feature extraction module to obtain sample image features corresponding to the predicted sample dynamic image; the sample image features include the features of each of the multiple predicted sample images; a semantic interaction module, used to perform semantic interaction on the features corresponding to different predicted sample images in the sample image features through the spatiotemporal interaction layer to obtain semantic interaction features; a feature determination module, used to determine the target sample features corresponding to the predicted sample dynamic image through the spatiotemporal interaction layer based on the sample image features and the semantic interaction features; and a training module, used to train the dynamic image generation model based on the target sample features.

[0007] Optionally, the feature determination module is also used to maintain the detailed features through the sample image features to obtain the detailed maintenance features; and to perform feature fusion of the detailed maintenance features and the semantic interaction features through the spatiotemporal interaction layer to obtain the target sample features.

[0008] Optionally, the feature determination module is also used to perform convolution processing on the sample image features through the spatiotemporal interaction layer to obtain detail maintenance features.

[0009] Optionally, the feature determination module is also used to determine the weight of the detailed maintenance features based on the detailed maintenance features through the spatiotemporal interaction layer; determine the weight of the semantic interaction features based on the semantic interaction features through the spatiotemporal interaction layer; and perform weighted fusion of the detailed maintenance features and the semantic interaction features based on the weight of the detailed maintenance features and the weight of the semantic interaction features to obtain the target sample features.

[0010] Optionally, the semantic interaction module is also used to downsample the sample image features through the spatiotemporal interaction layer to obtain downsampled image features; to perform self-attention operations on the downsampled image features through the spatiotemporal interaction layer to obtain attention-interactive image features; and to upsample the attention-interactive image features through the spatiotemporal interaction layer to obtain semantic interaction features.

[0011] Optionally, the feature extraction module is further configured to determine the sample image features corresponding to the t-th time step based on the input and time step features of the t-th time step; t∈[1,T], where t is an integer and T is a positive integer; the input of the 1st time step is the sample input information; correspondingly, the semantic interaction module is further configured to perform semantic interaction on the features corresponding to different predicted sample images in the sample image features corresponding to the t-th time step through the spatiotemporal interaction layer to obtain the semantic interaction features corresponding to the t-th time step; correspondingly, the feature determination module is further configured to determine the output of the t-th time step based on the sample image features and semantic interaction features corresponding to the t-th time step through the spatiotemporal interaction layer; if t is less than T, the output of the t-th time step is obtained as the input of the (t+1)-th time step, and t is incremented by 1, and the step of determining the sample image features corresponding to the t-th time step based on the input and time step features of the t-th time step is returned to be executed, until t=T, and the output of the T-th time step is obtained as the target sample feature.

[0012] Optionally, the sample prompt information includes a sample dynamic image and a corresponding sample image description text, wherein the sample dynamic image includes multiple sample images; the feature extraction module is further used to obtain at least one of the first cluster center and the sample image description text as sample supplementary information; wherein, the first cluster center is the cluster center masked by a preset mask among multiple cluster centers; multiple cluster centers are obtained by clustering multiple sample images; the feature extraction module performs feature extraction based on the sample input information and the sample supplementary information to obtain sample image features.

[0013] Optionally, the sample prompt information includes a sample dynamic image, which includes multiple sample images; the information determination module is also used to cluster the encoding results of the multiple sample images to obtain multiple cluster centers; and to determine the sample input information based on the multiple cluster centers and a preset noise distribution.

[0014] Optionally, the information determination module is further configured to mask multiple cluster centers using a preset mask to obtain a first cluster center masked by the preset mask and a second cluster center not masked by the preset mask; determine the random sample noise corresponding to the first cluster center from a preset noise distribution; and determine the sample input information based on the random sample noise corresponding to the first cluster center and the second cluster center.

[0015] Optionally, the training module is further configured to determine the features corresponding to the first cluster center from the features of the target samples, as the predicted sample noise corresponding to the first cluster center; determine the model loss based on the difference between the random sample noise corresponding to the first cluster center and the predicted sample noise; and adjust the parameters of the dynamic image generation model through the model loss.

[0016] Optionally, the training process of the dynamic image generation model includes multiple iterative processes; the training module is further used to determine the model loss corresponding to the t-th iteration based on the target sample features obtained in the t-th iteration, where t is a natural number greater than 1; determine the difference between the model loss corresponding to the t-th iteration and the historical cumulative model loss as the loss difference of the t-th iteration; the historical cumulative model loss is the cumulative result of the model loss of the t-1 iterations before the t-th iteration; determine the joint perplexity of the t+1 iteration based on the loss difference of the t-th iteration; obtain the target difficulty level corresponding to the joint perplexity of the t+1 iteration; obtain the mask corresponding to the target difficulty level as the preset mask of the t+1 iteration.

[0017] Optionally, the training module is also used to determine the rate of change of loss difference and the cumulative loss difference in the p-th iteration based on the loss difference in the p-th iteration; and to perform a weighted summation of the loss difference, the rate of change of loss difference and the cumulative loss difference in the p-th iteration to obtain the joint perplexity in the p+1-th iteration.

[0018] Fourthly, embodiments of this application provide a dynamic image display device, the device comprising: a display module, configured to display a target dynamic image in response to acquiring target text; the target dynamic image is generated based on target prompt information by a dynamic image generation model, the target prompt information being determined based on target text, and the dynamic image generation model being trained according to the aforementioned method.

[0019] Fifthly, embodiments of this application provide an electronic device, including a processor and a memory; the memory stores computer-readable instructions, which, when executed by the processor, implement the above-described method.

[0020] Sixthly, embodiments of this application provide a computer-readable storage medium storing computer-readable instructions that, when executed by a processor, implement the above-described method.

[0021] In a seventh aspect, embodiments of this application provide a computer program product including computer instructions, which, when executed by a processor, implement the method described above.

[0022] This application provides a method for training a dynamic image generation model, a dynamic image display method, an electronic device, and a storage medium. In this application, the dynamic image generation model includes a feature extraction block for extracting features, as well as a temporal interaction layer. The temporal interaction layer performs semantic interaction on the features corresponding to different predicted sample images in the sample image features extracted by the feature extraction block to mine the contextual information between the features corresponding to different predicted sample images and obtain semantic interaction features. Then, the temporal interaction layer continues to determine the target sample features based on the semantic interaction features and the sample image features. Thus, the target sample features not only include the features corresponding to different predicted sample images, but also the contextual information between the features corresponding to different predicted sample images, making the information expressed by the target sample features more comprehensive and complete. In other words, by adding a spatiotemporal interaction layer to the dynamic image generation model, this application improves the information processing capability of the dynamic image generation model, thereby making the information processing capability of the dynamic image generation model stronger and the accuracy of the dynamic images generated by the trained dynamic image generation model higher. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic diagram illustrating the application scenarios to which the embodiments of this application are applicable is shown;

[0025] Figure 2 A flowchart illustrating a training method for a dynamic image generation model according to an embodiment of this application is shown;

[0026] Figure 3 A schematic diagram illustrating a clustering process for a sample image in an embodiment of this application is shown;

[0027] Figure 4 A schematic diagram illustrating a data processing procedure for a spatiotemporal interaction layer in an embodiment of this application is shown;

[0028] Figure 5 A schematic diagram illustrating the process of determining model loss in an embodiment of this application is shown;

[0029] Figure 6 It shows Figure 2 A flowchart of step S120 in one embodiment is shown in the corresponding example;

[0030] Figure 7 It shows Figure 2A flowchart of the steps following step S130 in one embodiment is shown in the corresponding example.

[0031] Figure 8 This paper illustrates the relationship between the amount of mask data and the mask route for different prediction tasks in an embodiment of this application.

[0032] Figure 9 A flowchart of a dynamic image display method according to an embodiment of this application is shown;

[0033] Figure 10 A schematic diagram of a chat interface according to an embodiment of this application is shown;

[0034] Figure 11 A schematic diagram illustrating the training process of a dynamic image generation model according to an embodiment of this application is shown;

[0035] Figure 12 A block diagram of a training apparatus for a dynamic image generation model according to an embodiment of this application is shown;

[0036] Figure 13 A block diagram of a dynamic image display device according to an embodiment of this application is shown;

[0037] Figure 14 A structural block diagram of an electronic device for performing a training method for a dynamic image generation model and a dynamic image display method according to embodiments of this application is shown. Detailed Implementation

[0038] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0039] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0041] It should be noted that "multiple" in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0042] The abbreviations and key terms used in this application will be explained below:

[0043] Curriculum learning: Curriculum learning is a deep learning training strategy that gradually increases the difficulty of training samples in a specific order. In this method, the model first learns simple tasks or samples, and then gradually transitions to more complex tasks to promote a more efficient learning process.

[0044] Masking: Masking is used to block out or select specific portions of data, ensuring that the model does not process invalid or irrelevant input during training or inference. A mask can be a binary matrix, where 1 represents valid data and 0 represents invalid data, and it is frequently used in attention mechanisms. Typical applications include processing variable-length sequences, autoregressive tasks in generative models, and handling padded data. In this application, masks are used to mask unwanted images or cluster centers.

[0045] This application discloses a method for training a dynamic image generation model, a method for displaying dynamic images, an electronic device, and a storage medium, which relates to artificial intelligence technology.

[0046] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0047] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0048] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and pre-trained learning. Pre-trained models represent the latest development in deep learning, integrating all of these techniques.

[0049] like Figure 1 As shown, the application scenarios applicable to this application embodiment include terminal 20 and server 10, which are connected via wired or wireless networks. Terminal 20 can be a smartphone, tablet, laptop, desktop computer, smart home appliance, vehicle terminal, aircraft, wearable device terminal, virtual reality device, medical device, or other terminal device capable of displaying web pages.

[0050] Server 10 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0051] The server 10 can determine the sample input information based on the sample prompt information, and process the sample input information by training the feature extraction block and spatiotemporal interaction layer in the dynamic image generation model to obtain the target sample features. Then, the dynamic image generation model is trained by the target sample features and the trained dynamic image generation model is stored.

[0052] In some implementations, terminal 20 can acquire target text and send it to server 10. Server 10 then generates a target dynamic image based on the target text using a dynamic image generation model and returns the target dynamic image to terminal 20, which then displays the target dynamic image.

[0053] In some other implementations, after the server 10 obtains the trained dynamic image generation model, it can also send the trained dynamic image generation model to the terminal 20, which stores the trained dynamic image generation model. When the terminal 20 obtains the target text, the terminal 20 uses its stored trained dynamic image generation model to generate a dynamic image based on the target text, obtains the target dynamic image, and displays the target dynamic image.

[0054] It is understood that when terminal 20 executes the method of this application, server 10 can store the dynamic image generation model in a distributed cloud storage system. After terminal 20 obtains the dynamic image generation model from the distributed cloud storage system, it deploys the dynamic image generation model in the local storage space of terminal 20 so that terminal 20 can perform sampling and reconstruction processing based on the target diffusion model deployed in the local storage space.

[0055] For ease of explanation, the following embodiments will be described using examples of the training method for the dynamic image generation model and the dynamic image display method being executed by an electronic device.

[0056] Please see Figure 2 , Figure 2 This application presents a flowchart illustrating a training method for a dynamic image generation model according to an embodiment of the present application. This method can be applied to electronic devices, which may be... Figure 1 The method includes: (The method is described in the context of) terminal 20 or server 10.

[0057] S110. Determine the sample input information based on the sample prompt information.

[0058] The sample prompt information is used to indicate the content included in the predicted sample dynamic image to be generated; the predicted sample dynamic image includes multiple predicted sample images, wherein the predicted sample dynamic image to be generated refers to the dynamic image generated by the dynamic image generation model based on the input information.

[0059] In this application, the sample prompt information may include text and images to indicate the content included in the predicted sample dynamic image through text and / or images. The dynamic image can refer to an animated image, which may be an animated emoticon (also called an animated emoji), an animated sticker, a game animation, or a video composed of multiple frames, etc.

[0060] In some embodiments, the sample prompt information may include a sample animated image to guide the content of the generated animated image. For example, the sample animated image may be an animated emoticon indicating happiness, and the content of the generated animated image under the guidance of the sample animated image may be "happy".

[0061] In some other embodiments, the sample prompt information may also include descriptive text of the sample dynamic image (also called sample image description text), which more directly guides the content of the generated dynamic image: the sample image description text serves as the content of the generated dynamic image. For example, if the sample image description text is "little bear", then the content of the generated dynamic image will be "little bear".

[0062] The dynamic image generation model in this application includes a feature extraction block and a spatiotemporal interaction (STI) layer. As mentioned earlier, the predicted sample dynamic image comprises multiple predicted sample images. That is, the dynamic image generation model obtains multiple predicted sample images based on sample cue information, and then synthesizes these multiple predicted sample images into a dynamic image. The synthesized dynamic image is the predicted sample dynamic image. Since the predicted sample image is actually in the form of a discrete frame, the dynamic image generation model in this application can also be called a Discrete Frame Generation Network (DFGN).

[0063] In this application, when the sample prompt information includes both text and images, the text and / or images included in the sample prompt information can be obtained as sample input information. However, when the text and / or images included in the sample prompt information are used as sample input information, the text and / or images cannot be processed by the dynamic image generation model and need to be encoded to obtain the encoded result as input information.

[0064] For images, encoding can be achieved using visual encoders in the Clip model (Contrastive Language-Image Pre-Training, a multimodal pre-trained neural network), BERT models, and VAE encoders (Variational Autoencoders). For text, encoding can be achieved using text encoders in the Clip model and text encoders in the T5 model (Text-to-Text Transfer Transformer). It is important to understand that the models used for image and text encoding described here are merely examples and do not constitute a limitation of this application. Other models can be selected to encode text and images based on actual needs.

[0065] The input information can be the encoded text in the sample prompt information; the input information can be the encoded image in the sample prompt information; or the input information can be the fusion of the encoded text and the encoded image in the sample prompt information, and the fusion result can be used as the input information. Here, fusion can include summation, weighted summation, and concatenation.

[0066] In some implementations, the sample prompt information includes a sample dynamic image, which includes multiple sample images; accordingly, S110 may include: clustering the encoding results of the multiple sample images to obtain multiple cluster centers; and determining the sample input information based on the multiple cluster centers and a preset noise distribution.

[0067] In a sample dynamic image, multiple sample images are arranged sequentially. A clustering algorithm is used to divide the encoding results of these images into multiple clusters. Each cluster contains the encoding result of at least one sample image, and the center of each cluster is a cluster center. Algorithms such as K-MEANS, K-MEDOIDS, and CLARANS can be used to cluster these multiple sample images.

[0068] In some implementations, the sequence numbers of the sample images to which the encoding results belong in each cluster obtained by the clustering algorithm are consecutive in the sample dynamic image. Since multiple sample images in the sample dynamic image are arranged sequentially, the multiple cluster centers obtained after clustering are also ordered. For example, if the sample dynamic image includes 16 sample images, clustering the encoding results of these 16 sample images yields four clusters: the encoding results of sample images 1-5 form cluster b1, the encoding results of sample images 6-8 form cluster b2, the encoding results of sample images 9-11 form cluster b3, and the encoding results of sample images 12-16 form cluster b4. In this case, the four cluster centers corresponding to the four clusters are also ordered: the first cluster center is the cluster center corresponding to cluster b1, the second cluster center is the cluster center corresponding to cluster b2, the third cluster center is the cluster center corresponding to cluster b3, and the fourth cluster center is the cluster center corresponding to cluster b4.

[0069] In this application, the encoding results of multiple sample images can be clustered to obtain multiple cluster centers. Multiple cluster centers are used to indicate multiple sample images in a sample dynamic image, thereby indicating the entire sample dynamic image with less data. This reduces the amount of data while increasing the information density. On the basis of improving the training efficiency of the dynamic image generation model, it also improves the dynamic image generation effect of the trained dynamic image generation model.

[0070] After obtaining multiple cluster centers, at least one random sample noise can be randomly sampled from a preset noise distribution. Based on the multiple cluster centers and the sampled random sample noise, the sample input information can be determined. The preset noise distribution can be a Gaussian noise distribution.

[0071] In some implementations, the number of prediction sample images included in the dynamic image of the prediction sample to be generated may be used as a reference number, multiple random sample noises may be randomly sampled from a preset noise distribution, a reference cluster center may be selected from multiple clusters and a reference random sample noise may be selected from multiple random sample noises, and the sample input information may be determined based on the reference cluster center and the reference random sample noise, wherein the sum of the number of reference cluster centers and the number of reference random sample noises is the reference number.

[0072] Generally, the dynamic image generation model generates dynamic images with the same number of images, which is the aforementioned reference number. For example, the dynamic image generation model generates predicted sample dynamic images a1 and a2 based on different sample prompts, each containing 16 predicted sample images.

[0073] The reference cluster center can be fused with the reference random sample noise to obtain the sample input information. Here, the fusion can be splicing. Thus, the obtained sample input information can include the input data of each prediction sample image in the prediction sample dynamic image: the input data of one prediction sample image in one part of the prediction sample image is a reference cluster center, and the input data of one prediction sample image in another part of the prediction sample image is a reference random sample noise.

[0074] In some other implementations, multiple cluster centers can be masked by a preset mask to obtain a first cluster center masked by the preset mask and a second cluster center not masked by the preset mask; random sample noise corresponding to the first cluster center can be determined from a preset noise distribution; and sample input information can be determined based on the random sample noise corresponding to the first cluster center and the second cluster center.

[0075] The preset mask can be set based on requirements and is used to mask at least one cluster center among multiple cluster centers. The cluster center masked by the preset mask is designated as the first cluster center, and the cluster center not masked by the preset mask is designated as the second cluster center.

[0076] As mentioned earlier, the multiple cluster centers are ordered. Therefore, based on the arrangement order of the first and second cluster centers among the multiple cluster centers, the random sample noise corresponding to the first cluster center and the second cluster center are concatenated into a vector as the sample input information. For example, if the cluster centers include c1, c2, c3, c4, and c5, and cluster centers c2 and c4 are masked by a preset mask, then the obtained sample input information will be [random sample noise corresponding to c1 and c2, random sample noise corresponding to c3 and c4, c5].

[0077] For example, Figure 3 As shown, the sample dynamic image includes 12 sample images. The encoding results of each sample image in the sample dynamic image are clustered to obtain 3 clusters: cluster 301, cluster 302 and cluster 303. Cluster 301 is the cluster after clustering the encoding results of the 1st to 3rd sample images, cluster 302 is the cluster after clustering the encoding results of the 4th to 7th sample images, and cluster 303 is the cluster after clustering the encoding results of the 8th to 12th sample images. Thus, the cluster center of cluster 301 (that is, a cluster center) indicates the 1st to 3rd sample images, the cluster center of cluster 302 indicates the 4th to 7th sample images, and the cluster center of cluster 303 indicates the 8th to 12th sample images.

[0078] However, the size of the obtained cluster centers is inconsistent with the size of the random sample noise sampled in the preset noise distribution. When determining the sample input information based on the random sample noise corresponding to the first cluster center and the second cluster center, the size of the random sample noise corresponding to the first cluster center and the second cluster center can be adjusted to be consistent. Then, the random sample noise corresponding to the first cluster center and the second cluster center can be concatenated into a vector to obtain the sample input information.

[0079] S120. The feature extraction module extracts features from the sample input information to obtain the sample image features corresponding to the predicted sample dynamic image. The spatiotemporal interaction layer performs semantic interaction on the features corresponding to different predicted sample images in the sample image features to obtain semantic interaction features. Based on the sample image features and semantic interaction features, the spatiotemporal interaction layer determines the target sample features corresponding to the predicted sample dynamic image.

[0080] The predicted sample dynamic image includes multiple predicted sample images, and the sample image features include the features of each of the multiple predicted sample images. For example, the sample image features can be represented as h1∈R F×H×W×d Where F is the number of multiple prediction sample images, H is the height of the prediction sample image, W is the width of the prediction sample image, and d is the number of channels (generally, the prediction sample image is a color image, that is, d takes the value 3, representing the three channels R (red), G (green), and B (blue).

[0081] The feature extraction module in a dynamic image generation model can be a residual network, a convolutional neural network, a long short-term memory neural network, or a deep convolutional network. The feature extraction module first extracts features from the sample input information to obtain the sample image features.

[0082] In some implementations, the aforementioned feature extraction module extracts features from the sample input information to obtain sample image features corresponding to multiple predicted sample images to be generated, including: obtaining at least one of a first cluster center and sample image description text as sample supplementary information; wherein, the first cluster center is a cluster center masked by a preset mask among multiple cluster centers; multiple cluster centers are obtained by clustering multiple sample images; and the feature extraction module extracts features based on the sample input information and sample supplementary information to obtain sample image features.

[0083] In this application, features in the sample image description text can be extracted using the text encoder in CLIP or T5, and the extracted features are mapped to the same space through a linear layer, with the mapped result serving as additional text information. Alternatively, noise can be added to the sample image description text by replacing characters in the sample image description text with random symbols "#", thereby generalizing the sample image description text and obtaining generalized sample image description text. Then, features in the generalized sample image description text can be extracted using the text encoder in CLIP or T5, and the extracted features are mapped to the same space through a linear layer, with the mapped result serving as additional text information.

[0084] Simultaneously, the first cluster center can be obtained as image supplementary information, and at least one of the text supplementary information and image supplementary information can be obtained as the aforementioned sample supplementary information. Then, the feature extraction module continues to perform feature extraction based on the sample input information and sample supplementary information to obtain sample image features.

[0085] It is worth mentioning that when the preset mask masks all the cluster centers after clustering multiple predicted sample images (the sample input information only includes random noise), the sample additional information includes at least text additional information, so that the dynamic image generation model can generate predicted sample dynamic images based on the features included in the sample image description text in the sample prompt information.

[0086] Similarly, when all cluster centers of multiple predicted sample images are clustered without a preset mask (the sample input information includes random noise and at least one cluster center), the sample image features can be obtained directly by extracting features from the sample input information without obtaining additional sample information. Alternatively, at least one of the additional text information and additional image information can be obtained as additional sample information, and the sample image features can be obtained by extracting features based on the additional sample information and the sample input information.

[0087] After obtaining the sample image features, the self-attention network in the spatiotemporal interaction layer can be used to perform self-attention mechanism operations on the sample image features to perform semantic interaction on the features corresponding to different predicted sample images in the sample image features, and obtain semantic interaction features.

[0088] In some implementations, the sample image features can be downsampled through a spatiotemporal interaction layer to obtain downsampled image features; self-attention operations can be performed on the downsampled image features through a spatiotemporal interaction layer to obtain attention interaction image features; and the attention interaction image features can be upsampled through a spatiotemporal interaction layer to obtain semantic interaction features.

[0089] That is, firstly, for the sample image features h1∈R F×H×W×d Downsampling (e.g., downsampling by a factor of γ) is performed to reduce semantic interaction complexity, resulting in downsampled image features. However, the downsampled image features h2 cannot be directly used for self-attention operations. Therefore, further processing of the downsampled image features is required. Expanded along the time and space dimensions to form expanded sample features Next, expand the sample features. The self-attention mechanism is performed. Finally, upsampling is performed using the same magnification factor as downsampling (downsampling by γ times, upsampling is also by γ times). The upsampled result is then restored along the time and space dimensions through tensor transformation to obtain features with the same size as the sample image features, which serve as semantic interaction features.

[0090] In order to ensure the smooth training process of the dynamic image generation model, it is necessary to restore the attention interaction image features to features with the same size as the sample image features. Therefore, the aforementioned upsampling factor and downsampling factor need to be [missing information].

[0091] As mentioned earlier, downsampling is performed before self-attention operations to reduce the amount of data, decrease the complexity of semantic interactions, and improve data processing efficiency, thereby increasing the training efficiency of the dynamic image generation model. Simultaneously, after the self-attention operation, upsampling is used to restore the feature size, ensuring that the feature size of the attention interaction image is consistent with that of the sample image, thus guaranteeing feature size consistency and ensuring the smooth training process of the dynamic image generation model.

[0092] After obtaining the semantic interaction features, the spatiotemporal interaction layer can perform fusion processing based on the sample image features and the semantic interaction features. The fusion result is used as the target sample feature corresponding to the predicted sample dynamic image. The fusion processing here can include summation, weighted summation, and concatenation.

[0093] In some implementations, the aforementioned determination of the target sample features corresponding to the predicted sample dynamic image based on sample image features and semantic interaction features through the spatiotemporal interaction layer includes: maintaining detailed features through sample image features to obtain detailed maintenance features; and fusing the detailed maintenance features and semantic interaction features through the spatiotemporal interaction layer to obtain the target sample features.

[0094] In this application, the sample image features are convolved through a spatiotemporal interaction layer to obtain detail-preserving features. For example, a three-dimensional convolutional network (e.g., a kernel size of k×1×1, where k can be set based on requirements, e.g., k=3) in the spatiotemporal interaction layer can be used to convolve the sample image features to reduce the attenuation of fine-grained features and preserve the detail features of the sample image. Then, the spatiotemporal interaction layer fuses the detail-preserving features and semantic interaction features to obtain the target sample features. This feature fusion can include weighted summation, summation, and concatenation.

[0095] Optionally, the aforementioned feature fusion of detail maintenance features and semantic interaction features through a spatiotemporal interaction layer to obtain target sample features includes: determining the weights of detail maintenance features based on the detail maintenance features through the spatiotemporal interaction layer; determining the weights of semantic interaction features based on the semantic interaction features through the spatiotemporal interaction layer; and performing weighted fusion of the detail maintenance features and semantic interaction features based on the weights of the detail maintenance features and the semantic interaction features to obtain target sample features. The weight calculation network can be a multilayer perceptron (MLP), etc.

[0096] In other words, the weights of detail maintenance features and semantic interaction features can be determined by the weight calculation network in the spatiotemporal interaction layer. Then, the detail maintenance features and semantic interaction features are weighted and summed to achieve the fusion processing of detail maintenance features and semantic interaction features, and the weighted sum is used as the target sample feature.

[0097] For example, when the weight calculation network is an MLP, the process of determining the target sample features is as follows: Figure 4 As shown, firstly, the sample image features are convolved to obtain the convolutional result as the detail maintenance feature. Simultaneously, semantic interaction is performed on the sample image features to obtain the semantic interaction feature. Then, the weights of the semantic interaction feature and the detail maintenance feature are determined by MLP. Based on the weights of the semantic interaction feature and the detail maintenance feature, the semantic interaction feature and the detail maintenance feature are weighted and summed to obtain the sum result. The sum result is then normalized to reduce the differences between the data to obtain the target sample feature.

[0098] In this embodiment, the spatiotemporal interaction layer is used to maintain the detailed features of the sample image features, reducing the decay of fine-grained features. This results in the target sample features not only including the features corresponding to different predicted sample images and the contextual information between the features corresponding to different predicted sample images, but also having more fine-grained features. The accuracy of the target sample features is further improved. Thus, by adding the detailed feature maintenance step to the spatiotemporal interaction layer, the information processing capability of the dynamic image generation model is further improved, resulting in a stronger information processing capability of the trained dynamic image generation model and a further improvement in the accuracy of the dynamic images generated by the trained dynamic image generation model.

[0099] As mentioned above, the sample input information can include cluster centers and random sample noise. Therefore, when the dynamic image generation model generates multiple predicted sample images based on the sample input information, it is equivalent to denoising the random sample noise in the sample input information based on the second cluster center in the sample input information, and obtaining the result after denoising each random sample noise as a predicted sample image. Each second cluster center in the sample input information also obtains a predicted sample image after being processed by the dynamic image generation model.

[0100] Based on this, different preset masks can be set according to different needs to construct different training tasks and implement conditional masks (conditional masks are used to switch between different training tasks, achieved by masking out appropriate images and text guidance). The training tasks involved in this application may include:

[0101] Interpolation tasks, prediction tasks, and generation tasks.

[0102] Interpolation (IPT) refers to obtaining sample input information by masking out the middle cluster center among multiple cluster centers after clustering using a preset mask. Therefore, when a dynamic image generation model generates a predicted sample image based on the sample input information, it is equivalent to "interpolating" among the multiple cluster centers included in the sample input information.

[0103] Prediction task (Pre&Post Prediction, PDT) refers to using a preset mask to mask out the cluster centers at the front to retain multiple cluster centers at the back, or mask out the cluster centers at the back to retain multiple cluster centers at the front. Thus, when generating a prediction sample image based on sample input information, it is equivalent to predicting cluster centers forward or backward based on the unmasked second cluster centers.

[0104] Generation tasks can include text-based generation (GT) and image-based generation (IG). Text-based generation involves masking all cluster centers and generating multiple predicted sample images using only the text describing the sample images. Image-based generation involves retaining only one cluster center to generate multiple predicted sample images using that single cluster center.

[0105] Given the different training tasks mentioned above, different preset masks can be set to obtain sample data for different training tasks based on the same sample prompt information. The dynamic image generation model can then be trained based on the sample data for different training tasks, thereby achieving the goal of enhancing the diversity and usability of the dynamic image generation model based on limited data and improving the accuracy of the dynamic images generated by the trained dynamic image generation model.

[0106] S130. Train the dynamic image generation model based on the features of the target sample.

[0107] After obtaining the target sample features, the parameters of the dynamic image generation model (including the parameters of each network layer in the feature extraction block and the spatiotemporal interaction layer) can be adjusted based on the target sample features until the training termination condition is met. The training termination condition may include reaching a specified number of iterations or the model loss used to train the dynamic image generation model being less than a loss threshold. The specified number of iterations is, for example, 1000 times, and the loss threshold is, for example, 0.1.

[0108] In some implementations, dynamic images can be generated based on the features of the target samples to obtain multiple predicted sample images. Then, based on the differences between the multiple predicted sample images and the multiple sample images, the loss is calculated to obtain the model loss. The parameters of the dynamic image generation model are then adjusted using the model loss. The function used to calculate the model loss can be the cross-entropy loss function or the mean squared error loss function, etc.

[0109] In some other implementations, the features corresponding to the first cluster center can be determined from the features of the target samples and used as the predicted sample noise corresponding to the first cluster center; the model loss can be determined based on the difference between the random sample noise corresponding to the first cluster center and the predicted sample noise; and the parameters of the dynamic image generation model can be adjusted through the model loss.

[0110] In other words, the model loss is determined based solely on the difference between the random noise of the first cluster center in the masked sample and the feature corresponding to the first cluster center in the target sample features, without considering the feature corresponding to the second cluster center in the target sample features. This achieves loss masking, making the model loss more accurately indicate the expression generation capability of the dynamic image generation model and improving the training effect of the dynamic image generation model through model loss.

[0111] like Figure 5 As shown, after clustering and masking the sample images in the sample dynamic image, the target sample features are obtained. The target sample features are the features of the first cluster center corresponding to the masked image 501 and the features of the second cluster center corresponding to the unmasked image 5011. When calculating the model loss, only the ground truth (the preset ground truth of each cluster center, the preset ground truth of the first cluster center is the sampled random sample noise, and the preset ground truth of the second cluster center is itself) is considered as the partial ground truth 502 corresponding to the first cluster center, and the partial ground truth 5021 corresponding to the second cluster center (that is, the sampled random sample noise) is not considered. Correspondingly, only the predicted value (the feature of each cluster center in the target sample features) is considered as the partial predicted value 503 corresponding to the first cluster center (that is, the feature of the first cluster center in the target sample features), and the partial predicted value 5031 corresponding to the second cluster center is not considered. Thus, loss masking is achieved, and the accuracy of the determined model loss is improved.

[0112] In this embodiment, the dynamic image generation model includes not only a feature extraction block for extracting features, but also a temporal interaction layer. The temporal interaction layer performs semantic interaction on the features corresponding to different predicted sample images extracted from the sample image features by the feature extraction block, thereby mining the contextual information between the features corresponding to different predicted sample images and obtaining semantic interaction features. Then, the temporal interaction layer continues to determine the target sample features based on the semantic interaction features and the sample image features. Thus, the target sample features not only include the features corresponding to different predicted sample images, but also the contextual information between the features corresponding to different predicted sample images, making the information expressed by the target sample features more comprehensive and complete. In other words, by adding a spatiotemporal interaction layer to the dynamic image generation model, the information processing capability of the dynamic image generation model is improved, resulting in a stronger information processing capability and higher accuracy of the dynamic images generated by the trained dynamic image generation model.

[0113] Furthermore, the spatiotemporal interaction layer maintains the detailed features of the sample images, reducing the decay of fine-grained features. This ensures that the obtained target sample features not only include the features corresponding to different predicted sample images and the contextual information between the features corresponding to different predicted sample images, but also have more fine-grained features, further improving the accuracy of the target sample features. Thus, by adding the detailed feature maintenance step to the spatiotemporal interaction layer, the information processing capability of the dynamic image generation model is further improved, resulting in a stronger information processing capability of the trained dynamic image generation model and a further improvement in the accuracy of the dynamic images generated by the trained dynamic image generation model.

[0114] Furthermore, by setting different masks for different training tasks to implement conditional masking, multiple training tasks can be performed on the same samples during the same training process, increasing data diversity. Simultaneously, loss masking improves the accuracy of model loss, and the encoding results of clustered sample images enhance the information density of the data. Therefore, by employing a dual-mask strategy (conditional masking and loss masking) and feature clustering (encoding results of clustered sample images), the dynamic image generation model can be trained more effectively using short-frame data (i.e., cluster centers after clustering) and the information density of long-frame data (i.e., multiple sample images before clustering), thereby enhancing the learning performance of the dynamic image generation model.

[0115] In one embodiment, such as Figure 6 As shown, S120 also includes:

[0116] S210. Based on the input and time step features at the t-th time step, the feature extraction module determines the sample image features corresponding to the t-th time step.

[0117] Where t∈[1,T], t is an integer, and T is a positive integer; the input of the first time step is the sample input information. The time step of the t-th time step can be encoded by the text encoder in the clip model and the text encoder in the T5 model to obtain the time step features of the t-th time step.

[0118] In other words, in this application, the process of the dynamic image generation model predicting the sample dynamic image each time is actually a diffusion process of the dynamic image generation model at multiple time steps. Therefore, the sample image features determined by the feature extraction module include the sample image features at each time step. For the t-th time step, the input of the t-th time step and the time step features can be input into the feature extraction module, and the output of the feature extraction module is used as the sample image features corresponding to the t-th time step.

[0119] In some implementations, the aforementioned sample prompt information includes a sample dynamic image and sample image description text corresponding to the sample dynamic image, wherein the sample dynamic image includes multiple sample images; S210 may further include: obtaining at least one of a first cluster center and sample image description text as sample supplementary information; wherein, the first cluster center is a cluster center among multiple cluster centers that is masked by a preset mask; the multiple cluster centers are obtained by clustering multiple sample images; and the feature extraction module performs feature extraction based on the input corresponding to the t-th time step, the time step features, and the sample supplementary information to determine the sample image features corresponding to the t-th time step.

[0120] In other words, for each time step, the data input to the feature extraction module includes the input corresponding to the t-th time step, the time step features, and additional sample information. The feature extraction module then extracts features from the input corresponding to the t-th time step, the time step features, and the additional sample information to determine the sample image features corresponding to the t-th time step.

[0121] S220. Semantic interaction is performed on the features corresponding to different predicted sample images in the sample image features corresponding to the t-th time step through the spatiotemporal interaction layer to obtain the semantic interaction features corresponding to the t-th time step.

[0122] In other words, when multiple time steps are involved, the semantic interaction features determined by the spatiotemporal interaction layer include the semantic interaction features of each of the multiple time steps.

[0123] In some implementations, semantic interaction features corresponding to different predicted sample images in the sample image features corresponding to the t-th time step can be obtained directly by using the self-attention network in the spatiotemporal interaction layer.

[0124] In some other implementations, the spatiotemporal interaction layer can be used to downsample the sample image features corresponding to the t-th time step to obtain the downsampled image features corresponding to the t-th time step; the spatiotemporal interaction layer can then perform self-attention operations on the downsampled image features corresponding to the t-th time step to obtain the attention interaction image features corresponding to the t-th time step; finally, the spatiotemporal interaction layer can be used to upsample the attention interaction image features corresponding to the t-th time step to obtain the semantic interaction features corresponding to the t-th time step. Here, the upsampling factor and the downsampling factor are the same to ensure that the size of the semantic interaction features corresponding to the t-th time step is consistent with that of the sample image features, thus guaranteeing the smooth training process of the dynamic image generation model.

[0125] Specifically, the sample image features corresponding to the t-th time step can be downsampled first. Then, the downsampled image features corresponding to the t-th time step can be expanded in terms of time and space dimensions. Self-attention network is used to perform self-attention operation on the expanded features in terms of time and space dimensions. After that, the attention interaction image features corresponding to the t-th time step after the self-attention operation are upsampled, and the upsampled results are transformed by tensors to restore the time and space dimensions, thus obtaining the semantic interaction features corresponding to the t-th time step.

[0126] S230. Based on the sample image features and semantic interaction features corresponding to the t-th time step, the output of the t-th time step is determined by the spatiotemporal interaction layer.

[0127] The spatiotemporal interaction layer can be used to fuse the sample image features and semantic interaction features corresponding to the t-th time step, and the fusion result is used as the output of the t-th time step. Here, the fusion process can include summation, weighted summation, and concatenation.

[0128] In some implementations, the spatiotemporal interaction layer can be used to maintain the detailed features of the sample image corresponding to the t-th time step, thereby obtaining the detailed maintenance features corresponding to the t-th time step; the spatiotemporal interaction layer can be used to fuse the detailed maintenance features and semantic interaction features corresponding to the t-th time step, thereby obtaining the output of the t-th time step.

[0129] Specifically, the spatiotemporal interaction layer can be used to convolve the sample image features at time step t to reduce the attenuation of fine-grained features and maintain the detailed features of the sample image features, thus obtaining the detailed maintenance features corresponding to time step t. Then, the spatiotemporal interaction layer can be used to fuse the detailed maintenance features and semantic interaction features corresponding to time step t. This feature fusion can be achieved through summation, weighted summation, or concatenation, etc.

[0130] In some implementations, similar to the aforementioned process, the weights of the detailed maintenance features at time step t can be determined by the spatiotemporal interaction layer based on the detailed maintenance features at time step t; the weights of the semantic interaction features at time step t can be determined by the spatiotemporal interaction layer based on the semantic interaction features at time step t; and the detailed maintenance features and semantic interaction features at time step t can be weighted and fused based on their respective weights to obtain the output at time step t. Here, the weights of the detailed maintenance features and semantic interaction features at time step t can be determined by a weight calculation network in the spatiotemporal interaction layer.

[0131] S240. If t is less than T, obtain the output of the t-th time step as the input of the (t+1)-th time step, and increment t by 1.

[0132] When t is less than T, obtain the output of the t-th time step as the input of the (t+1)-th time step, and after incrementing t by 1, return to execute S210.

[0133] S250. If t = T, obtain the output at time step T as the feature of the target sample.

[0134] In this embodiment, the target sample features for generating the predicted sample image are obtained by the dynamic image generation model through diffusion processing at multiple time steps. This achieves efficient and accurate feature extraction, resulting in a high accuracy of the target sample features. This improves the accuracy of the predicted sample dynamic image generated based on the target sample features, enhances the training effect of the dynamic image generation model, and consequently makes the generated dynamic image more accurate when the dynamic image generation model is used for inverse diffusion processing.

[0135] In one embodiment, the training process of the dynamic image generation model includes multiple iterative processes; such as Figure 7 As shown, after S130, the method further includes:

[0136] S310. For the t-th iteration, based on the target sample features obtained in the t-th iteration, determine the model loss corresponding to the t-th iteration.

[0137] The process of determining the model loss has been described above and will not be repeated here.

[0138] S320. Determine the difference between the model loss corresponding to the t-th iteration and the historical cumulative model loss, and use it as the loss difference for the t-th iteration.

[0139] The historical cumulative model loss is the cumulative result of the model loss of the t-1 iterations prior to the t-th iteration.

[0140] In other words, the sum of the model loss from the first iteration to the (t-1)th iteration is calculated as the historical cumulative model loss. Then, the difference between the model loss corresponding to the tth iteration and the historical cumulative model loss is calculated as the loss difference for the tth iteration. This process can be summarized by the following formula:

[0141]

[0142] Among them, L c (t) represents the model loss during the t-th iteration, L s (t) represents the loss difference in the t-th iteration.

[0143] S330. Based on the loss difference of the t-th iteration, determine the joint perplexity of the (t+1)-th iteration process.

[0144] The joint perplexity of the (t+1)th iteration is used to indicate the accuracy of the dynamic expression predicted by the dynamic image generation model in the (t+1)th iteration. The higher the perplexity, the lower the accuracy of the dynamic expression predicted by the dynamic image generation model, and vice versa.

[0145] In some implementations, the rate of change of loss difference and the cumulative loss difference of the p-th iteration process can be determined based on the loss difference of the p-th iteration; the joint perplexity of the (p+1)-th iteration process can be obtained by weighted summing of the loss difference, the rate of change of loss difference and the cumulative loss difference of the p-th iteration.

[0146] Wherein, the rate of change of loss difference in the p-th iteration process can refer to the difference between the loss difference in the p-th iteration process and the loss difference in the (p-1)-th iteration process; the cumulative loss difference in the p-th iteration process can refer to the cumulative result of the loss differences in the p iteration processes. The weights of loss difference, rate of change of loss difference, and cumulative loss difference can be set based on requirements, and this application does not impose any restrictions. In this case, the determination process of the joint perplexity of the (p+1)-th iteration process is briefly described by the following formula:

[0147]

[0148] Among them, P * Let the joint perplexity be the degree of the (p+1)th iteration. The cumulative loss difference in the p-th iteration process, Let K be the rate of change of loss difference in the t-th iteration. pK i and K d Weights are assigned to the loss difference, cumulative loss difference, and rate of change of loss difference.

[0149] S340. Obtain the target difficulty level corresponding to the joint perplexity of the (t+1)th iteration process; obtain the mask corresponding to the target difficulty level as the preset mask for the (t+1)th iteration process.

[0150] Multiple perplexity intervals can be identified, and a corresponding difficulty level can be assigned to each interval. The difficulty level corresponding to the perplexity interval containing the joint perplexity of the t+1 iterations is the target difficulty level. A higher difficulty level indicates a more difficult sample, and a lower difficulty level indicates a less difficult sample. As mentioned earlier, higher perplexity leads to lower accuracy in predicting dynamic expressions by the dynamic image generation model. Therefore, a lower difficulty level is set for perplexity intervals with higher values.

[0151] Generally speaking, interpolation tasks are used to insert images into a given image. The interpolation process is equivalent to inserting images between images. The interpolation process can rely on the preceding and following images, making it relatively easy. For prediction tasks, the prediction process is based on the preceding image to predict the following image, or based on the following image to predict the preceding image. Therefore, the prediction process can only rely on the preceding or following images, making it more difficult than the interpolation task. For generation tasks, the prediction process is based on text or a very small number of images to generate more images. The prediction process can only rely on text or a very small number of images to generate more images, making it extremely difficult. Therefore, generation tasks are the most difficult.

[0152] Different training tasks can be implemented using different masks. For example, an interpolation task can be constructed using the middle image of the mask; a prediction task can be constructed using the front or rear image of the mask; or a generation task can be constructed by retaining only one image through the masking process. Based on this, multiple masks can be set according to the difficulty of different training tasks and the criteria for distinguishing different difficulties of the same type of task. Each mask is assigned to a difficulty level, and a corresponding perplexity interval is determined for each difficulty level. This allows the preset mask for the (t+1)th iteration process to be determined based on the joint perplexity of the (t+1)th iteration process.

[0153] For example, masks f1 and f2 are set for interpolation tasks, with mask f1 used for interpolation tasks involving a small number of images and mask f2 used for interpolation tasks involving a large number of images. Masks f3 and f4 are set for prediction tasks, with mask f3 used for prediction tasks involving a small number of images and mask f4 used for prediction tasks involving a large number of images. Mask f5 is set for generation tasks. In this case, the difficulty levels corresponding to masks f1, f2, f3, f4, and f5 increase sequentially.

[0154] like Figure 8 As shown, for different masking routes (mask route 1 refers to the conditional mask, and mask route 2 refers to the loss mask), the larger the amount of data in the mask (the less information is retained after masking), the higher the difficulty. The interpolation task is less difficult than the prediction task, and the generation task is more difficult than the prediction task.

[0155] Entropy is a metric for measuring sample uncertainty. Lower entropy results in lower time series complexity and higher self-similarity. Therefore, to gradually increase sample complexity and difficulty during training while ensuring stable convergence at different stages and avoiding instability during training, it is necessary to control the sample entropy to exhibit a monotonically increasing trend globally. The static probability for different training tasks needs to satisfy the following formula:

[0156]

[0157] in, This refers to the static probability of the interpolation task in the t-th iteration. This refers to the static probability of the prediction task in the t-th iteration. It refers to the static probability of the generation task (including image generation task and text generation task) in the t-th iteration process.

[0158] In other words, for interpolation tasks, the static probability continuously decreases, while for prediction and generation tasks, the static probability continuously increases. This ensures that the sample entropy exhibits a monotonically increasing trend globally during training. In this application, however, different training tasks of varying difficulty are selected using joint perplexity. This ensures that the static probability continuously decreases for interpolation tasks, while continuously increasing for prediction and generation tasks. This allows the sample difficulty to be adaptively adjusted based on the loss after each iteration, rather than gradually increasing with the number of iterations. This promotes the convergence of the dynamic image generation model and improves its generalization ability.

[0159] The aforementioned process of adaptively selecting training tasks of varying difficulty is also known as Difficulty-adaptive Curriculum Learning, a learning strategy used to promote model convergence, specifically in the context of a dual-mask strategy. The core idea of ​​Difficulty-adaptive Curriculum Learning is to gradually increase the difficulty of the samples by decomposing the sample entropy into a static component and an adaptive component, thereby achieving smooth model convergence.

[0160] Specifically, it involves configuring static components. To constrain the changing trend of static probabilities of different training tasks during the iteration process, an adaptive component is used to select different masks based on the joint perplexity during the iteration process, so as to select training tasks of different difficulty.

[0161] In this embodiment, by using adaptive learning courses to gradually increase the complexity and difficulty of the samples, the dynamic image generation model can be ensured to converge stably at different stages, avoiding instability during training and improving the convergence speed and generation quality of the dynamic image generation model.

[0162] Please see Figure 9 , Figure 9 This application illustrates a flowchart of a dynamic image display method according to an embodiment of the present application. This method can be applied to an electronic device, which may be... Figure 1 The method includes: (The method is described in the context of) terminal 20 or server 10.

[0163] S410. In response to obtaining the target text, display the target dynamic image.

[0164] The target dynamic image is generated based on target prompt information by a dynamic image generation model. The target prompt information is determined based on target text. The dynamic image generation model is trained according to the method described in the previous embodiment. The target text refers to the text used to generate the target dynamic image.

[0165] In some implementations, the target text can be text entered by the user; for example, if a user enters "happy" in the chat interface, then "happy" becomes the target text. Figure 10 As shown, enter "Hello" in the text input box 101 of the chat interface 10, and "Hello" will be the target text.

[0166] In some other implementations, the target text may also be text obtained by the electronic device through keyword extraction or semantic extraction of text information. The text information may be logs, chat logs, and literary works, etc., and the keywords may include emotions or greetings, etc.

[0167] For example, in multi-turn dialogue scenarios within intelligent question-answering and dialogue systems, this method can be used to endow the robot with multi-turn dialogue capabilities when a user converses with a chatbot (or intelligent customer service). This allows the robot to make appropriate contextual judgments based on the current dialogue context, thereby guiding the current dynamic image generation process. For instance, semantic extraction can be performed based on the user's chat history with the chatbot to obtain target text indicating emotions, and then the target text can be used to generate dynamic stickers or dynamic emoticons (i.e., the aforementioned target dynamic images).

[0168] In this application, in response to obtaining target text, at least one preset image matching the content of the target text can be obtained, and target input information for input dynamic image generation model can be constructed using the preset image and / or random noise collected from the preset noise distribution, so that the dynamic image generation model can process based on the target input information to obtain target features.

[0169] As mentioned above, the number of predicted images generated by the trained dynamic image generation model is the reference number. Therefore, the target input information can be determined based on the encoding result of a preset image and the reference number - 1 random noise samples in the preset noise distribution, and then based on the random noise sampled from the encoding result of the preset image. Specifically, the encoding result of the preset image and the sampled random noise can be concatenated into a vector in the manner described above, and the concatenated vector is used as the target input information.

[0170] Of course, in some implementations, the target input information can be constructed using only a number of random noise samples collected as a reference. Then, the encoded target text and the target input information are input into a dynamic image generation model to obtain the target features output by the dynamic image generation model. The processing of the encoded target text and the target input information here refers to the processing of the sample input information and sample supplementary information in the aforementioned embodiments, and will not be repeated here.

[0171] After obtaining the target features, image generation is performed based on these features to obtain multiple predicted images. Then, these multiple predicted images are combined to create a dynamic image of the target.

[0172] Once the target dynamic image is obtained, it can be directly displayed so that users can view it.

[0173] Of course, for the same target text, step S410 can be repeated to obtain multiple target animated images, and correspondingly, multiple target animated images can be displayed. For example Figure 10 As shown, after the target text "Hello" is entered in the text input box 101, the target dynamic image (i.e., ...) is displayed in the dynamic image display area 102. Figure 10 (Recommended animated emoticons in the app), Figure 10 The image displays six moving images of the target.

[0174] For example, in social media campaigns, animated stickers can be generated using motion image generation models to enhance the fun of social interaction. In social applications, motion image generation models can generate animated emoticons based on user input, eliminating the need for users to manually select emoticons, reducing manual steps, and improving the user's social experience.

[0175] Of course, for human-computer dialogue scenarios, the user's input message can be used as the target text, and the dynamic image generation model can generate dynamic emoticons based on the target text. This eliminates the need for the user to manually select dynamic emoticons, reducing the steps of manual operation and thus increasing the user experience of human-computer dialogue. Similarly, when intelligent customer service replies to user messages, the reply message can also be used as the target text, and the dynamic image generation model can generate dynamic emoticons based on the target text. The generated dynamic emoticons can then be sent back to the user to increase the fun of human-computer dialogue.

[0176] In addition, in game development or visual animation development scenarios, dynamic image generation models can automatically generate dynamic avatars, dynamic emoticons, and dynamic videos based on given text, thus eliminating the need for users to manually create dynamic avatars, dynamic emoticons, and dynamic videos, thereby improving the efficiency of game development and visual animation development.

[0177] In this embodiment, by adding a spatiotemporal interaction layer to the dynamic image generation model, the information processing capability of the dynamic image generation model is improved, thereby making the information processing capability of the dynamic image generation model stronger. This results in a higher accuracy of the target dynamic image generated by the trained dynamic image generation model based on the target text, and greatly improves the dynamic image generation effect.

[0178] To more clearly explain the technical solution of this application, the training method of the dynamic image generation model of this application will be explained below with reference to an example.

[0179] In this example, the dynamic image generation model is built on UNet. UNet includes multiple feature extraction modules, each of which includes at least one residual network for feature extraction. A temporal interaction layer is configured between each feature extraction module to further process the features extracted by the modules. Based on this structure, the training process of the dynamic image generation model is as follows: Figure 11 As shown.

[0180] First, animated emoticons can be obtained as sample animated images, and image clustering can be performed on the sample images in the sample animated emoticons (one frame of the animated emoticon is used as a sample image) to obtain 5 cluster centers. The 5 cluster centers actually indicate 5 images.

[0181] Different preset masks can be constructed for different training tasks, for example, Figure 11 In the context of interpolation tasks, masks can be constructed to mask the second and fourth cluster centers (meaning predicting the second cluster center based on the first and third cluster centers, and predicting the fourth cluster center based on the fifth and third cluster centers). For prediction tasks, masks can be constructed to mask the first and second cluster centers (meaning predicting the first and second cluster centers based on the third to fifth cluster centers), or masks can be constructed to mask the fifth and fourth cluster centers (meaning predicting the fifth and fourth cluster centers based on the first to third cluster centers). For generation tasks, masks can be constructed to mask the second to fifth cluster centers (meaning predicting the second to fifth cluster centers based on the first cluster center), or masks can be constructed to mask all five cluster centers (meaning predicting five cluster centers based on the sample image description text of the sample dynamic image).

[0182] The five cluster centers are masked using different masks to obtain unmasked second cluster centers. These second cluster centers are then concatenated with random sample noise from a preset noise distribution to obtain sample input information.

[0183] Descriptive text for sample images ( Figure 11 The text is described in Chinese. First, it is generalized. Then, the result of the generalization is encoded (e.g., by using the text encoder in clip or T5). The encoded result is then embedded to obtain the text input data. The second cluster center and / or text input data are obtained as additional information for the sample (in the case of masking all cluster centers, the additional information for the sample includes at least the text input data).

[0184] Then, the sample input information and sample additional information are input into the dynamic image generation model, which processes them. Specifically, for each time step t, the time step features, sample input information, and sample additional information of time step t are first input into the first feature extraction module for feature extraction to obtain the first intermediate sample image features corresponding to time step t. Then, the first intermediate sample image features of time step t are input into the first spatiotemporal interaction layer to obtain the first intermediate output of time step t. The first intermediate output, sample additional information, and time step features of time step t are then input into the second feature extraction module to obtain the second intermediate sample image features. Then, the second intermediate sample image features are input into the second spatiotemporal interaction layer to obtain the first intermediate output of time step t. This process continues until the output of the last spatiotemporal interaction layer is obtained, which is used as the output of time step t, until the output of the Tth time step is obtained, which is used as the target sample feature.

[0185] After obtaining the target sample features, the model loss is determined based on the difference between the features of the first cluster center in the target sample features and the random noise of the sample corresponding to the first cluster center. The parameters of the dynamic image generation model are then adjusted based on the model loss. This adjustment can be made to the features of each feature extraction module and the spatiotemporal interaction layer in the dynamic image generation model.

[0186] Of course, for each iteration, the joint perplexity of the next iteration can also be determined, so as to select the preset mask for the next iteration based on the joint perplexity of the next iteration, thereby determining samples of different difficulty for different iterations.

[0187] In this example, the design of the spatiotemporal interaction layer enables the dynamic image generation model to better capture the temporal features and semantic information of dynamic images, significantly improving the detail and coherence of the generated dynamic images. Furthermore, the dual-masking strategy of conditional masking and loss masking enhances data diversity and information density, allowing the model to make fuller use of limited training data and generate more diverse and innovative stickers. Moreover, based on the model loss at each iteration, the joint perplexity is determined, and a preset mask corresponding to the difficulty level is selected based on the joint perplexity. This achieves adaptive learning, ensuring stable convergence of the model at different stages while gradually increasing the complexity and difficulty of the samples, avoiding instability during training and improving the model's convergence speed and generation quality.

[0188] Please see Figure 12 , Figure 12This illustration shows a block diagram of a training apparatus for a dynamic image generation model according to an embodiment of this application. The training apparatus 900 for the dynamic image generation model includes:

[0189] The information determination module 910 is used to determine the sample input information based on the sample prompt information; the sample prompt information is used to indicate the content included in the predicted sample dynamic image to be generated; the predicted sample dynamic image includes multiple predicted sample images;

[0190] The feature extraction module 920 is used to extract features from the sample input information to obtain the sample image features corresponding to the predicted sample dynamic image; the sample image features include the features of each of the multiple predicted sample images.

[0191] The semantic interaction module 930 is used to perform semantic interaction on the features corresponding to different predicted sample images in the sample image features through the spatiotemporal interaction layer to obtain semantic interaction features.

[0192] The feature determination module 940 is used to determine the target sample features corresponding to the predicted sample dynamic image based on the sample image features and semantic interaction features through the spatiotemporal interaction layer.

[0193] Training module 950 is used to train the dynamic image generation model based on the features of the target sample.

[0194] Optionally, the feature determination module 920 is also used to maintain the detailed features through the sample image features to obtain the detailed maintenance features; and to perform feature fusion of the detailed maintenance features and the semantic interaction features through the spatiotemporal interaction layer to obtain the target sample features.

[0195] Optionally, the feature determination module 920 is also used to perform convolution processing on the sample image features through the spatiotemporal interaction layer to obtain detail maintenance features.

[0196] Optionally, the feature determination module 920 is further configured to determine the weight of the detail maintenance features based on the detail maintenance features through the spatiotemporal interaction layer; determine the weight of the semantic interaction features based on the semantic interaction features through the spatiotemporal interaction layer; and perform weighted fusion of the detail maintenance features and the semantic interaction features based on the weight of the detail maintenance features and the weight of the semantic interaction features to obtain the target sample features.

[0197] Optionally, the semantic interaction module 930 is further configured to downsample the sample image features through the spatiotemporal interaction layer to obtain downsampled image features; perform self-attention operations on the downsampled image features through the spatiotemporal interaction layer to obtain attention interaction image features; and upsample the attention interaction image features through the spatiotemporal interaction layer to obtain semantic interaction features.

[0198] Optionally, the feature extraction module 920 is further configured to determine the sample image features corresponding to the t-th time step based on the input and time step features of the t-th time step; t∈[1,T], t is an integer, T is a positive integer; the input of the 1st time step is the sample input information; correspondingly, the semantic interaction module 930 is further configured to perform semantic interaction on the features corresponding to different predicted sample images in the sample image features corresponding to the t-th time step through the spatiotemporal interaction layer to obtain the semantic interaction features corresponding to the t-th time step; correspondingly, the feature determination module 940 is further configured to determine the output of the t-th time step based on the sample image features and semantic interaction features corresponding to the t-th time step through the spatiotemporal interaction layer; if t is less than T, the output of the t-th time step is obtained as the input of the (t+1)-th time step, and t is incremented by 1, and the step of determining the sample image features corresponding to the t-th time step based on the input and time step features of the t-th time step is returned to be executed, until t=T, and the output of the T-th time step is obtained as the target sample feature.

[0199] Optionally, the sample prompt information includes a sample dynamic image and a corresponding sample image description text, wherein the sample dynamic image includes multiple sample images; the feature extraction module 920 is further used to obtain at least one of the first cluster center and the sample image description text as sample supplementary information; wherein, the first cluster center is the cluster center masked by a preset mask among multiple cluster centers; multiple cluster centers are obtained by clustering multiple sample images; the feature extraction module performs feature extraction based on the sample input information and the sample supplementary information to obtain sample image features.

[0200] Optionally, the sample prompt information includes a sample dynamic image, which includes multiple sample images; the information determination module 910 is also used to cluster the encoding results of the multiple sample images to obtain multiple cluster centers; and to determine the sample input information based on the multiple cluster centers and a preset noise distribution.

[0201] Optionally, the information determination module 910 is further configured to mask multiple cluster centers using a preset mask to obtain a first cluster center masked by the preset mask and a second cluster center not masked by the preset mask; determine the random sample noise corresponding to the first cluster center from a preset noise distribution; and determine sample input information based on the random sample noise corresponding to the first cluster center and the second cluster center.

[0202] Optionally, the training module is further configured to determine the features corresponding to the first cluster center from the features of the target samples, as the predicted sample noise corresponding to the first cluster center; determine the model loss based on the difference between the random sample noise corresponding to the first cluster center and the predicted sample noise; and adjust the parameters of the dynamic image generation model through the model loss.

[0203] Optionally, the training process of the dynamic image generation model includes multiple iterative processes; the training module 950 is further configured to, for the t-th iteration, determine the model loss corresponding to the t-th iteration based on the target sample features obtained in the t-th iteration; t is a natural number greater than 1; determine the difference between the model loss corresponding to the t-th iteration and the historical cumulative model loss as the loss difference of the t-th iteration; the historical cumulative model loss is the cumulative result of the model loss of the t-1 iterations prior to the t-th iteration; determine the joint perplexity of the t+1 iteration based on the loss difference of the t-th iteration; obtain the target difficulty level corresponding to the joint perplexity of the t+1 iteration; obtain the mask corresponding to the target difficulty level as the preset mask of the t+1 iteration.

[0204] Optionally, the training module 950 is also used to determine the rate of change of loss difference and the cumulative loss difference in the p-th iteration process based on the loss difference in the p-th iteration; and to perform a weighted summation of the loss difference, the rate of change of loss difference and the cumulative loss difference in the p-th iteration process to obtain the joint perplexity of the p+1-th iteration process.

[0205] Please see Figure 13 , Figure 13 This illustration shows a block diagram of a dynamic image display device according to an embodiment of this application. The dynamic image display device 1000 includes:

[0206] The display module 1010 is used to display a target dynamic image in response to the acquisition of target text. The target dynamic image is generated based on the target prompt information by a dynamic image generation model. The target prompt information is determined based on the target text. The dynamic image generation model is trained according to the aforementioned method.

[0207] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0208] Figure 14 A structural block diagram of an electronic device for performing an image processing method according to an embodiment of this application is shown. The electronic device may be... Figure 1 The terminal 20 or server 10, etc., should be noted that... Figure 14 The computer system 1200 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0209] like Figure 14As shown, the computer system 1200 includes a Central Processing Unit (CPU) 1201, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 1202 or programs loaded from storage portion 1208 into Random Access Memory (RAM) 1203. The RAM 1203 also stores various programs and data required for system operation. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An Input / Output (I / O) interface 1205 is also connected to the bus 1204.

[0210] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to I / O interface 1205 as needed. Removable media 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1210 as needed so that computer programs read from them can be installed into storage section 1208 as needed.

[0211] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by central processing unit (CPU) 1201, it performs various functions defined in the system of this application.

[0212] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0213] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0214] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0215] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries computer-readable instructions that, when executed by a processor, implement the methods in any of the above embodiments.

[0216] According to one aspect of the embodiments of this application, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the electronic device to perform the methods of any of the above embodiments.

[0217] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0218] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause an electronic device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0219] Other embodiments of this application will readily conceive of by those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. It should be understood that this application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

[0220] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A training method for a dynamic image generation model, characterized in that, The dynamic image generation model includes a feature extraction block and a spatiotemporal interaction layer, and the method includes: Based on the sample prompt information, the sample input information is determined; the sample prompt information is used to indicate the content included in the predicted sample dynamic image to be generated; the predicted sample dynamic image includes multiple predicted sample images; The feature extraction module extracts features from the sample input information to obtain sample image features corresponding to the predicted sample dynamic image; the sample image features include the features of each of the multiple predicted sample images. The spatiotemporal interaction layer performs semantic interaction on the features corresponding to different predicted sample images in the sample image features to obtain semantic interaction features. The spatiotemporal interaction layer determines the target sample features corresponding to the predicted sample dynamic image based on the sample image features and semantic interaction features. The dynamic image generation model is trained based on the features of the target sample.

2. The method according to claim 1, characterized in that, The step of determining the target sample features corresponding to the predicted sample dynamic image based on the sample image features and semantic interaction features through the spatiotemporal interaction layer includes: By maintaining the detail features through the features of the sample image, the detail-maintained features are obtained; The spatiotemporal interaction layer performs feature fusion on the detailed maintenance features and the semantic interaction features to obtain the target sample features.

3. The method according to claim 2, characterized in that, The maintenance of detail features through the sample image features to obtain detail-maintained features includes: The spatiotemporal interaction layer performs convolution processing on the sample image features to obtain the detail maintenance features.

4. The method according to claim 2, characterized in that, The step of fusing the detail maintenance features and the semantic interaction features through the spatiotemporal interaction layer to obtain the target sample features includes: The spatiotemporal interaction layer determines the weight of the detail maintenance feature based on the detail maintenance feature; The spatiotemporal interaction layer determines the weight of the semantic interaction features based on the semantic interaction features; Based on the weights of the detail maintenance features and the semantic interaction features, the detail maintenance features and the semantic interaction features are weighted and fused to obtain the target sample features.

5. The method according to any one of claims 1-4, characterized in that, The step of performing semantic interaction on the features corresponding to different predicted sample images in the sample image features through the spatiotemporal interaction layer to obtain semantic interaction features includes: The sample image features are downsampled through the spatiotemporal interaction layer to obtain the downsampled image features; The attention-interactive image features are obtained by performing self-attention operations on the downsampled image features through the spatiotemporal interaction layer. The semantic interaction features are obtained by upsampling the attention interaction image features through the spatiotemporal interaction layer.

6. The method according to claim 1, characterized in that, The process involves extracting features from the sample input information using the feature extraction module to obtain sample image features corresponding to the predicted sample dynamic image; performing semantic interaction on the features corresponding to different predicted sample images in the sample image features using the spatiotemporal interaction layer to obtain semantic interaction features; and determining the target sample features corresponding to the predicted sample dynamic image based on the sample image features and semantic interaction features using the spatiotemporal interaction layer, including: The feature extraction module determines the sample image features corresponding to the t-th time step based on the input and features of the t-th time step; t∈[1,T], where t is an integer and T is a positive integer; the input of the 1st time step is the sample input information; The spatiotemporal interaction layer performs semantic interaction on the features corresponding to different predicted sample images in the sample image features corresponding to the t-th time step, thereby obtaining the semantic interaction features corresponding to the t-th time step. The spatiotemporal interaction layer determines the output of the t-th time step based on the sample image features and semantic interaction features corresponding to the t-th time step. If t is less than T, obtain the output of the t-th time step as the input of the (t+1)-th time step, and increment t by 1. Then, return to the step of determining the sample image features corresponding to the t-th time step based on the input of the t-th time step and the time step features by the feature extraction module, until t = T, and obtain the output of the T-th time step as the target sample features.

7. The method according to claim 1, characterized in that, The sample prompt information includes a dynamic sample image and a corresponding sample image description text, wherein the dynamic sample image includes multiple sample images; The step of extracting features from the sample input information using the feature extraction module to obtain sample image features corresponding to multiple predicted sample images to be generated includes: At least one of the first cluster center and the sample image description text is obtained as additional information for the sample; wherein, the first cluster center is a cluster center masked by a preset mask among multiple cluster centers; the multiple cluster centers are obtained by clustering the multiple sample images; The feature extraction module extracts features based on the sample input information and the sample additional information to obtain the sample image features.

8. The method according to claim 1, characterized in that, The sample prompt information includes dynamic sample images, which include multiple sample images; The process of determining sample input information based on sample prompt information includes: The encoding results of the multiple sample images are clustered to obtain multiple cluster centers; Based on the multiple cluster centers and the preset noise distribution, the sample input information is determined.

9. The method according to claim 8, characterized in that, The step of determining sample input information based on the multiple cluster centers and a preset noise distribution includes: By masking the multiple cluster centers with a preset mask, a first cluster center that is masked by the preset mask and a second cluster center that is not masked by the preset mask are obtained; Determine the random sample noise corresponding to the first cluster center from the preset noise distribution; Based on the random sample noise corresponding to the first cluster center and the second cluster center, the sample input information is determined.

10. The method according to claim 9, characterized in that, The training of the dynamic image generation model based on the target sample features includes: The features corresponding to the first cluster center are determined from the features of the target sample and used as the predicted sample noise corresponding to the first cluster center; The model loss is determined based on the difference between the random sample noise corresponding to the first cluster center and the predicted sample noise. The parameters of the dynamic image generation model are adjusted using the model loss.

11. The method according to claim 7 or 9, characterized in that, The training process of the dynamic image generation model includes multiple iterative processes; After training the dynamic image generation model based on the target sample features, the method further includes: For the t-th iteration, based on the target sample features obtained in the t-th iteration, determine the model loss corresponding to the t-th iteration; t is a natural number greater than 1. The difference between the model loss corresponding to the t-th iteration and the historical cumulative model loss is determined as the loss difference of the t-th iteration; the historical cumulative model loss is the cumulative result of the model loss of the t-1 iterations prior to the t-th iteration. Based on the loss difference of the t-th iteration, determine the joint perplexity of the (t+1)-th iteration process; Obtain the target difficulty level corresponding to the joint perplexity of the (t+1)th iteration process; Obtain the mask corresponding to the target difficulty level, and use it as the preset mask for the (t+1)th iteration process.

12. The method according to claim 11, characterized in that, The determination of the joint perplexity of the (p+1)th iteration based on the loss difference of the p-th iteration includes: Based on the loss difference of the p-th iteration, determine the rate of change of loss difference and the cumulative loss difference of the p-th iteration process; The joint perplexity of the (p+1)th iteration is obtained by weighted summing of the loss difference, the rate of change of loss difference, and the cumulative loss difference for the p-th iteration.

13. A method for displaying dynamic images, characterized in that, The method includes: In response to obtaining the target text, a target dynamic image is displayed; the target dynamic image is generated based on the target prompt information by a dynamic image generation model, the target prompt information is determined based on the target text, and the dynamic image generation model is trained according to any one of the methods in 1-12 above.

14. A training device for a dynamic image generation model, characterized in that, The dynamic image generation model includes a feature extraction block and a spatiotemporal interaction layer, and the device includes: An information determination module is used to determine sample input information based on sample prompt information; the sample prompt information is used to indicate the content included in the predicted sample dynamic image to be generated; the predicted sample dynamic image includes multiple predicted sample images; The feature extraction module is used to extract features from the sample input information to obtain sample image features corresponding to the predicted sample dynamic image; the sample image features include the features of each of the plurality of predicted sample images. The semantic interaction module is used to perform semantic interaction on the features corresponding to different predicted sample images in the sample image features through the spatiotemporal interaction layer to obtain semantic interaction features; The feature determination module is used to determine the target sample features corresponding to the predicted sample dynamic image based on the sample image features and semantic interaction features through the spatiotemporal interaction layer. The training module is used to train the dynamic image generation model based on the features of the target sample.

15. A dynamic image display device, characterized in that, The device includes: The display module is used to display a target dynamic image in response to the acquisition of target text; the target dynamic image is generated based on target prompt information by a dynamic image generation model, the target prompt information is determined based on the target text, and the dynamic image generation model is trained according to any one of the methods in 1-12 above.

16. An electronic device, characterized in that, include: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1-12, or the method as described in claim 13.

17. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by a processor, implement the method as described in any one of claims 1-12, or implement the method as described in claim 13.

Images