Image generation method and device based on spatio-temporal data interaction and electronic equipment

Abstract

The invention provides an image generation method and device based on spatio-temporal data interaction and electronic equipment, relates to the technical field of computer vision, and aims to realize efficient image generation. The method comprises the following steps: acquiring a previous image representation sequence; the image characterization sequence comprises visual characterization and mask characterization, the mask characterization represents unknown image content, and the visual characterization represents known image content and is used for providing image information for the mask characterization to infer unknown image content; carrying out self-attention interaction-based coding on the visual representation to obtain a visual representation feature; decoding based on cross-self-attention interaction is carried out on the mask representation according to the visual representation features, and a new image representation sequence is obtained; and carrying out multiple times of image representation sequence iteration generation according to the steps, and generating an image according to the new image representation sequence under the condition that the new image representation sequence does not contain mask representation.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an image generation method, device and electronic device based on the interaction of spatiotemporal data. Background Art

[0002] The field of AI-generated content has seen unprecedented growth in recent years. In the context of natural language processing, content is often synthesized by generating discrete representations (tokens) using transformers (attention mechanisms).

[0003] Token-based generation methods have good results in synthesizing visual content. As a representative method, non-autoregressive transformers can generate multiple image tokens at the same time to generate high-quality images in a few steps. However, in each generation step of the non-autoregressive transformer, the visual token and the mask token are processed and interacted concurrently, that is, multiple types of data interactions are performed simultaneously. Therefore, the non-autoregressive transformer has the problems of large amount of calculation and low computational efficiency. Summary of the invention

[0004] In view of the above problems, the embodiments of the present application provide an image generation method, device and electronic device based on the interaction of spatiotemporal data, so as to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect of an embodiment of the present application, a method for generating an image based on the interaction of spatiotemporal data is disclosed, comprising:

[0006] Acquire the last image representation sequence; the image representation sequence includes a visual representation and a mask representation, the mask representation represents unknown image content, and the visual representation represents known image content, which is used to provide image information for the mask representation to infer the unknown image content;

[0007] Encoding the visual representation based on self-attention interaction to obtain visual representation features;

[0008] Decoding the mask representation based on cross-self-attention interaction according to the visual representation feature to obtain a new image representation sequence, wherein the new image representation sequence includes a newly added visual representation;

[0009] The image representation sequence is iteratively generated multiple times according to the above steps, and when the new image representation sequence does not contain a mask representation, an image is generated according to the new image representation sequence.

[0010] Optionally, in two adjacent new image representation sequences generated, the image representations at the newly added visual representation positions are significantly different, and the image representations at other positions are similar; the visual representation features include the newly added visual representation features; encoding the visual representation based on self-attention interaction to obtain the visual representation features includes:

[0011] Determining a newly added visual representation from the previous sequence of image representations;

[0012] The newly added visual representation is encoded based on cross-self-attention interaction to obtain new visual representation features.

[0013] Optionally, decoding the mask representation based on cross-self-attention interaction according to the visual representation feature to obtain a new image representation sequence includes:

[0014] Splicing the features of the previous image representation sequence with the newly added visual mark features to obtain a spliced ​​representation feature;

[0015] The mask representation is decoded based on cross-self-attention interaction using the spliced ​​representation features to obtain a new image representation sequence.

[0016] Optionally, the mask representation is decoded based on cross-self-attention interaction using the visual representation feature to obtain a new image representation sequence, including:

[0017] Generate a key representation and a value representation according to the visual representation feature and the mask representation feature corresponding to the mask representation;

[0018] Using the mask representation feature as a query representation;

[0019] Attention interaction is performed according to the key representation, the value representation and the query representation to obtain a new image representation sequence.

[0020] Optionally, the new image representation sequence is generated based on a disentanglement model, the disentanglement model includes a multi-layer encoder and a multi-layer decoder, and the number of layers of the multi-layer encoder is greater than the number of layers of the multi-layer decoder;

[0021] The multi-layer encoder is used to encode the visual representation based on self-attention interaction, and the multi-layer decoder is used to decode the mask representation based on cross-self-attention interaction according to the visual representation features.

[0022] Optionally, the disentanglement model is trained according to the following steps:

[0023] Constructing an image representation sequence set, wherein the image representation sequence set includes a plurality of image representation sequence samples;

[0024] Randomly masking the image representations in the image representation sequence samples to obtain image representation sequence samples containing masked representations;

[0025] A target training strategy is determined according to the target probability, and the disentanglement model is trained according to the target training strategy to predict the image representation corresponding to the mask representation position based on the image representation sequence samples including the mask representation.

[0026] Optionally, the target training strategy includes a first training strategy; training the disentanglement model according to the first training strategy to predict the image representation sequence samples based on the image representation sequence samples containing the mask representation includes:

[0027] The image representation sequence sample including the mask representation is used as the current image representation sequence sample, and the image representation sequence sample including the mask representation is randomly masked to simulate the previous image representation sequence sample;

[0028] Compare the current image representation sequence sample with the previous image representation sequence sample to determine a newly added visual mark sample;

[0029] Splicing the features of the previous image representation sequence sample with the features of the newly added visual mark sample to obtain a spliced ​​representation feature sample;

[0030] The concatenated representation feature sample is used to decode the mask representation in the image representation sequence sample at the previous moment based on cross-self-attention interaction to predict the image representation corresponding to the mask representation position.

[0031] A second aspect of the embodiments of the present application discloses an image generation device based on the interaction of spatiotemporal data, comprising:

[0032] An acquisition module, used to acquire a previous image representation sequence; the image representation sequence includes a visual representation and a mask representation, the mask representation represents unknown image content, and the visual representation represents known image content, and is used to provide image information for the mask representation to infer the unknown image content;

[0033] An encoding module, used for encoding the visual representation based on self-attention interaction to obtain visual representation features;

[0034] A decoding module, configured to decode the mask representation based on cross-self-attention interaction according to the visual representation feature to obtain a new image representation sequence, wherein the new image representation sequence includes a newly added visual representation;

[0035] A generating module is used to iteratively generate multiple image representation sequences according to the above steps, and to generate an image according to the new image representation sequence when the new image representation sequence does not contain a mask representation.

[0036] According to a third aspect of an embodiment of the present application, an electronic device is disclosed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the image generation method based on the interaction of spatiotemporal data described in the first aspect of the embodiment of the present application are implemented.

[0037] The fourth aspect of the embodiments of the present application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the image generation method based on the interaction of spatiotemporal data described in the first aspect of the embodiments of the present application are implemented.

[0038] According to a fifth aspect of an embodiment of the present application, a computer program product is disclosed, including a computer program, which, when executed by a processor, implements the steps of the image generation method based on spatiotemporal data interaction described in the first aspect of an embodiment of the present application.

[0039] The embodiments of the present application include the following advantages:

[0040] In an embodiment of the present application, a previous image representation sequence is obtained; the image representation sequence includes a visual representation and a mask representation, the mask representation represents unknown image content, and the visual representation represents known image content, which is used to provide image information for the mask representation to infer the unknown image content; and the visual representation is encoded based on self-attention interaction to obtain visual representation features; the mask representation is decoded based on cross-self-attention interaction according to the visual representation features to obtain a new image representation sequence, and the new image representation sequence includes a newly added visual representation; according to the above steps, multiple image representation sequences are iteratively generated, and when the new image representation sequence does not contain a mask representation, an image is generated according to the new image representation sequence.

[0041] In this way, in each generation process of an image representation sequence, the current visible and reliable image information is encoded by independently encoding the visual representation (i.e., the visual token); thus, the mask representation (i.e., the mask token) is decoded based on the encoded visual representation features to correctly predict the unknown image content based on the visible and reliable image information, thereby obtaining a new image representation sequence. Since this method performs different types of attention interaction processes independently in the generation process of the representation sequence, it is not necessary to perform all types of attention interactions, but only the key attention interactions, i.e., the self-attention interaction of the visual representation and the cross-self-attention interaction of the mask representation with the visual representation; thus, the amount of data calculation in the image generation process is reduced, and efficient image generation is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 It is a schematic diagram of the process of generating images based on non-autoregressive transformers;

[0044] Figure 2 It is a schematic diagram of the data interaction of a non-autoregressive transformer to generate an image;

[0045] Figure 3 is a flowchart of the steps of an image generation method based on the interaction of spatiotemporal data provided by an embodiment of the present application;

[0046] Figure 4 It is a structural schematic diagram of a de-entanglement model provided in an embodiment of the present application;

[0047] Figure 5 is a flowchart of another method for generating an image based on the interaction of spatiotemporal data provided by an embodiment of the present application;

[0048] Figure 6 is a schematic diagram of an image representation sequence generation method provided in an embodiment of the present application;

[0049] Figure 7 is a schematic diagram of the structure of another disentanglement model provided in an embodiment of the present application;

[0050] Figure 8 It is a structural schematic diagram of an image generation device based on spatiotemporal data interaction provided by an embodiment of the present application;

[0051] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] In order to better understand the technical solution of the present application, a brief introduction to the non-autoregressive transformer is first given:

[0054] like Figure 1 As shown in the figure, non-autoregressive transformers (NATs) follow a progressive generation paradigm. In each generation step, a certain number of latent image representations of the generated image are decoded in parallel. The non-autoregressive transformer performs this process iteratively to generate a final complete image representation map. Specifically, in each step, the unknown latent image representation of the image is represented by a mask representation and concatenated with the decoded image representation (i.e., the visual representation); then, the sequence of the complete mask representation and the visual representation is input into the transformer-based model, the appropriate value of the mask representation is predicted, and the most reliable prediction is retained as the increment of the visual representation for the next step. That is, for a generated image, starting from a sequence of image representations that are all mask representations, multiple image representations are decoded at each step, and then the pre-trained decoder is used to generate the image.

[0055] However, in each generation step of the non-autoregressive transformer, the visual representation and the mask representation are processed and interacted concurrently, that is, multiple types of data interactions are performed simultaneously. Therefore, the non-autoregressive transformer has the problems of large computational complexity and low computational efficiency.

[0056] In order to achieve efficient image generation, the embodiment of the present application analyzes the mechanism behind the effectiveness of the progressive generation process of the non-autoregressive transformer. Through analysis, it is found that at the spatial level, in each generation step, even if the mask representation and the visual representation are treated equally in the computational graph of the non-autoregressive transformer, the visual representation naturally learns to provide image information for the mask representation to infer the unknown image content, and their corresponding deep features can be constructed without the mask representation.

[0057] Specifically, to better understand this mechanism, we conducted an ablation study on four types of spatial interactions involved in the image generation process of the non-autoregressive transformer: (1) attention from mask representation to visual representation; (2) attention from visual representation to mask representation; (3) attention from visual representation to visual representation; (4) attention from mask representation to mask representation. These four types of spatial interactions have significantly different effects on the performance of the non-autoregressive transformer. Figure 2 As shown, Figure 2 The FID in is an indicator for evaluating the quality of the generated model. It can be seen that the most important spatial interaction is the attention from the mask representation to the visual representation (i.e., the information propagation from the visual representation → the mask representation). Without this attention, the model cannot converge at all. In addition, the attention from the mask representation to the mask representation and the visual representation to the visual representation (i.e., the self-attention during the feature extraction process of the visual representation and the mask representation, respectively) both improve the model moderately. However, removing the attention from the visual representation to the mask representation (i.e., the information propagation from the mask representation → the visual representation) only slightly hurts the performance of the model.

[0058] The unbalanced importance of these four types of spatial interactions highlights the different roles of visual and mask representations. Specifically, the processing of visual representations is mainly to build certain internal representations based on currently available and reliable image information and propagate them to mask representations; and the deep features corresponding to the visual representations can be built on top of the visual representations themselves. In contrast, the mask representation gradually acquires image information from the visual representation to predict appropriate representations corresponding to unknown parts of the image. That is, when learning to generate images effectively, the roles of visual and mask representations are naturally separated, even though mask and visual representations are treated equivalently in the computational graph of non-autoregressive transformers.

[0059] In addition, the interactions between adjacent generation steps are mainly focused on updating the representation of a small number of “key image representations” based on the previous steps, and the calculation of most of the remaining image representations is usually repeated.

[0060] Therefore, in order to achieve efficient image generation, based on the mechanism behind the effectiveness of the progressive generation process of non-autoregressive transformers, this application proposes the following technical concepts:

[0061] By encouraging inherent key interactions, efficient image generation is achieved. Specifically, at the spatial level, the computation of visual representation and mask representation (mask token) is disentangled by independently encoding the visual representation (visual token), and the mask representation is decoded based on the visual representation features obtained by encoding the visual representation, so as to correctly predict the unknown image content based on the visual and reliable image information, and achieve fast image generation. At the temporal level, the computation of the key representation of each step is prioritized, while maximizing the reuse of previously calculated representation features to supplement the necessary information.

[0062] Based on the above technical concept, the present application provides an image generation method based on the interaction of spatiotemporal data, referring to Figure 3 As shown, Figure 3 1 is a flowchart of a method for generating an image based on the interaction of spatiotemporal data provided by an embodiment of the present application. Figure 3 As shown, the image generation method based on the interaction of spatiotemporal data may include steps S310 to S340:

[0063] Step S310: Acquire the last image representation sequence; the image representation sequence includes a visual representation and a mask representation, the mask representation represents unknown image content, and the visual representation represents known image content, which is used to provide image information for the mask representation to infer the unknown image content.

[0064] Among them, the image representation sequence refers to the image token sequence, which contains multiple image representations. The image representation is divided into a visual representation (i.e., a visual token) and a mask representation (i.e., a mask token). The previous image representation sequence refers to the image representation sequence generated last time. For the first generation of an image representation sequence, the previous image representation sequence refers to the input initial image representation sequence, and the initial image representation sequence is an image representation sequence composed of mask representations.

[0065] Step S320: Encode the visual representation based on self-attention interaction to obtain visual representation features.

[0066] In an embodiment of the present application, the visual representation is processed independently of the mask representation. The current visible and reliable image information is encoded by encoding the visual representation based on self-attention interaction. That is, an internal representation of the visual representation is established based on the current visible and reliable image information to obtain the visual representation feature.

[0067] Specifically, encoding the visual representation based on self-attention interaction to obtain visual representation features includes: performing self-attention interaction and forward propagation processing on the visual representation through a multi-layer encoder to obtain the visual representation features. Each layer of the encoder includes a self-attention interaction layer and a forward propagation layer.

[0068] Step S330: decoding the mask representation based on cross-self-attention interaction according to the visual representation features to obtain a new image representation sequence, wherein the new image representation sequence includes a newly added visual representation.

[0069] In the embodiment of the present application, decoding the mask representation refers to predicting the unknown image content based on the current image information (i.e., the visual representation features), that is, obtaining image information from the visual representation to predict the appropriate image representation corresponding to the unknown part of the image, thereby obtaining a new visual representation. The new image representation sequence has more visual representations than the previous image representation sequence.

[0070] For example, the mask representation is decoded based on cross-self-attention interaction (i.e., SC-attention) according to the visual representation features, a probability distribution is predicted for each mask representation, and the probability distribution is sampled to obtain a prediction result corresponding to the mask representation, and the corresponding mask representation in the previous image representation sequence is replaced based on the reliable prediction result to obtain a new image representation sequence, wherein the reliable prediction result is the newly added visual representation.

[0071] Specifically, the mask representation is decoded based on cross-self-attention interaction using the visual representation features to obtain a new image representation sequence, including: generating a key representation and a value representation based on the visual representation features and the mask representation features corresponding to the mask representation; using the mask representation features as a query representation; and performing attention interaction based on the key representation, the value representation and the query representation to obtain a new image representation sequence.

[0072] In this way, the encoded visual representation (i.e., visual representation features) is effectively integrated into the decoding process of the mask representation through cross-self-attention interaction, and the unknown image content is quickly predicted based on the visual and reliable image information to obtain the newly added visual representation, thereby obtaining a new image representation sequence.

[0073] Step S340: performing multiple iterations of image representation sequence generation according to the above steps, and generating an image according to the new image representation sequence when the new image representation sequence does not contain a mask representation.

[0074] In an embodiment of the present application, multiple image representation sequences are iteratively generated according to the above steps S310 to S330. As the image representation sequence is iteratively generated, the number of mask representations decoded in the image representation sequence gradually increases. When a new image representation sequence generated in a certain iteration does not contain a mask representation, it means that all image information is already known. At this time, the iterative generation is terminated, and an image is generated according to the current new image representation sequence.

[0075] Specifically, generating an image according to the new image representation sequence includes: decoding the new image representation sequence using a decoder to obtain a corresponding image. For example, generating an image according to the new image representation sequence can be expressed as:

[0076]

[0077] in, represents the generated image, D VQ Represents a decoder, represents a new sequence of image representations, in which all image representations are visual representations.

[0078] Through the above implementation process, in each generation process of an image representation sequence, the current visible and reliable image information is encoded by independently encoding the visual representation (i.e., the visual token); thereby, the mask representation (i.e., the mask token) is decoded based on the encoded visual representation features to correctly predict the unknown image content based on the visible and reliable image information, thereby obtaining a new image representation sequence. Since this method performs different types of attention interaction processes independently in the generation process of the representation sequence, it is not necessary to perform all types of attention interactions, but only the key attention interactions, i.e., the self-attention interaction of the visual representation, and the cross-self-attention interaction of the mask representation with the visual representation; thus, the amount of data calculation in the image generation process is reduced, and efficient image generation is achieved.

[0079] In a specific embodiment, the new image representation sequence is generated based on a disentanglement model, which includes a multi-layer encoder and a multi-layer decoder, and the number of layers of the multi-layer encoder is greater than the number of layers of the multi-layer decoder; wherein the multi-layer encoder is used to encode the visual representation based on self-attention interaction, and the multi-layer decoder is used to decode the mask representation based on cross-self-attention interaction according to the visual representation features.

[0080] Reference Figure 4 As shown, Figure 4: is a schematic diagram of the structure of a disentanglement model provided in an embodiment of the present application. In which, each encoder layer includes a self-attention interaction layer and a forward propagation layer, and each decoder layer includes a cross-self-attention interaction layer and a forward propagation layer.

[0081] Specifically, the disentangled model independently encodes the visual representation to obtain visual representation features; thus, the mask representation is decoded based on the visual representation features based on cross-self-attention interactions to predict unknown image content and obtain a new image representation sequence. Compared with the method of generating image representation sequences using non-autoregressive deformers, the method of generating a new image representation sequence based on the disentangled model only needs to perform key attention interactions, that is, only the self-attention interaction of the visual representation and the cross-self-attention interaction of the mask representation with the visual representation, thereby achieving efficient image generation.

[0082] In addition, the performance of the disentanglement model is analyzed by the number of layers of different encoders and decoders, as shown in Table 1 (GFLOPs represents one trillion floating-point operations per second). It can be found that increasing the number of encoder layers and reducing the number of decoder layers can improve the performance of the disentanglement model. Therefore, in the disentanglement model, the number of layers of the multi-layer encoder is greater than the number of layers of the multi-layer decoder to allocate more calculations to the visual representation, thereby significantly improving performance without sacrificing efficiency, and the calculation of the mask representation can be reduced to only one layer. This further illustrates the importance of independently processing visual representations in the embodiments of the present application in achieving efficient image generation.

[0083] Table 1 shows the analysis results of the performance of the disentanglement model with different encoder and decoder layers.

[0084] Number of encoder layers Number of decoder layers GFLOPs FID 8 8 40.2 5.50 12 4 38.2 4.98 15 1 39.8 4.78

[0085] In the embodiment of the present application, in order to explain the new image representation sequence generated by the current iteration, the correlation between the new image representation sequences generated in the previous iteration is analyzed by analyzing the similarity of the new image representation sequences generated by two adjacent iterations (the familiarity of the new image representation sequences generated by two adjacent iterations is evaluated by the pre-similarity). It is found that the image representations at some "key positions" are quite different, while the image representations at other positions have a higher similarity; wherein, the "key position" refers to the position corresponding to the newly added visual representation. In other words, the significance of generating a new image representation sequence in each iteration is to update the newly decoded image representation, while the calculation of most other image representations is repeated.

[0086] The embodiment of the present application provides another image generation method based on the interaction of spatiotemporal data. Specifically, the image representations at the newly added visual representation positions in two adjacent new image representation sequences are quite different, and the image representations at other positions are similar. Therefore, the method only encodes the newly added visual representations and reuses the features of the previous image representation sequence to the maximum extent to supplement the necessary information. Figure 5 As shown, the method includes the following steps S510 to S540:

[0087] Step S510: Acquire the last image representation sequence; the image representation sequence includes a visual representation and a mask representation, the mask representation represents unknown image content, and the visual representation represents known image content, which is used to provide image information for the mask representation to infer the unknown image content.

[0088] Step S520: determining a newly added visual representation from the previous image representation sequence; encoding the newly added visual representation based on cross-self-attention interaction to obtain a newly added visual representation feature.

[0089] Step S530: concatenate the features of the previous image representation sequence with the newly added visual mark features to obtain concatenated representation features; and use the concatenated representation features to decode the mask representation based on cross-self-attention interaction to obtain a new image representation sequence.

[0090] Step S540: performing multiple iterations of image representation sequence generation according to the above steps, and generating an image according to the new image representation sequence when the new image representation sequence does not contain a mask representation.

[0091] In the embodiment of the present application, since the image representations at the newly added visual representation positions in the two adjacent new image representation sequences generated are quite different, and the image representations at the other positions are similar. Therefore, only the newly added visual representation is encoded to inject knowledge about the image, thereby obtaining the newly added visual representation features. Specifically, in step S520, "encoding the newly added visual representation based on cross-self-attention interaction" includes: using a lightweight projection network to process the features of the previous image representation sequence to obtain projection features, and using the projection features to encode the newly added visual representation based on cross-self-attention interaction, so that the newly added visual representation is integrated into the knowledge about the image in the previous step to obtain the newly added visual representation features.

[0092] Furthermore, in order to ensure the quality of the generated image, the features of the previous image representation sequence are directly reused to assist in the decoding of the mask representation of the current iteration. In some embodiments, the spliced ​​representation features may be projection features corresponding to the features of the previous image representation sequence, spliced ​​with newly added visual marker features. Specifically, in step S530, "using the spliced ​​representation features to decode the mask representation based on cross-self-attention interaction" specifically includes: generating a key representation and a value representation based on the spliced ​​representation features and the mask representation corresponding to the mask representation; using the mask representation features as a query representation; thereby performing attention interaction based on the key representation, the value representation and the query representation to obtain a new image representation sequence.

[0093] The image representation sequence is iteratively generated multiple times according to the above steps S510 to S530. As the image representation sequence is iteratively generated, the number of mask representations decoded in the image representation sequence gradually increases. When a new image representation sequence generated in a certain iteration does not contain a mask representation, it means that all the image information is already known. At this time, the iterative generation is terminated, and an image is generated according to the current new image representation sequence.

[0094] For example, the new image representation sequence generated according to the method of step S510 to step S530 can be expressed as:

[0095] z=Forward(v Δ ,v M ,f(z prev )),

[0096] Among them, z represents the characteristics of the new image representation sequence, Forward represents forward propagation, and v Δ represents the newly added image representation, v M represents the mask representation, z prev represents the features of the previous image representation sequence, and f(·) represents the lightweight projection module.

[0097] In the embodiment of the present application, the new image representation sequence is generated based on the disentanglement model, referring to Figure 6 As shown, Figure 6 It is a schematic diagram of an image representation sequence generation method provided in an embodiment of the present application. For the T-th image representation sequence generation, only the newly added visual representation of the T-1th time is encoded, and the features of the T-1th image representation sequence are reused to supplement the necessary information. Specifically, the features of the T-1th image representation sequence are processed by the lightweight projection module and input into the disentanglement model, and the T-1th image representation sequence is input into the disentanglement model, so that the disentanglement model generates a new image representation sequence according to the above-mentioned steps S510 to S530.

[0098] For example, Figure 7 It is a structural diagram of another disentanglement model provided by an embodiment of the present application. Disentanglement model encoder and decoder; in the encoder, the projection feature is used to encode the T-1th newly added visual representation based on cross-self-attention interaction, so that the newly added visual representation learns knowledge about the image and obtains the newly added visual representation feature; thereby, the newly added visual representation feature and the projection feature are spliced ​​to obtain the spliced ​​representation feature; and in the decoder, the mask representation is decoded based on the cross-self-attention interaction using the spliced ​​representation feature to obtain a new image representation sequence.

[0099] Through the above implementation process, in each generation process of the image representation sequence, only the newly added visual representation is encoded, the features of the previous image representation sequence are spliced ​​with the newly added visual marker features, and the mask representation is decoded based on the cross-self-attention interaction using the spliced ​​representation features. In this way, the previously calculated features can be reused to the maximum extent, supplementing the necessary information in the decoding and encoding process of the current iteration, greatly reducing the computational cost and achieving efficient image generation.

[0100] In an optional embodiment, the disentanglement model is trained according to the following steps A1 to A3:

[0101] Step A1: construct an image representation sequence set, wherein the image representation sequence set includes a plurality of image representation sequence samples.

[0102] Step A2: Randomly mask the image representations in the image representation sequence samples to obtain image representation sequence samples containing masked representations.

[0103] Step A3: determining a target training strategy according to the target probability, and training the disentanglement model according to the target training strategy to predict the image representation corresponding to the mask representation position based on the image representation sequence samples including the mask representation.

[0104] In the embodiment of the present application, the encoder Q and the quantizer ε can be used VQ A plurality of original sample images are converted into a series of image representation sequences, thereby obtaining a set of image representation sequences. For example, the original sample images are converted into image representation sequence samples which can be expressed as:

[0105] v = Q(ε vQ (x)),

[0106] Where v = [v i ] i＝1:N represents the image representation sequence sample, and N represents the length of the image representation sequence sample.

[0107] The image representations in the image representation sequence samples are randomly masked to obtain image representation sequence samples containing masked representations, which are used as input for training the disentanglement model, so that the disentanglement model can predict the image representation corresponding to the masked representation position based on the unmasked image representation.

[0108] In order to enable the disentanglement model to reuse the features of the previous image representation sequence and decode the mask representation, the disentanglement model is trained according to the target training strategy determined by the target probability. The target probability is set according to the actual situation. For example, in one example, the target probability is 50%.

[0109] In a specific implementation, the target training strategy includes a first training strategy; training the disentanglement model according to the first training strategy to predict the image representation corresponding to the mask representation position based on the image representation sequence sample containing the mask representation, including steps B1 to B3:

[0110] Step B1: taking the image representation sequence sample containing the mask representation as the current image representation sequence sample, and performing random masking on the image representation sequence sample containing the mask representation to simulate the previous image representation sequence sample.

[0111] Step B2: Compare the current image representation sequence sample with the previous image representation sequence sample to determine a newly added visual mark sample.

[0112] Step B3: splicing the features of the previous image representation sequence sample with the features of the newly added visual mark sample to obtain a spliced ​​representation feature sample.

[0113] Step B4: using the concatenated representation feature sample to decode the mask representation in the image representation sequence sample at the previous moment based on cross-self-attention interaction, so as to predict the image representation corresponding to the mask representation position.

[0114] In an embodiment of the present application, by simulating the previous image representation sequence samples so that the de-entanglement model can reuse the features of the previous image representation sequence samples, necessary information in the encoding process of the new image representation and the decoding process of the mask representation is supplemented, so that the trained de-entanglement model can reuse the previous calculation features to the maximum extent, thereby greatly reducing the calculation cost and achieving efficient image generation.

[0115] In a specific implementation, the target training strategy includes a second training strategy; the disentanglement model is trained according to the first training strategy to predict the image representation corresponding to the mask representation position based on the image representation sequence samples containing the mask representation, specifically including: with the goal of minimizing the negative log-likelihood loss of the mask representation, the mask representation is predicted using the visual representation in the image representation sequence samples, thereby obtaining the image representation corresponding to the mask representation position. In this way, the disentanglement model can quickly learn the ability to decode the mask representation based on the visual representation.

[0116] Through the above implementation process, the first training strategy and the second training strategy are used to train the disentanglement model with the target probability, so that the disentanglement model can have the ability to reuse the features of the previous image representation sequence and the ability to decode the mask representation. In this way, the image representation sequence can be efficiently generated based on the disentanglement model.

[0117] The present application embodiment also provides an image generation device based on the interaction of spatiotemporal data, as shown in FIG8 , Figure 8 : is a structural schematic diagram of an image generation device based on spatiotemporal data interaction provided by an embodiment of the present application, the device comprising:

[0118] An acquisition module 810 is used to acquire a previous image representation sequence; the image representation sequence includes a visual representation and a mask representation, the mask representation represents unknown image content, and the visual representation represents known image content, and is used to provide image information for the mask representation to infer the unknown image content;

[0119] An encoding module 820, configured to encode the visual representation based on self-attention interaction to obtain visual representation features;

[0120] A decoding module 830, configured to decode the mask representation based on cross-self-attention interaction according to the visual representation feature to obtain a new image representation sequence, wherein the new image representation sequence includes a newly added visual representation;

[0121] The generating module 840 is configured to iteratively generate the image representation sequence multiple times according to the above steps, and generate an image according to the new image representation sequence when the new image representation sequence does not contain a mask representation.

[0122] In an optional embodiment, the image representations at the newly added visual representation positions in two adjacent generated new image representation sequences are significantly different, and the image representations at other positions are similar; the visual representation features include the newly added visual representation features; and the encoding module includes:

[0123] A first determining module, configured to determine a newly added visual representation from the previous image representation sequence;

[0124] The first encoding submodule is used to encode the newly added visual representation based on cross-self-attention interaction to obtain the newly added visual representation features.

[0125] In an optional embodiment, the decoding module includes:

[0126] A first splicing module, used for splicing the features of the previous image representation sequence with the newly added visual mark features to obtain a spliced ​​representation feature;

[0127] The first decoding submodule is used to use the spliced ​​representation features to decode the mask representation based on cross-self-attention interaction to obtain a new image representation sequence.

[0128] In an optional embodiment, the decoding module includes:

[0129] A representation generation module, configured to generate a key representation and a value representation according to the visual representation feature and the mask representation feature corresponding to the mask representation;

[0130] A query representation module, used to use the mask representation feature as a query representation;

[0131] The interaction module is used to perform attention interaction according to the key representation, the value representation and the query representation to obtain a new image representation sequence.

[0132] In an optional embodiment, the new image representation sequence is generated based on a disentanglement model, the disentanglement model includes a multi-layer encoder and a multi-layer decoder, and the number of layers of the multi-layer encoder is greater than the number of layers of the multi-layer decoder;

[0133] The multi-layer encoder is used to encode the visual representation based on self-attention interaction, and the multi-layer decoder is used to decode the mask representation based on cross-self-attention interaction according to the visual representation features.

[0134] In an optional embodiment, the apparatus further includes a training module for training a disentanglement model, wherein the training module includes:

[0135] A construction module, used to construct an image representation sequence set, wherein the image representation sequence set includes a plurality of image representation sequence samples;

[0136] A first random masking module, configured to randomly mask the image representations in the image representation sequence samples to obtain image representation sequence samples containing masked representations;

[0137] A training prediction module is used to determine a target training strategy according to a target probability, and train the disentanglement model according to the target training strategy to predict the image representation corresponding to the mask representation position based on the image representation sequence samples containing the mask representation.

[0138] In an optional embodiment, the target training strategy includes a first training strategy; the training prediction module includes:

[0139] a second random mask module, configured to use the image representation sequence sample containing the mask representation as the current image representation sequence sample, and to perform random masking on the image representation sequence sample containing the mask representation to simulate the previous image representation sequence sample;

[0140] A second determination module, configured to compare the current image representation sequence sample with the previous image representation sequence sample to determine a newly added visual mark sample;

[0141] A second splicing module is used to splice the features of the previous image representation sequence sample with the features of the newly added visual mark sample to obtain a spliced ​​representation feature sample;

[0142] The training prediction submodule is used to use the spliced ​​representation feature sample to decode the mask representation in the image representation sequence sample at the previous moment based on cross-self-attention interaction to predict the image representation corresponding to the mask representation position.

[0143] The present application also provides an electronic device, referring to Fig. 9 As shown, Fig. 9 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig. 9 As shown, the electronic device 900 includes: a memory 910 and a processor 920. The memory 910 and the processor 920 are connected via a bus communication. A computer program is stored in the memory 910. The computer program can be run on the processor 920 to implement the steps of the image generation method based on spatiotemporal data interaction described in the embodiment of the present application.

[0144] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the image generation method based on spatiotemporal data interaction described in the embodiment of the present application are implemented.

[0145] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the image generation method based on spatiotemporal data interaction described in the embodiment of the present application.

[0146] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0147] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses and devices according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0148] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0149] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0150] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0151] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0152] The above is a detailed introduction to the image generation method, device and electronic device based on the interaction of spatiotemporal data provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image generation method based on the interaction of spatiotemporal data, characterized in that: include: Acquire the last image representation sequence; the image representation sequence includes a visual representation and a mask representation, the mask representation represents unknown image content, and the visual representation represents known image content, which is used to provide image information for the mask representation to infer the unknown image content; Encoding the visual representation based on self-attention interaction to obtain visual representation features; Decoding the mask representation based on cross-self-attention interaction according to the visual representation feature to obtain a new image representation sequence, wherein the new image representation sequence includes a newly added visual representation; The image representation sequence is iteratively generated multiple times according to the above steps, and when the new image representation sequence does not contain a mask representation, an image is generated according to the new image representation sequence.

2. The method according to claim 1, characterized in that In two adjacent new image representation sequences generated, the image representations at the newly added visual representation positions are quite different, and the image representations at the remaining positions are similar; The visual representation features include newly added visual representation features; The visual representation is encoded based on self-attention interaction to obtain visual representation features, including: Determining a newly added visual representation from the previous sequence of image representations; The newly added visual representation is encoded based on cross-self-attention interaction to obtain new visual representation features.

3. The method according to claim 2, characterized in that Decoding the mask representation based on cross-self-attention interaction according to the visual representation feature to obtain a new image representation sequence, including: Splicing the features of the previous image representation sequence with the newly added visual mark features to obtain a spliced ​​representation feature; The mask representation is decoded based on cross-self-attention interaction using the spliced ​​representation features to obtain a new image representation sequence.

4. The method according to claim 1, characterized in that: Using the visual representation features, the mask representation is decoded based on cross-self-attention interaction to obtain a new image representation sequence, including: Generate a key representation and a value representation according to the visual representation feature and the mask representation feature corresponding to the mask representation; Using the mask representation feature as a query representation; Attention interaction is performed according to the key representation, the value representation and the query representation to obtain a new image representation sequence.

5. The method according to any one of claims 1 to 4, characterized in that: The new image representation sequence is generated based on a disentanglement model, wherein the disentanglement model includes a multi-layer encoder and a multi-layer decoder, and the number of layers of the multi-layer encoder is greater than the number of layers of the multi-layer decoder; The multi-layer encoder is used to encode the visual representation based on self-attention interaction, and the multi-layer decoder is used to decode the mask representation based on cross-self-attention interaction according to the visual representation features.

6. The method according to claim 5, characterized in that The disentanglement model is trained according to the following steps: Constructing an image representation sequence set, wherein the image representation sequence set includes a plurality of image representation sequence samples; Randomly masking the image representations in the image representation sequence samples to obtain image representation sequence samples containing masked representations; A target training strategy is determined according to the target probability, and the disentanglement model is trained according to the target training strategy to predict the image representation corresponding to the mask representation position based on the image representation sequence samples including the mask representation.

7. The method according to claim 6, characterized in that The target training strategy includes a first training strategy; training the disentanglement model according to the first training strategy to predict the image representation corresponding to the mask representation position based on the image representation sequence samples containing the mask representation, including: The image representation sequence sample including the mask representation is used as the current image representation sequence sample, and the image representation sequence sample including the mask representation is randomly masked to simulate the previous image representation sequence sample; Compare the current image representation sequence sample with the previous image representation sequence sample to determine a newly added visual mark sample; Splicing the features of the previous image representation sequence sample with the features of the newly added visual mark sample to obtain a spliced ​​representation feature sample; The concatenated representation feature sample is used to decode the mask representation in the image representation sequence sample at the previous moment based on cross-self-attention interaction to predict the image representation corresponding to the mask representation position.

8. An image generation device based on the interaction of spatiotemporal data, characterized in that: include: An acquisition module, used to acquire a previous image representation sequence; the image representation sequence includes a visual representation and a mask representation, the mask representation represents unknown image content, and the visual representation represents known image content, and is used to provide image information for the mask representation to infer the unknown image content; An encoding module, used for encoding the visual representation based on self-attention interaction to obtain visual representation features; A decoding module, configured to decode the mask representation based on cross-self-attention interaction according to the visual representation feature to obtain a new image representation sequence, wherein the new image representation sequence includes a newly added visual representation; A generating module is used to iteratively generate multiple image representation sequences according to the above steps, and to generate an image according to the new image representation sequence when the new image representation sequence does not contain a mask representation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the image generation method based on the interaction of spatiotemporal data described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image generation method based on the interaction of spatiotemporal data described in any one of claims 1 to 7 are implemented.

Images