Content generation method and device, electronic equipment and storage medium

Abstract

Embodiments of the invention disclose a content generation method and apparatus, an electronic device and a storage medium. The method comprises the steps of firstly obtaining first input content of a first mode and second input content of a second mode; encoding the first input content to obtain a first target feature, encoding the second input content to obtain a second target feature, and inserting the second target feature into the first target feature to obtain a fused target feature, thereby realizing feature fusion of the input contents of different modalities; then, performing noise addition on the fused target features by adopting a mode of masking coding marks with a preset proportion in the fused target features to obtain noisy target features; and inputting the noisy target feature into the diffusion model for denoising based on bidirectional attention to obtain a denoised target feature, thereby realizing global information interaction, and therefore, when the first target content of the first modal and the second target content of the second modal are generated based on the denoised target feature subsequently, diversified target contents can be accurately generated.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a content generation method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence technology, the task of using multimodal input for content generation has become more and more common. In related technologies, a multimodal large language model can be used to perform content generation tasks based on multimodal input. During the reasoning process, the multimodal large language model generally combines the coding tags corresponding to different modalities for content generation. However, in complex reasoning tasks, how to effectively combine inputs of different modalities remains a challenge. The coding tags corresponding to modalities other than the current text modality are still subject to the unidirectional restriction of the causal attention mask in the multimodal large language model, which makes the accuracy of content generation need to be improved. Summary of the invention

[0003] The following is a summary of the subject matter of the detailed description of the present disclosure. This summary is not intended to limit the scope of the claims.

[0004] The embodiments of the present disclosure provide a content generation method, device, electronic device, and storage medium, which can improve the accuracy of content generation.

[0005] In one aspect, an embodiment of the present disclosure provides a content generation method, comprising:

[0006] Obtaining first input content of a first modality and second input content of a second modality;

[0007] Encoding the first input content to obtain a first target feature, encoding the second input content to obtain a second target feature, inserting the second target feature into the first target feature to obtain a fused target feature, wherein the fused target feature includes a plurality of encoding tags;

[0008] By masking a preset proportion of the coding marks in the fused target feature, the fused target feature is denoised to obtain a noisy target feature;

[0009] The noisy target features are input into a diffusion model for denoising based on bidirectional attention to obtain denoised target features, and the first target content of the first modality and the second target content of the second modality are generated based on the denoised target features.

[0010] On the other hand, the present disclosure also provides a content generation device, including:

[0011] A content acquisition module, used to acquire first input content of a first modality and second input content of a second modality;

[0012] an encoding module, configured to encode the first input content to obtain a first target feature, encode the second input content to obtain a second target feature, and insert the second target feature into the first target feature to obtain a fused target feature, wherein the fused target feature includes a plurality of encoding tags;

[0013] A noise adding module, configured to add noise to the fused target feature by masking a preset proportion of the coding marks in the fused target feature, so as to obtain a noisy target feature;

[0014] A denoising module is used to input the noisy target features into a diffusion model for denoising based on bidirectional attention to obtain denoised target features, and generate first target content of the first modality and second target content of the second modality based on the denoised target features.

[0015] Furthermore, the denoising module is specifically used for:

[0016] Inputting the noisy target features into the diffusion model for denoising based on bidirectional attention to obtain the denoising result of the current time step;

[0017] The denoising result of the current time step is input into the diffusion model again for denoising based on bidirectional attention until the current time step reaches the preset time step to obtain the denoising target feature.

[0018] Furthermore, the denoising module is specifically used for:

[0019] Inputting the noisy target features into a diffusion model;

[0020] Determine the query vector, key vector and value vector of each of the coding tags, traverse the multiple coding tags, and obtain the attention score between the current coding tag and each of the remaining coding tags according to the dot product between the query vector of the current coding tag and the key vectors of each of the remaining coding tags;

[0021] Traversing the plurality of coding tags, and performing weighted summation on the value vectors of the remaining coding tags according to the attention scores of the current coding tag, to obtain the attention vector of the current coding tag;

[0022] The attention vectors of the multiple encoded labels are used as the denoising result of the current time step obtained after denoising based on bidirectional attention.

[0023] Furthermore, the denoising module is specifically used for:

[0024] Obtaining a query weight matrix, a key weight matrix and a value weight matrix for each of the encoding tags;

[0025] For any one of the coding tags, the query vector of the coding tag is obtained according to the product of the coding tag and the query weight matrix, the key vector of the coding tag is obtained according to the product of the coding tag and the key weight matrix, and the value vector of the coding tag is obtained according to the product of the coding tag and the value weight matrix.

[0026] Furthermore, the noise adding module is specifically used for:

[0027] Generate a random number, wherein the random number is a positive integer less than the preset time step;

[0028] The preset ratio is determined according to a ratio between the random number and the preset time step.

[0029] Furthermore, the noise adding module is specifically used for:

[0030] Inputting the fused target features into the target model to predict the position to be masked, and obtaining a number of position numbers corresponding to the preset ratio, wherein the target model is obtained by joint training with the diffusion model;

[0031] The fused target feature is denoised by masking the coding mark of the position number in the fused target feature to obtain a noisy target feature.

[0032] Furthermore, the noise adding module is specifically used for:

[0033] Inputting the fused target features into a target model;

[0034] Creating a position coding feature for the fusion target feature, wherein the position coding feature is used to indicate position information of a plurality of the coding marks in the fusion target feature;

[0035] Splicing the fusion target feature and the position coding feature to obtain a splicing feature;

[0036] A self-attention mechanism is used to encode the concatenated features to obtain a self-attention feature;

[0037] Classification is performed based on the self-attention features to predict the positions to be masked, and a classification result is obtained. Based on the classification result, a number of position numbers corresponding to the preset ratio is determined, wherein the classification result is used to indicate the probability of each of the coding marks being masked.

[0038] Furthermore, the above encoding module is specifically used for:

[0039] Adding a start tag and an end tag for indicating the second mode at both ends of the second target feature respectively;

[0040] The second target feature after adding the start tag and the end tag is inserted into the first target feature to obtain a fused target feature.

[0041] Furthermore, the above encoding module is specifically used for:

[0042] Mapping each of the coded marks in the first target feature to a continuous space;

[0043] The second target feature is inserted into the mapped first target feature to obtain a fused target feature.

[0044] Furthermore, the content acquisition module is also used for:

[0045] Obtain first sample content of the first modality and second sample content of the second modality.

[0046] The above encoding module is also used for:

[0047] Encoding the first sample content to obtain a first sample feature, and encoding the second sample content to obtain a second sample feature;

[0048] The above noise adding module is also used for:

[0049] Inserting the second sample feature into the first sample feature and then adding noise to obtain a noisy sample feature;

[0050] The above denoising module is also used to:

[0051] Calling the diffusion model to obtain denoised sample features based on the noisy sample features, wherein the denoised sample features include the text probability distribution corresponding to the first modality and the denoised sub-features corresponding to the second modality;

[0052] Acquire a first label content of the first modality and a second label content of the second modality, encode the first label content to obtain a first label feature, and encode the second label content to obtain a second label feature;

[0053] Mapping the text probability distribution to a continuous space, determining a first mean square error loss based on the mapped text probability distribution and the first label feature, and determining a second mean square error loss based on the denoising sub-feature and the second label feature;

[0054] The first mean square error loss and the second mean square error loss are weightedly summed to obtain a target loss, and parameters of the diffusion model are adjusted based on the target loss.

[0055] Furthermore, the denoising module is specifically used for:

[0056] Normalizing the text probability distribution;

[0057] The first sample features are weighted averaged according to the normalized text probability distribution to map the text probability distribution to a continuous space.

[0058] On the other hand, an embodiment of the present disclosure further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned content generation method when executing the computer program.

[0059] On the other hand, an embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the above-mentioned content generation method.

[0060] On the other hand, the embodiment of the present disclosure further provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned content generation method.

[0061] The disclosed embodiment includes at least the following beneficial effects: by obtaining the first input content of the first modality and the second input content of the second modality, the first input content is encoded to obtain the first target feature, the second input content is encoded to obtain the second target feature, the second target feature is inserted into the first target feature to obtain the fused target feature, thereby realizing the feature fusion of the input content of different modalities, and then the fused target feature is denoised by masking the coding mark of a preset proportion in the fused target feature to obtain the noisy target feature. Through this denoising method, discrete noise can be uniformly applied to the coding marks corresponding to different modalities, which is convenient for subsequent unified modeling of multimodal through the diffusion model. On this basis, by inputting the noisy target feature into the diffusion model for denoising based on bidirectional attention, the denoised target feature is obtained. In the denoising process, the information carried by all the coding marks in the noisy target feature can be introduced based on bidirectional attention, thereby realizing global information interaction. Therefore, when the first target content of the first modality and the second target content of the second modality are subsequently generated based on the denoised target feature, the accuracy of content generation can be improved while generating diversified target content.

[0062] Other features and advantages of the present disclosure will be set forth in the following description, and in part will be apparent from the description, or may be learned by practicing the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.

[0064] Figure 1 A schematic diagram of an optional implementation environment provided for an embodiment of the present disclosure;

[0065] Figure 2 A schematic diagram of an optional flow chart of a content generation method provided in an embodiment of the present disclosure;

[0066] Figure 3 A schematic diagram of an optional process of inserting a second target feature into a first target feature provided by an embodiment of the present disclosure;

[0067] Figure 4 A schematic diagram of an optional process of inserting a second target feature into a first target feature provided by an embodiment of the present disclosure;

[0068] Figure 5 A schematic diagram of an optional process of mapping text modality data to a continuous space provided in an embodiment of the present disclosure;

[0069] Figure 6 A schematic diagram of an optional process of fusion target feature noise processing provided in an embodiment of the present disclosure;

[0070] Figure 7 A schematic diagram of an optional process of fusion target feature noise processing provided in an embodiment of the present disclosure;

[0071] Figure 8 Another optional flow chart of multimodal data mixing processing provided in the embodiment of the present disclosure;

[0072] Fig. 9 An optional flowchart of text modality data processing provided by an embodiment of the present disclosure;

[0073] Fig.10 An optional flow chart of a training diffusion model provided in an embodiment of the present disclosure;

[0074] Fig.11 A schematic diagram of an optional flow chart of a content generation method provided in an embodiment of the present disclosure;

[0075] Fig.12A schematic diagram of an optional structure of a content generation device provided in an embodiment of the present disclosure;

[0076] Fig.13 A partial structural block diagram of a terminal provided in an embodiment of the present disclosure;

[0077] Fig.14 A partial structural block diagram of a server provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not used to limit the present disclosure.

[0079] It should be noted that in various specific embodiments of the present disclosure, when it comes to the need to perform relevant processing based on data related to the characteristics of the target object such as the target object attribute information or attribute information set, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. Among them, the target object can be a user. In addition, when the embodiment of the present disclosure needs to obtain the attribute information of the target object, the separate permission or separate consent of the target object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the target object, the necessary target object-related data used to enable the normal operation of the embodiment of the present disclosure will be obtained.

[0080] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0081] To facilitate understanding of the technical solution provided by the embodiments of the present disclosure, some key terms used in the embodiments of the present disclosure are explained here:

[0082] Masking is a data processing technique that is usually used to block out certain specific parts of the data to ensure that the model does not process invalid or irrelevant inputs during training or inference. A mask is usually an array or matrix with the same size as the original data to be processed. When using a mask, only the corresponding elements in the mask array with a value of 1 (or True) are processed, and elements with a value of 0 (or False) are ignored.

[0083] Bi-directional Attention is a technology used in natural language processing and image description tasks. The bi-directional attention mechanism combines forward and backward contextual information to capture more comprehensive semantic dependencies. Compared with the unidirectional attention mechanism, the bi-directional attention mechanism can better capture distance dependencies and help improve model performance.

[0084] Self-Attention is a mechanism that allows the input sequence to decide which parts are worth paying attention to. The self-attention mechanism does not rely on any external information, but calculates the similarity between the elements in the input sequence and assigns a weight to each element to determine which parts are more important for the current task.

[0085] The causal attention mask is a mechanism used to limit the flow of information in model training, preventing the model from obtaining future information when predicting. That is, it ensures that the model can only look forward in the time series, cannot see future information, and cannot obtain ungenerated text information on the right. This unidirectional attention calculation ensures the autoregressive nature of the generation process.

[0086] With the development of artificial intelligence technology, the task of using multimodal input for content generation has become more and more common. In the related art, a multimodal large language model can be used to perform content generation tasks based on multimodal input. In the reasoning process, the multimodal large language model generally combines the coding tags corresponding to different modalities for content generation. However, in complex reasoning tasks, how to effectively combine inputs of different modalities is still a challenge. At present, the model architecture for multimodal data processing is mainly divided into three types. The first is based on a large language model (LLM). Through a visual encoder, such as the CLIP model (Contrastive Language-Image Pre-Training), the image information is mapped to the large language model, so that the large language model can simultaneously process text and image multimodal information, but the large language model cannot directly generate the corresponding image, and it is necessary to rely on other methods to generate images. The second method is a technology that mainly generates images. Through mixed alignment training of text and image, text-to-image is realized, focusing on converting text information into images, but this method is limited to input text information. The third method is to mix text and images and generate results. For example, the Transfusion model, a new method that combines the Transformer model and the Diffusion Model, encodes the image input through the Variational AutoEncoder (VAE) and the U-shaped convolutional neural network to obtain an implicit vector representation, which is then input into the Transformer model together with the text input. The text content is generated by autoregression, while the image content is generated by diffusion model training. Obviously, the first visual language model based on the large language model, or the model based on text and images, focuses on the data generation of a single modality and cannot effectively generate accurate multimodal content; while the third model that mixes text and images, although it fuses multiple modal inputs, still cannot overcome the unidirectional information dependence generated by autoregression. Therefore, when dealing with multimodal content generation tasks, the encoding tags corresponding to other modalities other than the current text modality are still subject to the unidirectional restriction of the causal attention mask in the multimodal large language model, which makes the accuracy of content generation need to be improved.

[0087] Based on this, the embodiments of the present disclosure provide a content generation method, device, electronic device and storage medium, which can improve the accuracy of content generation.

[0088] Reference Figure 1 , Figure 1A schematic diagram of an optional implementation environment provided for an embodiment of the present disclosure, the implementation environment includes a terminal 101 and a server 102, wherein the terminal 101 and the server 102 are connected via a communication network.

[0089] The terminal 101 may be a mobile phone, a computer, an intelligent voice interaction device, an intelligent wearable device, an intelligent home appliance, a vehicle-mounted terminal, etc., but is not limited thereto. The terminal 101 and the server 102 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present disclosure.

[0090] Exemplarily, the terminal 101 may be provided with a module for acquiring data content in multiple different modalities. For example, the data content in different modalities may be image content, audio content, video content or text content, etc. Specifically, the terminal 101 may be provided with a camera component, through which image content or video content can be collected, or the terminal 101 may also be provided with an input unit, through which text content such as input numbers or character information is received.

[0091] Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In addition, server 102 can also be a node server in a blockchain network.

[0092] Exemplarily, the server 102 may obtain input content of different modes sent by the terminal 101, wherein the input content includes first input content of a first mode and second input content of a second mode, for example, the input content includes first input content of a text type and second input content of an image type; the server 102 may encode the first input content to obtain a first target feature, and encode the second input content to obtain a second target feature, and then insert the second target feature into the first target feature to obtain a fused target feature, wherein, since both the first input content and the second input content are obtained by encoding, the first input content and the second input content both include the encoding mark Note that, therefore, the fused target feature also includes multiple coding tags; then, the fused target feature is denoised by masking a preset proportion of the coding tags in the fused target feature, so that a noisy target feature can be obtained, and then the noisy target feature is input into the diffusion model for denoising based on bidirectional attention to obtain a denoised target feature, and a first target content of the first modality and a second target content of the second modality are generated based on the denoised target feature; after generating the target content of the corresponding modality, the generated target content, i.e., the first target content of the first modality and the second target content of the second modality, can be returned to the terminal 101 for the terminal 101 to perform subsequent content processing.

[0093] The server 102 can obtain the first input content of the first modality and the second input content of the second modality, encode the first input content to obtain the first target feature, encode the second input content to obtain the second target feature, insert the second target feature into the first target feature to obtain the fused target feature, thereby realizing the feature fusion of the input content of different modalities, and then adopt the method of masking the coding mark of a preset proportion in the fused target feature to add noise to the fused target feature to obtain the noisy target feature. Through this noisy method, discrete noise can be uniformly applied to the coding marks corresponding to different modalities, which is convenient for subsequent unified modeling of multimodal through the diffusion model. On this basis, by inputting the noisy target feature into the diffusion model for denoising based on bidirectional attention, the denoised target feature is obtained. The information carried by all the coding marks in the noisy target feature can be introduced based on bidirectional attention during the denoising process, thereby realizing global information interaction. Therefore, when the first target content of the first modality and the second target content of the second modality are subsequently generated based on the denoised target feature, the accuracy of content generation can be improved while generating diversified target content.

[0094] The method provided in the embodiments of the present disclosure can be applied to different scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, data analysis, visual intelligence and other scenarios.

[0095] Reference Figure 2 , Figure 2An optional flow chart of a content generation method provided in an embodiment of the present disclosure. The content generation method can be executed by a server, or by a terminal, or by a server in cooperation with a terminal. The content generation method includes but is not limited to the following steps 201 to 204.

[0096] Step 201: Obtain first input content of a first modality and second input content of a second modality.

[0097] In a possible implementation, in the field of multimedia and computer science, "modality" can refer to the presentation method or type of information or data. For example, text, image, audio, video, etc. are all different modalities, and each modality has its specific representation method and application scenario. The input content corresponding to each modality can be the content that users enter through the user interface. For example, in an application or website, a text box can be provided for users to enter text, and a file upload button can be provided for users to upload images, etc.; or, text and image data can be obtained from public datasets; or, the application programming interface (API interface) provided by online services can be used to obtain the corresponding text and image data (such as obtaining article content and accompanying pictures from news websites).

[0098] In a possible implementation, the first modality and the second modality are two different modalities, and the first input content of the first modality may refer to content containing discrete elements, or refer to information composed of discrete elements. For example, the first modality may refer to a text modality, that is, information is presented in the form of text, and therefore, the first input content of the first modality may refer to input text information, and the first input content may be any form of text, such as a sentence, paragraph, article, conversation, etc. The second input content of the second modality may refer to content containing continuous elements, or refer to information composed of continuous elements. For example, the second modality may refer to an image modality, that is, information is presented in the form of an image, and therefore, the second input content of the second modality may refer to input or selected image information, which may be any form of image, such as a photo, icon, drawing, scan, etc. It should be noted that the second modality may also refer to a video modality, an audio modality, etc.

[0099] In a possible implementation, the content generation method can be applied to a visual agent scene, giving the visual agent the ability to understand and respond to the environment according to text instructions. The visual agent can receive and understand the text instructions from the user, and can also capture and analyze the image information of its own environment in real time. When the user issues an instruction to the visual agent in the form of text, this constitutes the first input content of the first modality. For example, the user can send an instruction: "Please find the salt shaker in the kitchen and tell me the location of the salt shaker." In response to this instruction, the visual agent can use the camera set by itself to obtain and process the image of its own environment in real time, and these images are used as the second input content of the second modality. Next, the visual agent can use the content generation method provided by the embodiment of the present disclosure to fuse and analyze the text instructions with the image information, accurately identify the salt shaker in the kitchen, and determine the specific location of the salt shaker, and generate corresponding content results (such as text description, image annotation or voice feedback) to present to the user. For example, the visual agent can present an image in the kitchen to the user and use text information to feedback to the user that "the salt shaker is on the shelf in the kitchen."

[0100] Step 202: Encode the first input content to obtain a first target feature, encode the second input content to obtain a second target feature, and insert the second target feature into the first target feature to obtain a fused target feature.

[0101] In a possible implementation, since the first input content and the second input content belong to different modalities, the first input content and the second input content have different structures and characteristics, and direct fusion may cause information loss or inconsistency. In order to fuse the two to form multimodal features, the first input content and the second input content need to be converted into the same feature space or representation space. Therefore, by encoding the first input content and the second input content respectively, the first target feature and the second target feature in the same space are obtained, and then the multimodal data is fused by inserting the second target feature into the first target feature to obtain the fused target feature.

[0102] In a possible implementation, when the first modality is a text modality, a text encoder may be used to encode the first input content, and the first target feature obtained after encoding may be a vector or a sequence of vectors, which may refer to coding tags, and these coding tags capture the voice information in the text. Similarly, when the second modality is an image modality, an image encoder may be used to encode the second input content, and the second target feature obtained after encoding may also be a vector or a sequence of vectors, which may also refer to coding tags, and these coding tags represent the visual features in the image. Since the features of text and image have different dimensions or structures, the first target feature and the second target feature may be aligned, and the alignment may be achieved by various methods, such as dimensional transformation (using a fully connected layer or a projection layer to map the features to the same dimension), feature fusion technology (such as concatenation, weighted sum, etc.), or an alignment algorithm using an attention mechanism. After the first target feature and the second target feature are in the same space, since the second target feature is an image modality and contains continuous elements, the second target feature may be inserted into the first target feature to protect the accuracy of the information of the continuous elements. It is worth noting that the specific encoding method may vary according to the application scenario and task requirements. In practical applications, different encoders may need to be tried and optimized to obtain the best output results.

[0103] In a possible implementation, when the first mode is a text mode, the first input content may be encoded using a Tokenizer text encoding tool, wherein Tokenizer is a basic tool in text processing, and can divide a continuous text string in the first input content into discrete Token units, i.e., encoding tags. Specifically, the first input content may be preprocessed, including removing noise (such as tags, special characters), format conversion (converting uppercase and lowercase), and possible stem extraction or word form restoration. According to the language characteristics, different word segmentation strategies can be selected, for example, a rule-based word segmentation strategy relies on preset rules, such as regular expressions, to identify and segment text, or a statistical-based word segmentation strategy uses statistical methods to count the frequency of character pairs in the text, gradually merges the most common character pairs, and forms new tags until a predetermined vocabulary size or coverage is reached, or a deep learning-based word segmentation strategy uses a neural network to learn the way of text segmentation and capture complex language features. Therefore, for languages ​​with obvious separations such as English, space segmentation, that is, a rule-based word segmentation strategy, can be used, and for languages ​​without obvious word separations such as Chinese, a statistical or deep learning-based word segmentation strategy can be used; then, based on the word segmentation results, a vocabulary is constructed to map each encoded tag to a unique integer identifier. Therefore, according to the vocabulary, the first input content is converted into a corresponding vocabulary serial number, that is, an integer tag, to form a numerical sequence, that is, the words in the first input content are mapped to the serial numbers in the vocabulary, and the resulting numerical sequence can be used as the first target feature.

[0104] In one possible implementation, when the second modality is an image modality, a variational autoencoder (VAE) can be used to encode the second input content. Specifically, the second input content, i.e., the picture, is cut into multiple small tiles, and then the cut small tiles are normalized. Then, the normalized small tiles are input into the variational autoencoder for processing, wherein the variational autoencoder includes an encoder, a latent layer, and a decoder. The encoder can map the input small tiles to the distribution parameters of the latent space. The encoder is composed of multiple fully connected layers or convolutional layers, and outputs parameters such as the mean and variance of the latent distribution. These parameters are used to define the distribution of the latent space; the latent layer can represent the distribution of the input data in the latent space. By sampling the latent layer, new samples can be generated. The distribution of the latent layer will gradually approach the given one as the training process continues. Prior distribution; the decoder maps the samples sampled from the latent layer back to the original input space. The decoder is also composed of multiple fully connected layers or convolutional layers to reconstruct the input small tiles. Therefore, after the tiles are input into the variational autoencoder for processing, the encoder in the variational autoencoder can perform feature extraction and dimensionality reduction on the input small tiles through a multi-layer neural network, and output parameters such as the mean and variance of the latent distribution to define a distribution in the latent space. Then, a latent vector is sampled from the distribution of the latent space. This latent vector is a point in the latent space, which contains the main feature information of the input small tile, and the second target feature can be obtained. The decoder can map this latent vector back to the original input space and reconstruct the small tile to train the variational autoencoder by comparing the difference between the reconstructed small tile and the original input small tile.

[0105] Based on this, the first target feature and the second target feature obtained after encoding both include multiple encoding tags, and the encoding tag can refer to the smallest unit in the target feature, for example, it can refer to the sequence element obtained after the text is processed by word segmentation, or it can refer to the latent vector represented by the small block obtained after the image is segmented. Therefore, the second target feature can be directly inserted into the first target feature to obtain the fused target feature, so as to realize the fusion of data of different modalities. At this time, the obtained fused target feature can refer to a sequence that fuses multimodal information, and the fused target feature includes multiple encoding tags.

[0106] In a possible implementation, there are various ways to directly insert the second target feature into the first target feature. For example, the second target feature can be spliced ​​together as an additional part of the first target feature, such as Figure 3As shown, when the first input content has multiple first target features, multiple corresponding first target features can be generated by encoding, that is, the first text generates the first text target feature corresponding to the first text by encoding, and the second text generates the second text target feature corresponding to the second text by encoding, and the second input content generates the second target feature corresponding to the second text by encoding, which can be inserted between the multiple first target features. For example, the first video of the second input content generates the first video target feature corresponding to the first video by encoding, and the first text target feature, the first video target feature and the second text target feature are spliced ​​in sequence, that is, the second target feature is inserted into the first target feature to obtain the fused target feature. Alternatively, the second target feature may be inserted into a specific position in the first target feature to obtain a fused target feature, wherein the inserted specific position may be a midpoint position in the first target feature or a position of a fixed proportion; alternatively, the first input content and the second input content may be sorted according to the acquisition time of both, the coding marks in the first target feature corresponding to the first input content may be sorted according to the timestamp, and the second target feature corresponding to the second input content may be inserted into the corresponding position according to the acquisition time of the second input content; alternatively, the second target feature may be inserted into a random position of the first target feature by generating a random number, determining the coding mark corresponding to the random number in the first target feature, and then inserting the second target feature into the corresponding position.

[0107] In a possible implementation, a start tag and an end tag for indicating the second modality are respectively added to both ends of the second target feature; the second target feature after the start tag and the end tag are added is inserted into the first target feature to obtain a fused target feature.

[0108] Among them, the Beginning of Image (BOI) can be used to indicate the beginning of the sequence of the second target feature, to prompt the model that the following data belongs to the second target feature of the second modality, rather than the data of the first modality, which helps to clearly divide the data boundaries of different modalities during the fusion analysis process. The End of Image (EOI) can be used to indicate the end of the sequence of the second target feature, to prompt the model that the second target feature belonging to the second modality ends here, and the subsequent data no longer belongs to the features of the second modality. By using the start marker BOI and the end marker EOI in combination, the scope of different modalities can be clearly defined, which can help the model accurately identify and process multimodal feature sequences.

[0109] In one possible implementation, Figure 4As shown, the third text of the first input content is encoded to generate a third text target feature, wherein the third text target feature is composed of multiple encoding tags, and the second target feature can be inserted between the multiple encoding tags in the first target feature, that is, the second video of the second input content is encoded to generate the second video target feature corresponding to the second video, which can be inserted between the encoding tag X and the encoding tag Y in the third text target feature. At this time, first, a start tag BOI is added to the starting position of the sequence of the second video target feature, and at the same time, an end tag EOI is added to the end position of the sequence of the second video target feature, and then, the second video target feature containing the start tag BOI and the end tag EOI is inserted between the encoding tag X and the encoding tag Y in the third text target feature, so as to insert the second target feature into the first target feature and obtain the fused target feature.

[0110] In a possible implementation, when the first modality is a text modality, the coding mark in the first target feature can be a serial number in the vocabulary, that is, an integer mark. At this time, the first target feature is presented as a discrete sequence, and the second modality is an image modality. The coding mark in the second target feature is composed of pixels. These pixel values ​​are usually continuous real numbers, representing the brightness, color and other features of the image. Therefore, the second target feature is presented as a continuous sequence. If the second target feature (continuous sequence) of the image modality is directly inserted into the first target feature (discrete sequence) of the text modality, there will be a data type mismatch problem. Therefore, it is necessary to map each coding mark in the first target feature to a continuous space, that is, to be in the same continuous space as each coding mark in the second target feature, and then the second target feature can be inserted into the mapped first target feature to obtain a fused target feature.

[0111] Among them, the first target feature can be mapped to the continuous space by means of an embedding matrix. The embedding matrix is ​​a technology for mapping input data to a high-dimensional vector space. Specifically, the embedding matrix can map discrete input data to a continuous high-dimensional vector space. This process can be regarded as mapping each discrete input to a vector. After the first input content of the text modality is encoded into a vocabulary number, the embedding matrix can be used to map these integer numbers to a continuous vector space. Specifically, the vocabulary used to encode the first input content is obtained. Each word or character in this vocabulary has a unique serial number, that is, an integer index, and the embedding matrix is ​​a two-dimensional array, in which rows represent different words or characters, and columns represent the dimension of the vector space. For a given serial number, the corresponding vector representation can be obtained by looking up the corresponding row of the embedding matrix. For example Figure 5As shown, the first input content is a simple sentence as an example: "Go to the shelf." Next, a vocabulary is established, and a unique serial number, i.e., an integer tag, is assigned to each word in the first input content. In this vocabulary, each word has a corresponding serial number, for example, "Go" corresponds to serial number 0, "to" corresponds to serial number 1, "the" corresponds to serial number 2, "shelf" corresponds to serial number 3, and "." corresponds to serial number 4. Then, assuming that the second target feature is in a three-dimensional vector space, the generated embedding matrix is ​​also a corresponding three-dimensional array. The rows in the embedding matrix represent different words, which can be represented by serial numbers (integer tags), and the columns represent the dimensions of the vector space. At this time, the sentence "Go to the shelf." of the first input content can be converted into a sequence consisting of serial numbers in the vocabulary: [0, 1, 2, 3, 4]. Then, by looking up the corresponding row of the embedding matrix, the vector representation of each word or character can be obtained. The vector representation of each word or character is combined to form the vector representation of the entire sentence, and the first target feature mapped to the continuous space is obtained. This can be achieved by simply concatenating the vector representation of each word or character, or by using complex combination methods such as averaging and weighted averaging. Figure 5 As shown, the vector representation of each word or character is simply concatenated to obtain the first target feature mapped to the continuous space.

[0112] It is worth noting that after obtaining the fusion target feature, the fusion target feature can be input into the Transformer model and processed by the bidirectional attention mechanism. The bidirectional attention mechanism used in the Transformer model can take into account all the information before and after the position at the same time, capture the global context information, and better understand the overall structure of the fusion target feature, learn the interaction between the input different modal features, and then generate the corresponding vector as the output, that is, the Transformer model can perform hybrid encoding based on the bidirectional attention mechanism on the fusion target feature, that is, the sequence of the input multimodal continuous vector, and output the corresponding vector, and use these output vectors as new fusion target features. Therefore, the vector output after the hybrid encoding of the Transformer model is used as the input of the subsequent processing steps, such as the denoising process of the denoising process of the diffusion model. Therefore, after the fusion target feature is input into the Transformer model and processed by the bidirectional attention mechanism, a high-level feature representation after deep feature extraction and sequence modeling is obtained, and the bidirectional attention encoding can avoid the shortcomings of the lack of global information in the autoregressive generation model, which helps the model generate accurate and coherent output content.

[0113] Therefore, the content generation method proposed in the embodiment of the present disclosure realizes the simultaneous modeling and encoding of different modalities (such as text modality and image modality). By adopting the Transformer model architecture, it can rely on the scalability of the large language model and use the bidirectional attention mechanism to perform hybrid encoding of multi-modal fusion information, effectively avoiding the defects of the autoregressive model.

[0114] Step 203: masking a preset proportion of coding marks in the fused target feature to add noise to the fused target feature to obtain a noisy target feature.

[0115] In one possible implementation, in the process of fusing multimodal features, the fused target features need to be denoised to enhance the robustness of the model or simulate noise interference in real scenes. By masking a preset proportion of the coding marks in the fused target features and denoising the fused target features, the causal attention masking method of the self-attention mechanism used in related technologies can be avoided, and the limitation of one-way information acquisition can be broken through, so that a two-way attention mechanism can be used in model processing to improve the accuracy of content generation.

[0116] In one possible implementation, the preset ratio can be a fixed ratio (such as a fixed 20%, 30%, etc.), or a random ratio, or a ratio value set according to task requirements or model performance. For example, it is set according to the number of coding marks in the fusion target feature. This ratio determines how many coding marks will be masked. According to this preset ratio, a mask matrix with the same dimension as the fusion target feature is randomly generated. The elements in this mask matrix can be binary (0 or 1), where 1 indicates that the feature at the corresponding position will be masked (i.e., noise is added), and 0 indicates that it remains unchanged, and the ratio of elements with a value of 1 in this mask matrix is ​​equal to the preset ratio. Then, the generated mask matrix is ​​applied to the fusion target feature. For the position with a value of 1 in the mask matrix, the corresponding coding mark is replaced with random discrete noise (such as Gaussian noise, uniform noise, etc.) or a specific value (such as 0, average value, etc.), or noise marks can be added to these corresponding coding marks. At this time, these coding marks containing noise marks are in an absorbing mask state, indicating that the model understands the coding mark as random noise and does not contain any information content, and the feature after mask processing is the noisy target feature.

[0117] In one possible implementation, Figure 6As shown in the figure, the preset ratio is 25%, which means that 25% of the coding marks in the fusion target feature will be masked, and also means that 25% of the elements in the generated mask matrix are 1, where the fusion target feature contains 8 coding marks [A1, A2, A3, A4, A5, A6, A7, A8]. According to this preset ratio of 25%, a mask matrix with the same dimension as the fusion target feature is randomly generated, that is, [0, 1, 0, 1, 1, 0, 1, 0]. Then, for the position with a value of 1 in the mask matrix, the corresponding coding mark in the fusion target feature is replaced with random noise, that is, the coding marks A2, A4, A5 and A7 are all replaced with random noise, and the noisy target feature [A1, (mask) A2, A3, (mask) A4, (mask) A5, A6, (mask) A7, A8] is obtained, where the coding mark replaced with random noise carries a noise mark (mask), indicating that the coding mark is in an absorbing mask state.

[0118] In a possible implementation, the upper limit value and the lower limit value of the mask ratio can be obtained first, and then a random number generator is used to generate a random number between the upper limit value and the lower limit value of the ratio, and the random number is used as the preset ratio. Then, a mask matrix of the same size as the fusion target feature is generated, and then according to the preset ratio, the elements in the mask matrix are replaced by 1, and the remaining elements are 0, thereby obtaining a mask matrix for adding noise; for example, the upper limit value of the ratio can be 1, the lower limit value of the ratio can be 0, and the corresponding mask ratio value interval can be [0,1]. If the generated random number is 0, then the preset ratio can be 0%, that is, all the coding marks in the fusion target feature remain unchanged. If the generated random number is 1, the preset ratio is determined to be 100%, and all the coding marks in the fusion target feature are replaced with random noise. If the generated random number is 0.4, the preset ratio is 40%, and 40% of the total number of coding marks in the fusion target feature are replaced with random noise, and the remaining coding marks remain unchanged.

[0119] In one possible implementation, a neural network model can be used to perform mask processing on the fused target features. Specifically, the fused target features are input into the target model that has been jointly trained with the diffusion model. The target model will perform inference based on the fused target features to predict the positions that need to be masked. The prediction results can be a series of coordinates or indexes, indicating which positions of the elements in the fused target features will be masked. According to the output prediction results, a position number corresponding to a preset ratio is generated, and then a mask matrix with the same size as the fused target feature is created according to the predicted position number. The elements of the corresponding position numbers in the mask matrix will be set to 1 (indicating that these positions will be masked), and the remaining positions will be 0. Then, for the positions that are 1 in the mask matrix, noise marks can be added to the elements of the corresponding positions in the fused target features, or the generated random noise can be superimposed on the elements in the corresponding positions to achieve a noise adding effect and obtain a noisy target feature.

[0120] In one possible implementation, since the fused target feature is a feature vector obtained by fusing features of different modalities, the target model can directly predict the position to be masked for the fused target feature, or assign a weight to each coding mark in the fused target feature. This weight can reflect the importance of different coding marks for the prediction of the position to be masked. By adjusting these weights, it can help the target model balance the impact of masking coding marks of different modalities on the prediction results.

[0121] In one possible implementation, the fused target feature can be input into the target model, and the target model creates a position coding feature for the fused target feature; then, the fused target feature and the position coding feature are spliced ​​to obtain a spliced ​​feature; then, the spliced ​​feature is encoded using a self-attention mechanism to obtain a self-attention feature; classification is performed based on the self-attention feature to predict the position to be masked to obtain a classification result, and a position number corresponding to a preset ratio is determined based on the classification result, wherein the classification result is used to indicate the probability of each coding mark being masked.

[0122] It is worth noting that the position coding feature can be the position serial number of each coding mark in the fusion target feature, which is used to indicate the position information of multiple coding marks in the fusion target feature. Specifically, the position coding feature contains multiple position codes, and each position code corresponds one-to-one to each coding mark in the fusion target feature.

[0123] In one possible implementation, Figure 7As shown in the figure, the fused target feature [A1, A2, A3, A4, A5] contains 5 coding tags. The fused target feature is input into the target model. The target model creates a position code for each coding tag and combines all the position codes to obtain the position code feature [W1, W2, W3, W4, W5]. Then, the fused target feature is concatenated with the position code feature, and the concatenated feature is encoded by the self-attention mechanism. The self-attention mechanism calculates the correlation between each vector and other vectors in the concatenated feature, and updates the representation of each vector according to these correlations to obtain the self-attention feature [C1, C2, C3, C4, C5]. Each element in the self-attention feature represents the self-attention score of each vector in the concatenated feature. The elements in the self-attention feature are classified by a classifier. The output of the classifier can be a 5×2 probability distribution matrix, and the elements in each column represent the probability of a coding tag being masked and the probability of not being masked. According to the preset ratio and the number of coding marks, the number to be masked is determined. For example, if the preset ratio is 40%, it means that 2 of the 5 coding marks will be masked. Therefore, according to the probability in the classification result, the 2 coding marks with the highest probability of being masked are selected as the positions to be masked, that is, the elements C1 and C4 in the self-attention feature, so that the coding marks to be masked are determined to be A1 and A4, and then the position numbers of these coding marks in the fusion target feature are determined to be 1 and 4 through the position coding feature. Therefore, after obtaining the position number, the coding mark of the corresponding position signal in the fusion target feature is denoised to obtain the noisy target feature.

[0124] Step 204: Input the noisy target features into the diffusion model for denoising based on bidirectional attention to obtain denoised target features, and generate first target content of the first modality and second target content of the second modality based on the denoised target features.

[0125] In a possible implementation, the noisy target feature may refer to feature data that is mixed with noise or interference information in the fused target feature, or to a coding mark in the fused target feature that is in an absorption mask state, and the diffusion model interprets the coding mark as random noise and does not contain any information. The noisy target feature is input into the diffusion model for denoising. The diffusion model can learn the inherent laws and structures of the data by simulating the diffusion process of the data, thereby effectively removing the noise and restoring a purer and more accurate denoised target feature.

[0126] In a possible implementation, when the fusion target feature is denoised, a preset proportion of the coding marks in the fusion target feature is masked instead of the causal attention mask in the self-attention mechanism. Therefore, when the bidirectional attention mechanism can be used for denoising in the diffusion model, the unidirectional limitation of the causal attention mask is broken. In the denoising process, for each element in the noisy target feature, the diffusion model can calculate the attention weight between it and the context elements through the bidirectional attention mechanism, and then perform weighted summation of the context elements according to these weights to obtain the context representation of the current element. This context representation not only contains the information of the current element itself, but also integrates the relevant information of the surrounding elements, which helps to more accurately identify and remove noise. Therefore, the use of the bidirectional attention mechanism for denoising in the diffusion model can capture the local and global dependencies in the noisy target feature and improve the accuracy and robustness of denoising.

[0127] In a possible implementation, after obtaining the denoised target features, the first target content of the first modality and the second target content of the second modality can be generated according to the denoised target features based on the task requirements and modality characteristics. For example, a generative model of the corresponding modality can be used, the denoised target features can be input into the generative model, and the target content of the corresponding modality can be generated according to the output of each generative model.

[0128] Based on this, the content generation method provided by the embodiment of the present disclosure relies on the full-sequence diffusion process of the bidirectional encoding mechanism in the diffusion model to realize the bidirectional interaction of image data and text data, thereby enhancing the ability of multimodal fusion; by encoding the two modal contents of image data and text data and then conducting mixed training, the model can better realize the alignment between the modalities, thereby solving the problems of modal inconsistency or insufficient reasoning in multimodal tasks in the related technology, and ensuring the accuracy of content generation; in addition, the absorption mask state is used to process the two different modalities of image data and text data, which can jointly model data of different modalities, integrate and reason multimodal information, and effectively improve the decision-making and planning capabilities when processing complex multimodal tasks.

[0129] For example, in an intelligent agent application scenario, if the intelligent agent is required to perform the task of "sprinkling salt on the steak", the relevant technology may only rely on information from a single modality, such as text instructions or image clues, which is prone to decision-making bias when multimodal information conflicts. However, the content generation method provided by the embodiment of the present disclosure can obtain information from multimodal input. For example, reasonable reasoning can be performed from the text instruction "sprinkle salt on the steak" and the salt shaker seen in the image through two-way interaction and global planning capabilities. Even if there are other interfering factors in the image, such as a pepper spice bottle, the salt shaker related to the task can be identified and prioritized, thereby helping the intelligent agent to generate a series of logical action steps, such as "pick up the salt shaker", thereby effectively supporting the intelligent agent to complete complex tasks.

[0130] In one possible implementation, Figure 8 As shown, information can be obtained from the multimodal input, the first input content "Go to the shelf." and "Pick up the saltbottle." of the text modality can be obtained from the text instruction, and the second input content "image containing the salt bottle" of the image modality can be obtained from the camera. By encoding the first input content and the second input content respectively, the first target feature corresponding to the first input content and the second target feature corresponding to the second input content can be obtained. Then, the second target feature is inserted into the first target feature to obtain the fused target feature. It can be seen that when the second target feature is inserted into the first target feature, a start marker BOI is added to the start position of the sequence of the second target feature, and an end marker EOI is added to the end position of the sequence of the second target feature. Next, the fused target features are input into the Transformer neural network model and processed using a bidirectional attention mechanism. The processed fused target features are output, and then the coded marks of a preset proportion in the fused target features output by the Transformer model are masked to add noise to the fused target features to obtain noisy target features. The noisy target features are then input into the diffusion model for denoising based on bidirectional attention to obtain denoised target features, and then the denoised target features can be used to generate the first target content of the first modality and the second target content of the second modality. It is worth noting that after obtaining the denoised target features, the denoised target features and the fused target features can also be used to calculate the loss, optimize the model parameters of the diffusion model, and realize the training of the diffusion model.

[0131] In one possible implementation, Fig. 9As shown in the figure, after encoding the input content of the text modality, the target features corresponding to the text modality can be obtained. At this time, the encoding mark in the target features corresponding to the text modality refers to the sequence number of the vocabulary, and then the target features corresponding to the text modality are mapped to the continuous space by means of an embedding matrix. For a given sequence number, the corresponding vector representation can be obtained by finding the corresponding row of the embedding matrix, and then these vector representations are input into the Transformer model for processing by the bidirectional attention mechanism, and the target vector is output. Then, the softmax function is used to normalize the target vector and convert it into a probability distribution. This probability distribution is the probability distribution of each element in the vocabulary, and the word or character with the largest probability distribution can be selected as the output result.

[0132] In one possible implementation, the diffusion model can be used for an iterative denoising process, which is based on a bidirectional attention mechanism and is multi-step until a preset time step is reached. In the denoising process of the current time step, the noisy target feature (which may refer to the denoising result of the previous time step) is input into the diffusion model. The diffusion model uses the bidirectional attention mechanism of the Transformer to capture the complex relationship between the encoding tags and performs denoising to obtain the denoising result of the current time step. Next, an iterative denoising process of the time step is performed, and the denoising result of the current time step is used as input again. The above denoising process is repeated, and each iteration will further reduce the noise and improve the quality of the denoising result. When the iterative process ends, the feature obtained is the denoising target feature.

[0133] In a possible implementation, the denoising process is a multi-step iterative process. Therefore, the preset ratio of the masked coding mark in the fusion target feature can be designed according to the preset time step. The preset time step can refer to the maximum number of iterations of the denoising process of the diffusion model or a fixed time scale related to the denoising process. Specifically, a random number can be generated according to the preset time step of the denoising process. The random number is a positive integer less than the preset time step. The preset ratio is determined according to the ratio between the random number and the preset time step. For example, the positive integer of the preset time step is T, then T gradually decreases in each iteration round, for example, each iteration round is reduced by 1, 2, etc., and a positive integer random number t is generated in the interval between greater than 0 and less than the preset time step, that is, t∈[1,T), so that the preset ratio can be determined as t / T. When the positive integer of the preset time step T is 100, the generated random number t is 30, then the determined preset ratio is 30%. After the preset ratio is determined, the encoding marks of the preset ratio in the fusion target feature are randomly selected for masking to obtain the noisy target feature, and then the noisy target feature is used as the input of the subsequent diffusion model denoising process. The diffusion model will be carried out in the iterative denoising method described above until the preset time step is reached or other termination conditions are met. In the next time step, the random number t can be reduced according to the preset step size (such as 1 or 2 per iteration), and then the preset ratio t / T is re-determined using the new random number t. For example, if the generated random number t is 30 as mentioned above, then the random number t can be 29 in the next time step, and it will be reduced by 1 in subsequent iterations until the random number t is 1. Alternatively, the random number t starts from 29 and is reduced by a larger step size each iteration, such as 2, 4, etc., until the random number t is 1.

[0134] In a possible implementation, the iterative denoising process of the diffusion model may include the following steps: first, inputting the noisy target features into the diffusion model; then, determining the query vector, key vector and value vector of each coding tag, traversing multiple coding tags, and obtaining the attention score between the current coding tag and each of the remaining coding tags based on the dot product between the query vector of the current coding tag and the key vectors of the remaining coding tags; then, traversing multiple coding tags, and according to the attention scores of the current coding tag, weightedly summing the value vectors of the remaining coding tags to obtain the attention vector of the current coding tag; and then using the attention vectors of the multiple coding tags as the denoising result of the current time step after denoising based on bidirectional attention.

[0135] In a possible implementation, after the noisy target feature is input into the diffusion model, for each coding mark in the noisy target feature, the diffusion model will assign a query vector (Query), a key vector (Key) and a value vector (Value). These vectors can be calculated by the linear layer or the embedding layer of the diffusion model. The query vector, the key vector and the value vector can represent the representation of the coding mark in different spaces. By traversing each coding mark in the noisy target feature, for the current coding mark, the dot product between its query vector and the key vectors of the remaining coding marks is calculated. The dot product result is the attention score, which reflects the degree of association or similarity between the current coding mark and the remaining coding marks. It is worth noting that the calculation of the attention score is bidirectional. For each coding mark, its relationship with all other coding marks in the sequence is considered, not just the previous or subsequent coding marks. According to the calculated attention score, the value vectors of the remaining coding marks are weighted summed, and the result of the weighted sum is the attention vector of the current coding mark. The attention vector integrates the information of all other coding marks in the noisy target feature sequence. Then the attention vectors of all coding marks in the noisy target feature sequence are used as the denoising result.

[0136] Specifically, assume that the noisy target feature contains three coding tags, represented by T1, T2 and T3 respectively. For each coding tag, first determine the corresponding query vector Q i , key vector K i Sum value vector V i , where i represents the index of each coding tag (1, 2 or 3). Next, calculate the query vector Q1 of the current coding tag T1 and the key vector K of all coding tags (including itself) i The dot product between them, for T 31 and T i The dot product between (where i can be 1, 2 or 3) can be referred to as shown in the following formula (1):

[0137] ε 1i =Q1·K i (1);

[0138] Among them, ε 1i Indicates the current coding mark T1 and coding mark T i The dot product result of , therefore, the attention score of each encoding tag can be calculated separately. In addition, the dot product result can also be scaled by a scaling factor. Specifically, the scaling process of the dot product result can refer to the following formula (2):

[0139]

[0140] ε′ 1iRepresented as the scaled current coding mark T1 and coding mark T i When the dimension d of the vector is large, the dot product result between the query vector and the key vector will be large, resulting in uneven distribution of the attention scores, and even leading to the problem of gradient vanishing or gradient exploding. Therefore, the dot product result can be scaled to alleviate this situation. For example, the scaling factor in formula (2) can be used. That is, the reciprocal of the square root of the vector dimension d, or the reciprocal of the vector dimension d can also be used as the scaling factor. The scaling factor can play a role in adjusting the distribution of attention scores. By scaling, the attention scores can be more evenly distributed on each value vector, avoiding some value vectors from receiving too much attention and ignoring other value vectors.

[0141] Next, the dot product result can be normalized using the softmax function. The dot product result can be mapped to the interval (0,1) by using the softmax function for conversion, and the sum of these numbers is equal to 1. In this way, each coding mark will get an attention score associated with it, indicating the degree of association between the current coding mark and other coding marks in the sequence. After obtaining the attention score, the attention vector of each coding mark can be calculated. The attention vector is the weighted sum of all value vectors, and the weight is the corresponding attention score. The specific calculation process can refer to the following formula (3):

[0142]

[0143] Among them, δ 1i Indicates the current coding mark T1 and coding mark T i The attention score between them, and A1 represents the attention vector of the current coding tag T1. Therefore, the attention vector A2 of the coding tag T2 and the attention vector A3 of the coding tag T3 can be calculated respectively according to formula (3), and then these attention vectors A1, A2 and A3 are combined to obtain the denoising result of the current time step after denoising based on bidirectional attention.

[0144] In a possible implementation, the query vector, key vector and value vector of each coding tag can be obtained through the following steps: first obtain the query weight matrix, key weight matrix and value weight matrix of each coding tag; for any coding tag, obtain the query vector of the coding tag according to the product of the coding tag and the query weight matrix, obtain the key vector of the coding tag according to the product of the coding tag and the key weight matrix, and obtain the value vector of the coding tag according to the product of the coding tag and the value weight matrix.

[0145] Among them, the query weight matrix, key weight matrix and value weight matrix are parameters learned during the diffusion model training process. They are not directly calculated through a fixed algorithm or formula, but are continuously optimized and adjusted during the training process through the back propagation algorithm. Specifically, before the model training begins, the query weight matrix, key weight matrix and value weight matrix will be initialized to some random values. For each encoding token in the input sequence, the model will multiply it with the query weight matrix to obtain a query vector; multiply it with the key weight matrix to obtain a key vector; multiply it with the value weight matrix to obtain a value vector, so as to map the input encoding token to different representation spaces, thereby capturing different features in the input sequence; after obtaining the query vector, key vector and value vector, the diffusion model can use a bidirectional attention mechanism to calculate the attention score, and use the attention score to perform weighted summation on the value vector to obtain the attention vector of each encoding token; then, the loss function is used to evaluate the gap between the predicted results and the actual results of the diffusion model to obtain the loss function value, and then the back-propagation algorithm is used to calculate the gradient of the loss function with respect to the model parameters, and then the optimization algorithm is used to update the diffusion model parameters to reduce the loss function value. In the process of updating the model parameters, the query weight matrix, key weight matrix and value weight matrix will also be continuously optimized and adjusted.

[0146] In a possible implementation, when the first modality is a text modality, the following steps may be used to train the diffusion model: first, obtain a first sample content of the first modality and a second sample content of the second modality, encode the first sample content to obtain a first sample feature, and encode the second sample content to obtain a second sample feature; then, insert the second sample feature into the first sample feature and perform noise addition to obtain a noisy sample feature, and call the diffusion model to obtain a denoised sample feature based on the noisy sample feature, wherein the denoised sample feature includes a text probability distribution corresponding to the first modality and a denoising sub-feature corresponding to the second modality; then, obtain a first label content of the first modality and a second label content of the second modality, encode the first label content to obtain a first label feature, and encode the second label content to obtain a second label feature; then, map the text probability distribution to a continuous space, determine a first mean square error loss based on the mapped text probability distribution and the first label feature, and determine a second mean square error loss based on the denoising sub-feature and the second label feature; perform weighted summation of the first mean square error loss and the second mean square error loss to obtain a target loss, and adjust the parameters of the diffusion model based on the target loss.

[0147] Among them, when the first modality is a text modality, that is, the first sample content is text data, a text encoder can be used to convert the first sample content into a vector sequence, the first sample feature can include key information in the first sample content, and the first sample content can include multiple coding tags. Similarly, the second modality can refer to an image modality, or a video modality or an audio modality. Taking the second modality as an image modality as an example, an image encoder can be used to encode the second sample content and convert it into a vector sequence, and the second sample content includes visual features in the second sample content. Since the features of different modalities are different in form, they need to be aligned, that is, converted to the same dimension. For example, the first sample feature can be mapped to the same continuous space as the second sample feature, and then the second sample feature can be randomly inserted into the first sample feature mapped to the continuous space to obtain a fused sample feature. It is worth noting that before inserting the second sample feature into the first sample feature, it is also necessary to add a start mark and an end mark at both ends of the second sample feature, and then insert the second sample feature after adding the start mark and the end mark. At this time, the obtained fused sample features can be input into the Transformer model for processing, and the global features of the fused sample features are extracted using the bidirectional attention mechanism of the Transformer model to obtain the optimized fused sample features. Then, the optimized fused sample features are subjected to noise processing. Specifically, a preset time step is obtained, a random number of a positive integer less than the preset time step is generated, and a preset ratio is determined according to the ratio of the generated random number to the preset time step. Then, the coding mark of the preset ratio in the fused sample features is masked to complete the noise processing of the fused sample features and obtain the noisy sample features. After obtaining the noisy sample features, the diffusion model is called to perform denoising on the noisy sample features based on the bidirectional attention mechanism to obtain the text probability distribution corresponding to the first modality and the denoising sub-features corresponding to the second modality. Obviously, after the fused sample features are input into the Transformer model for bidirectional attention mechanism processing, the output optimized fused sample features still retain the start mark and end mark corresponding to the second modality. Therefore, when the diffusion model processes the fused sample features, the start mark and end mark can be identified, and the features of different modalities are processed separately. The features corresponding to the first modality are converted to obtain the text probability distribution, so that the words or characters with the largest probability distribution can be selected from the vocabulary as the output result corresponding to the first modality through the text probability distribution, and the features corresponding to the second modality are converted to denoising sub-features, so that more accurate image results can be generated through the generative model.

[0148] After obtaining the denoised sample features, the first label content of the first modality and the second label content of the second modality can be obtained. The label content of each modality can refer to the corresponding real value or expected output in each modality. The first label feature refers to the feature obtained after the first label content is encoded in the same way as the first sample content, that is, the vector sequence obtained by converting the first label content through the same text encoder. Similarly, the second label feature refers to the feature obtained after the second label content is encoded in the same way as the second sample content, that is, the vector sequence obtained by converting the second label content through the same image encoder.

[0149] Since the first modality is the text modality, the text probability distribution and the first label feature are both discrete signals, and it is impossible to perform a unified loss calculation on the features of different modalities. Therefore, the discrete signal can be re-parameterized, that is, the text probability distribution and the first label feature of the first modality can be re-parameterized. After mapping the text probability distribution and the first label feature to the continuous space, they are in the same continuous space as the denoising sub-features and the second label features of the second modality. Therefore, the first mean square error loss can be calculated based on the mapped text probability distribution and the first label feature, and the second mean square error loss can be calculated based on the denoising sub-features and the second label feature. Then, the first mean square error loss and the second mean square error loss can be weightedly summed to obtain the target loss. Based on the target loss, an optimization algorithm (such as gradient descent) is used to adjust the parameters of the diffusion model to minimize the loss and improve the model performance.

[0150] In a possible implementation, if the text probability distribution needs to be mapped to a continuous space, the text probability distribution can be normalized first, and then the first sample feature is weighted averaged according to the normalized text probability distribution. The weighted average result is the mapped text probability distribution. Therefore, the above steps can be performed on the second label feature to map the second label feature to a continuous space.

[0151] In a possible implementation, since the diffusion model processes sequences containing different modalities, and the features of different modalities have different loss calculation methods, for example, for image modality data, the diffusion model usually uses mean square error loss to measure the gap between the denoised pattern predicted by the model and the real pattern signal; but unlike continuous image signals, the signal of the text modality is discrete. Therefore, when using the diffusion model to process text, the cross entropy loss is usually used. It is difficult to ensure the training efficiency and the balanced influence of different modal data on the model during the training process by using different loss functions. Therefore, after mapping the feature data of the first modality, i.e., the text modality, to the continuous space, the loss functions of the text modality and the image modality can be unified into the mean square error loss function. The specific calculation process can refer to the following equations (4) and (5):

[0152] x′ t(text) =W Embed ·softmax(f θ (x t(text) )) (4);

[0153]

[0154] Among them, f θ (x t(text) ) represents the logarithmic probability output by the diffusion model at time step t under parameter θ. The logarithmic probability is normalized by the softmax function to obtain the normalized text probability distribution, and the output logarithmic probability is converted into text probability distribution. Embed It can refer to the embedding matrix corresponding to the first sample feature, taking the normalized text probability distribution as the weight, and performing weighted averaging on the first sample feature to realize the conversion of the text probability distribution to a new text vector in the continuous space, that is, the mapped text probability distribution x′ t(text) Then, as shown in formula (5), according to the mapped text probability distribution x′ t(text) Calculate the first mean square error term with the first label feature x'0 and pass the loss weight w of the text loss text and expected value The first mean square error term is weighted to obtain the first mean square error; similarly, according to the denoising sub-feature x t(vision) Calculate the second mean square error term with the second label feature x0 and pass the loss weight w of the image loss vision and expected value The second mean square error term is weighted to obtain the second mean square error, and then the first mean square error and the second mean square error are summed to obtain the target loss L t (θ).

[0155] In one possible implementation, Fig.10As shown in FIG. 1 , when the diffusion model is trained, first, step 1001 is executed: data samples are randomly extracted, and a group of data samples are randomly extracted from the multimodal mixed data set, wherein the group of data samples includes the first sample content of the first modality and the second sample content of the second modality; secondly, step 1002 is executed: a time step is selected, and a time step is randomly selected from a uniform distribution, and this time step represents the sampling time point of the current data sample; then, step 1003 is executed: a noise sample is generated, and the first sample content of the first modality and the second sample content of the second modality in each extracted data sample are respectively encoded to obtain the encoded first sample feature and the second sample feature, and then the second sample feature is inserted into the first sample feature to obtain the fused sample feature, and the coded mark in the ratio of the current time step to the preset time step in the fused sample feature is masked to generate the noisy sample feature; then, step 1004 is executed: the loss function is calculated, and the loss function of the first modality is calculated according to the loss function of the first modality. The loss weight of the first mode and the loss weight of the second mode are used to calculate the target loss through a unified loss function; next, execute step 1005: optimize the model parameters, and continuously update the parameters of the diffusion model by minimizing the target loss to improve the performance of the diffusion model on the current sample; after each parameter update, execute step 1006: check the stability of the model, test the diffusion model after the parameter update, and observe whether the updated diffusion model has signs of performance collapse, ensure the stability of the model, and prevent uncontrollable situations during the training process. If the model shows signs of collapse, terminate the training cycle immediately. Otherwise, use the model after the parameter update as the processing model for the next time step and continue to optimize; execute step 1007: iterate the optimization process, and re-execute the above steps until the preset number of iterations is reached. Through multiple iterations and optimizations, a diffusion model with stable performance and excellent performance can be obtained, which can effectively process multimodal mixed data and generate data similar to the original data.

[0156] In a possible implementation, when training the model, the reinforcement learning method can also be used to iterate the model training to optimize the model's ability to make decisions and reasoning in the environment. Specifically, the model includes a codec part, a Transformer model part, and a diffusion model part. First, the model parameters of the model are initialized, including neural network weights, learning rates, etc., and reinforcement learning related parameters are set, such as discount factors, exploration rates, etc.; a group of sample data is randomly selected from the image and text mixed data set as the environment state, and the sample data is input into the model. The model selects actions according to the current state, that is, the input sample data and the strategy, that is, generating the next word of the text data, denoising the multimodal data, etc.; according to the actions taken by the model and the environment state, the reward is calculated, wherein the reward can be based on the denoising effect, the quality of the generated text, etc. as evaluation indicators. If the action is correct, that is, the denoising effect is good and the quality of the generated text is high, a positive reward is given, otherwise a negative reward or zero reward is given; then, the policy gradient algorithm is used to update the model parameters according to the reward and action probability gradient, and the above steps are repeated until the preset number of iterations is reached. Therefore, by using reinforcement learning to train the model, it helps the model learn more optimized decision-making strategies in complex environments and improve the accuracy of generating tasks. Through the reward function, reinforcement learning enables the model to learn the long-term optimal strategy and adaptively adjust the strategy to cope with different environmental conditions, thereby enhancing the generalization ability of the model.

[0157] In one possible implementation, the performance of the diffusion model can be evaluated using evaluation methods in different fields. For example, tasks that combine visual input (such as images and videos) with mathematical or scientific reasoning can be included in the evaluation. These tasks can be used to evaluate the diffusion model's ability to handle complex logical reasoning and multimodal information integration. Through these tasks, we can understand the performance of the diffusion model in scenarios that require deep understanding and reasoning. Alternatively, the evaluation tasks can include robot planning and graphic user interface agent tasks to observe the performance of the diffusion model in handling complex multi-step reasoning and decision-making. Alternatively, the diffusion model can be evaluated by combining quantitative indicators (such as accuracy and response time) with qualitative analysis (such as task completion and reasoning path). Quantitative indicators will provide objective data support for the model's performance, while qualitative analysis will be used to evaluate the model's performance in the reasoning process and decision-making logic. Therefore, the use of evaluation methods in different fields can help adjust the applicability of the model in different actual application scenarios.

[0158] The content generation method provided by the embodiment of the present disclosure is described in detail below with specific examples.

[0159] Reference Fig.11 As shown, Fig.11 is a specific flow chart of the content generation method provided by a specific example. Fig.11 In the schedule synchronization method, the following steps 1101 to 1118 may be included.

[0160] Step 1101: Obtain first input content of a first modality and second input content of a second modality.

[0161] Step 1102: Encode the first input content to obtain a first target feature, and encode the second input content to obtain a second target feature.

[0162] Step 1103: Map each encoding mark in the first target feature to a continuous space.

[0163] Step 1104: Add a start tag and an end tag for indicating the second mode at both ends of the second target feature.

[0164] Step 1105: insert the second target feature after adding the start mark and the end mark into the mapped first target feature to obtain a fused target feature.

[0165] Step 1106: Generate a random number that is a positive integer less than a preset time step.

[0166] Step 1107: Determine a preset ratio according to the ratio between the random number and the preset time step.

[0167] Step 1108: Input the fused target features into the target model to create position coding features for the fused target features.

[0168] Step 1109: concatenate the fused target features and the position coding features to obtain concatenated features.

[0169] Step 1110: Use the self-attention mechanism to encode the concatenated features to obtain self-attention features.

[0170] Step 1111: Classify based on the self-attention feature to predict the position to be masked, obtain the classification result, and determine the position number corresponding to the preset ratio based on the classification result.

[0171] Step 1112: masking the coding mark of the position number in the fused target feature to add noise to the fused target feature to obtain a noisy target feature.

[0172] Step 1113: Input the noisy target features into the diffusion model to determine the query vector, key vector and value vector of each encoding tag.

[0173] Step 1114: traverse multiple coding tags, and obtain the attention score between the current coding tag and the remaining coding tags based on the dot product between the query vector of the current coding tag and the key vectors of the remaining coding tags.

[0174] Step 1115: traverse multiple coding tags, and according to the attention scores of the current coding tag, perform weighted summation on the value vectors of the remaining coding tags to obtain the attention vector of the current coding tag.

[0175] Step 1116: Use the attention vectors of multiple encoded tags as the denoising result of the current time step obtained after performing bidirectional attention-based denoising.

[0176] Step 1117: The denoising result of the current time step is input into the diffusion model again for denoising based on bidirectional attention until the current time step reaches the preset time step to obtain the denoising target feature.

[0177] Step 1118: Generate first target content of the first modality and second target content of the second modality based on the denoised target features.

[0178] Through the content generation method of the above steps 1101 to 1118, firstly, the first input content of the first modality and the second input content of the second modality are obtained, and the input contents of the two different modalities are respectively encoded to obtain the corresponding first target features and second target features. In order to effectively fuse the features of the two different modalities, the start mark and the end mark are added to both ends of the second target feature, and the second target feature with the added mark is inserted into the first target feature mapped to the continuous space to construct the fused target feature. Next, a new noise adding mechanism is introduced to dynamically determine a preset ratio according to the ratio of the random number to the preset time step, and then select part of the coding marks in the fused target feature according to this ratio for mask processing, thereby generating a noisy target feature. This noise adding method not only uniformly applies discrete noise to the coding marks of different modalities, but also lays the foundation for the subsequent unified modeling of multi-modality through the diffusion model. Subsequently, the noisy target features are input into the diffusion model and denoised using the bidirectional attention mechanism. In this process, the model generates attention vectors for each coded tag based on the query vector, key vector, and value vector of a coded tag by calculating the attention score and weighted summation. These attention vectors not only contain the information of the current coded tag itself, but also integrate the contextual information of all other codes, thereby achieving effective interaction of global information. After multiple iterations of denoising until the preset time step is reached, the denoised target features can be obtained, which not only retains the key information of the original input content, but also makes the generated denoised target features richer, more accurate and diversified through the introduction of bidirectional attention. Therefore, when the first target content of the first modality and the second target content of the second modality are subsequently generated based on the denoised target features, the accuracy of content generation can be improved while generating diversified target content.

[0179] The following are some practical examples to illustrate the application scenarios of the embodiments of the present disclosure.

[0180] It should be noted that the content generation method provided in the embodiments of the present disclosure can be applied to different application scenarios such as multimedia content creation scenarios, intelligent customer service dialogue scenarios or robot interaction scenarios. The following is an explanation using multimedia content creation scenarios, intelligent customer service dialogue scenarios and robot interaction scenarios as examples.

[0181] Scene 1

[0182] The content generation method provided by the embodiment of the present disclosure can be applied to multimedia content creation scenarios. In the field of multimedia content creation, such as video creation, advertising planning, etc., it is often necessary to combine information in multiple modes such as text, images, audio, etc. to conceive and create content. For example, to make a video introducing a new product, text is needed to describe the product features, images and video clips are needed to show the product appearance and usage effects, and audio may be needed as background music to enhance the atmosphere. Therefore, the content generation method provided by the embodiment of the present disclosure can be used to obtain first input content of a first modality and second input content of a second modality, that is, to use text content as the input content of the first modality, and to use images or video clips as the input content of the second modality, to encode the first input content to obtain a first target feature, and to encode the second input content. A second target feature is obtained, and the second target feature is inserted into the first target feature to obtain a fused target feature, wherein the fused target feature includes a plurality of coding tags; then, the fused target feature is denoised by masking a preset proportion of the coding tags in the fused target feature to obtain a noisy target feature; then, the noisy target feature is input into a diffusion model for denoising based on bidirectional attention to obtain a denoised target feature, and a first target content of a first modality and a second target content of a second modality are generated based on the denoised target feature, thereby obtaining multimedia content that is consistent with the text description and rich in visual expression, such as generating a matching image or video clip according to the text content, or generating a corresponding text description according to the image or video clip, thereby enriching the content of the video and enhancing the attractiveness and communication effect of the generated content.

[0183] Scene 2

[0184] The content generation method provided by the embodiment of the present disclosure can be applied to intelligent customer service dialogue scenarios. In the intelligent customer service dialogue scenarios, customers can raise questions and demands in a variety of ways such as text, voice, and pictures, and the intelligent customer service needs to accurately understand the customer's intentions and give appropriate responses. In order to improve the response quality and customer experience of intelligent customer service, it is necessary for the intelligent customer service to be able to efficiently integrate and process information in multiple modes. Therefore, the content generation method provided by the embodiment of the present disclosure can be used to obtain first input content of the first mode and second input content of the second mode, that is, the text information input by the customer is used as the input content of the first mode, and the image or video clip uploaded by the customer is used as the input content of the second mode, the first input content is encoded to obtain a first target feature, and the second input content is encoded to obtain a first target feature. The input content is encoded to obtain a second target feature, and the second target feature is inserted into the first target feature to obtain a fused target feature, wherein the fused target feature includes multiple coding tags; then, the fused target feature is denoised by masking a preset proportion of the coding tags in the fused target feature to obtain a noisy target feature; then, the noisy target feature is input into the diffusion model for denoising based on bidirectional attention to obtain a denoised target feature, and the first target content of the first modality and the second target content of the second modality are generated based on the denoised target feature. Therefore, when a customer asks a question about the use of a certain product, the intelligent customer service can not only answer it through text, but also attach relevant product usage pictures or videos, so that customers can understand the solution more intuitively.

[0185] Scene 3

[0186] The content generation method provided by the embodiment of the present disclosure can be applied to robot interaction scenarios. In the robot interaction scenario, the customer can interact with the robot through multiple modes such as text, gestures, voice, and pictures to give instructions or ask for information. The robot needs to be able to accurately understand the customer's intentions and generate appropriate responses or perform corresponding actions based on multimodal information. The content generation method provided by the embodiment of the present disclosure is used to obtain first input content of the first modality and second input content of the second modality, that is, the text information input by the customer is used as the input content of the first modality, and the image or video clip containing the customer's gesture is obtained as the input content of the second modality. The first input content is encoded to obtain a first target feature. The second input content is encoded to obtain a second target feature, and the second target feature is inserted into the first target feature to obtain a fused target feature, wherein the fused target feature includes a plurality of coding tags; then, the fused target feature is denoised by masking a preset proportion of the coding tags in the fused target feature to obtain a noisy target feature; then, the noisy target feature is input into a diffusion model for denoising based on bidirectional attention to obtain a denoised target feature, and a first target content of a first modality and a second target content of a second modality are generated based on the denoised target feature. Therefore, when a customer gives an instruction, the robot can accurately understand the customer's intention, respond to the customer's instruction, and perform actions according to the customer's instruction.

[0187] It is to be understood that, although the steps in the above-mentioned various flow charts are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in the present embodiment, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above-mentioned flow charts can include a plurality of steps or a plurality of stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps.

[0188] Reference Fig.12 , Fig.12 This is an optional structural diagram of a content generation device provided in an embodiment of the present disclosure. The content generation device 1200 includes:

[0189] Content acquisition module 1201, used to acquire first input content of a first modality and second input content of a second modality;

[0190] The encoding module 1202 is used to encode the first input content to obtain a first target feature, encode the second input content to obtain a second target feature, and insert the second target feature into the first target feature to obtain a fused target feature, wherein the fused target feature includes a plurality of encoding tags;

[0191] The noise adding module 1203 is used to add noise to the fused target feature by masking the coding marks of a preset proportion in the fused target feature to obtain a noisy target feature;

[0192] The denoising module 1204 is used to input the noisy target features into the diffusion model for denoising based on bidirectional attention to obtain denoised target features, and generate first target content of the first modality and second target content of the second modality based on the denoised target features.

[0193] In a possible implementation, the denoising module 1204 is specifically used for:

[0194] Input the noisy target features into the diffusion model for denoising based on bidirectional attention to obtain the denoising result of the current time step;

[0195] The denoising result of the current time step is input into the diffusion model again for denoising based on bidirectional attention until the current time step reaches the preset time step to obtain the denoising target feature.

[0196] In a possible implementation, the denoising module 1204 is specifically used for:

[0197] Inputting the noisy target features into the diffusion model;

[0198] Determine the query vector, key vector and value vector of each coding token, traverse multiple coding tokens, and obtain the attention score between the current coding token and the remaining coding tokens according to the dot product between the query vector of the current coding token and the key vectors of the remaining coding tokens;

[0199] Traverse multiple coding tags, and perform weighted summation on the value vectors of the remaining coding tags according to the attention scores of the current coding tag to obtain the attention vector of the current coding tag;

[0200] The attention vectors of multiple encoding tags are used as the denoising result of the current time step after bidirectional attention-based denoising.

[0201] In a possible implementation, the denoising module 1204 is specifically used for:

[0202] Get the query weight matrix, key weight matrix and value weight matrix for each encoding token;

[0203] For any coding tag, the query vector of the coding tag is obtained by multiplying the coding tag with the query weight matrix, the key vector of the coding tag is obtained by multiplying the coding tag with the key weight matrix, and the value vector of the coding tag is obtained by multiplying the coding tag with the value weight matrix.

[0204] In a possible implementation, the noise adding module 1203 is specifically used for:

[0205] Generate a random number, wherein the random number is a positive integer less than a preset time step;

[0206] The preset ratio is determined according to the ratio between the random number and the preset time step.

[0207] In a possible implementation, the noise adding module 1203 is specifically used for:

[0208] Input the fused target features into the target model to predict the position to be masked, and obtain the position number corresponding to the preset ratio, wherein the target model is obtained by joint training with the diffusion model;

[0209] The fused target features are denoised by masking the coded marks of the position numbers in the fused target features to obtain noisy target features.

[0210] In a possible implementation, the noise adding module 1203 is specifically used for:

[0211] Inputting the fused target features into the target model;

[0212] Creating a position coding feature for the fusion target feature, wherein the position coding feature is used to indicate position information of multiple coding marks in the fusion target feature;

[0213] The fusion target features and the position coding features are spliced ​​to obtain spliced ​​features;

[0214] The self-attention mechanism is used to encode the concatenated features to obtain the self-attention features;

[0215] Classification is performed based on the self-attention feature to predict the position to be masked, and a classification result is obtained. Based on the classification result, a position number corresponding to a preset ratio is determined, wherein the classification result is used to indicate the probability of each coding mark being masked.

[0216] In a possible implementation, the encoding module 1202 is specifically used for:

[0217] Adding a start tag and an end tag for indicating the second mode at both ends of the second target feature respectively;

[0218] The second target feature after adding the start mark and the end mark is inserted into the first target feature to obtain the fused target feature.

[0219] In a possible implementation, the encoding module 1202 is specifically used for:

[0220] Mapping each coded tag in the first target feature to a continuous space;

[0221] The second target feature is inserted into the mapped first target feature to obtain a fused target feature.

[0222] In a possible implementation, the content acquisition module 1201 is further used to: acquire first sample content of the first modality and second sample content of the second modality.

[0223] The encoding module 1202 is further used to: encode the first sample content to obtain a first sample feature, and encode the second sample content to obtain a second sample feature;

[0224] The noise adding module 1203 is further used to: insert the second sample feature into the first sample feature and then add noise to obtain a noisy sample feature;

[0225] The denoising module 1204 is further used to: call the diffusion model to obtain denoised sample features based on the noisy sample features, wherein the denoised sample features include the text probability distribution corresponding to the first modality and the denoising sub-features corresponding to the second modality;

[0226] Obtaining a first label content of a first modality and a second label content of a second modality, encoding the first label content to obtain a first label feature, and encoding the second label content to obtain a second label feature;

[0227] Mapping the text probability distribution to a continuous space, determining a first mean square error loss based on the mapped text probability distribution and the first label feature, and determining a second mean square error loss based on the denoising sub-feature and the second label feature;

[0228] The first mean square error loss and the second mean square error loss are weightedly summed to obtain the target loss, and the parameters of the diffusion model are adjusted based on the target loss.

[0229] In a possible implementation, the denoising module 1204 is specifically used for:

[0230] Normalize the text probability distribution;

[0231] The first sample feature is weighted averaged according to the normalized text probability distribution to map the text probability distribution to a continuous space.

[0232] The above-mentioned content generation device 1200 and the content generation method are based on the same inventive concept. By obtaining the first input content of the first modality and the second input content of the second modality, the first input content is encoded to obtain the first target feature, the second input content is encoded to obtain the second target feature, and the second target feature is inserted into the first target feature to obtain the fused target feature, thereby realizing the feature fusion of the input content of different modalities. Then, the fused target feature is denoised by masking the coding marks of a preset proportion in the fused target feature to obtain the noisy target feature. Through this denoising method, discrete noise can be uniformly applied to the coding marks corresponding to different modalities, which is convenient for subsequent unified modeling of multimodal models through a diffusion model. On this basis, by inputting the noisy target feature into the diffusion model for denoising based on bidirectional attention, a denoised target feature is obtained. In the denoising process, the information carried by all the coding marks in the noisy target feature can be introduced based on bidirectional attention, thereby realizing global information interaction. Therefore, when the first target content of the first modality and the second target content of the second modality are subsequently generated based on the denoised target feature, the accuracy of content generation can be improved while generating diversified target content.

[0233] The electronic device for executing the above content generation method provided in the embodiment of the present disclosure may be a terminal. Fig.13 , Fig.13 This is a partial structural block diagram of a terminal provided in an embodiment of the present disclosure, and the terminal includes: a camera assembly 1310, a first memory 1320, an input unit 1330, a display unit 1340, a sensor 1350, an audio circuit 1360, a wireless fidelity (WiFi) module 1370, a first processor 1380, and a first power supply 1390. Those skilled in the art can understand that Fig.13 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0234] The camera assembly 1310 can be used to capture images or videos. Optionally, the camera assembly 1310 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions.

[0235] The first memory 1320 may be used to store software programs and modules. The first processor 1380 executes various functional applications and data processing of the terminal by running the software programs and modules stored in the first memory 1320 .

[0236] The input unit 1330 may be used to receive input digital or character information and generate key signal input related to the terminal's settings and function control. Specifically, the input unit 1330 may include a touch panel 1331 and other input devices 1332 .

[0237] The display unit 1340 may be used to display input information or provided information and various menus of the terminal. The display unit 1340 may include a display panel 1341 .

[0238] The audio circuit 1360 , the speaker 1361 , and the microphone 1362 may provide an audio interface.

[0239] The first power source 1390 may be alternating current, direct current, a disposable battery, or a rechargeable battery.

[0240] The number of sensors 1350 may be one or more, and the one or more sensors 1350 include but are not limited to: acceleration sensors, gyroscope sensors, pressure sensors, optical sensors, etc. Among them:

[0241] The acceleration sensor can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal. For example, the acceleration sensor can be used to detect the components of gravity acceleration on the three coordinate axes. The first processor 1380 can control the display unit 1340 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor. The acceleration sensor can also be used for collecting game or user motion data.

[0242] The gyroscope sensor can detect the body direction and rotation angle of the terminal, and the gyroscope sensor can cooperate with the acceleration sensor to collect the user's 3D actions on the terminal. The first processor 1380 can implement the following functions based on the data collected by the gyroscope sensor: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0243] The pressure sensor can be set in the side frame of the terminal and / or the lower layer of the display unit 1340. When the pressure sensor is set in the side frame of the terminal, the user's holding signal of the terminal can be detected, and the first processor 1380 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor. When the pressure sensor is set in the lower layer of the display unit 1340, the first processor 1380 controls the operability controls on the UI interface according to the user's pressure operation on the display unit 1340. The operability control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0244] The optical sensor is used to collect the ambient light intensity. In one embodiment, the first processor 1380 can control the display brightness of the display unit 1340 according to the ambient light intensity collected by the optical sensor. Specifically, when the ambient light intensity is high, the display brightness of the display unit 1340 is increased; when the ambient light intensity is low, the display brightness of the display unit 1340 is decreased. In another embodiment, the first processor 1380 can also dynamically adjust the shooting parameters of the camera assembly 1310 according to the ambient light intensity collected by the optical sensor.

[0245] In this embodiment, the first processor 1380 included in the terminal can execute the content generating method of the previous embodiment.

[0246] The electronic device for executing the above-mentioned content generation method provided in the embodiment of the present disclosure may also be a server. Fig.14 , Fig.14 Partial structural block diagram of a server provided in an embodiment of the present disclosure. The server may have relatively large differences due to different configurations or performances, and may include one or more second processors 1410 and a second memory 1430, and one or more storage media 1440 (e.g., one or more mass storage devices) storing application programs 1443 or data 1442. Among them, the second memory 1430 and the storage medium 1440 may be short-term storage or persistent storage. The program stored in the storage medium 1440 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the second processor 1410 may be configured to communicate with the storage medium 1440 to execute a series of instruction operations in the storage medium 1440 on the server.

[0247] The server may also include one or more second power supplies 1420, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1460, and / or one or more operating systems 1441, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0248] The second processor 1410 in the server may be configured to execute the content generating method.

[0249] The embodiments of the present disclosure further provide a computer-readable storage medium, which is used to store a computer program, and the computer program is used to execute the content generation method of each of the above embodiments.

[0250] The embodiment of the present disclosure also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned content generation method.

[0251] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate to describe the embodiments of the present disclosure, such as being able to be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0252] It should be understood that in the present disclosure, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0253] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is more than two, greater than, less than, exceed, etc. are understood to not include the number, and above, below, within, etc. are understood to include the number.

[0254] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0255] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0256] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0257] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0258] It should also be understood that the various implementations provided in the embodiments of the present disclosure can be combined arbitrarily to achieve different technical effects.

[0259] The above is a specific description of the preferred implementation of the present disclosure, but the present disclosure is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present disclosure. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present disclosure.

Claims

1. A content generation method, characterized in that: include: Obtaining first input content of a first modality and second input content of a second modality; Encoding the first input content to obtain a first target feature, encoding the second input content to obtain a second target feature, inserting the second target feature into the first target feature to obtain a fused target feature, wherein the fused target feature includes a plurality of encoding tags; By masking a preset proportion of the coding marks in the fused target feature, the fused target feature is denoised to obtain a noisy target feature; The noisy target features are input into a diffusion model for denoising based on bidirectional attention to obtain denoised target features, and the first target content of the first modality and the second target content of the second modality are generated based on the denoised target features.

2. The content generation method according to claim 1, characterized in that: The step of inputting the noisy target feature into a diffusion model for denoising based on bidirectional attention to obtain the denoised target feature includes: Inputting the noisy target features into the diffusion model for denoising based on bidirectional attention to obtain the denoising result of the current time step; The denoising result of the current time step is input into the diffusion model again for denoising based on bidirectional attention until the current time step reaches the preset time step to obtain the denoising target feature.

3. The content generation method according to claim 2, characterized in that: The step of inputting the noisy target feature into the diffusion model for denoising based on bidirectional attention to obtain the denoising result of the current time step includes: Inputting the noisy target features into a diffusion model; Determine the query vector, key vector and value vector of each of the coding tags, traverse the multiple coding tags, and obtain the attention score between the current coding tag and each of the remaining coding tags according to the dot product between the query vector of the current coding tag and the key vectors of each of the remaining coding tags; Traversing the plurality of coding tags, and performing weighted summation on the value vectors of the remaining coding tags according to the attention scores of the current coding tag, to obtain the attention vector of the current coding tag; The attention vectors of the multiple encoded labels are used as the denoising result of the current time step obtained after denoising based on bidirectional attention.

4. The content generation method according to claim 3, characterized in that: The determining of the query vector, the key vector and the value vector of each of the encoding tags comprises: Obtaining a query weight matrix, a key weight matrix and a value weight matrix for each of the encoding tags; For any one of the coding tags, the query vector of the coding tag is obtained according to the product of the coding tag and the query weight matrix, the key vector of the coding tag is obtained according to the product of the coding tag and the key weight matrix, and the value vector of the coding tag is obtained according to the product of the coding tag and the value weight matrix.

5. The content generation method according to claim 2, characterized in that: Before adding noise to the fused target feature by masking the coding marks of a preset proportion in the fused target feature to obtain the noisy target feature, the content generation method further includes: Generate a random number, wherein the random number is a positive integer less than the preset time step; The preset ratio is determined according to a ratio between the random number and the preset time step.

6. The content generation method according to claim 1, characterized in that: The method of masking the coding marks of a preset proportion in the fused target features to add noise to the fused target features to obtain the noisy target features includes: Inputting the fused target features into the target model to predict the position to be masked, and obtaining a number of position numbers corresponding to the preset ratio, wherein the target model is obtained by joint training with the diffusion model; The fused target feature is denoised by masking the coding mark of the position number in the fused target feature to obtain a noisy target feature.

7. The content generation method according to claim 6, characterized in that: The step of inputting the fused target features into the target model to predict the position to be masked, and obtaining a number of position numbers corresponding to the preset ratio, includes: Inputting the fused target features into a target model; Creating a position coding feature for the fusion target feature, wherein the position coding feature is used to indicate position information of a plurality of the coding marks in the fusion target feature; Splicing the fusion target feature and the position coding feature to obtain a splicing feature; A self-attention mechanism is used to encode the concatenated features to obtain a self-attention feature; Classification is performed based on the self-attention features to predict the positions to be masked, and a classification result is obtained. Based on the classification result, a number of position numbers corresponding to the preset ratio is determined, wherein the classification result is used to indicate the probability of each of the coding marks being masked.

8. The content generation method according to claim 1, characterized in that: The inserting the second target feature into the first target feature to obtain a fused target feature includes: Adding a start tag and an end tag for indicating the second mode at both ends of the second target feature respectively; The second target feature after adding the start tag and the end tag is inserted into the first target feature to obtain a fused target feature.

9. The content generation method according to claim 1, characterized in that: The first mode is a text mode, the coding mark in the first target feature is a sequence number in a vocabulary, and the inserting the second target feature into the first target feature to obtain a fused target feature includes: Mapping each of the coded marks in the first target feature to a continuous space; The second target feature is inserted into the mapped first target feature to obtain a fused target feature.

10. The content generation method according to claim 1, characterized in that: The first mode is a text mode, and before the noisy target feature is input into the diffusion model for denoising based on bidirectional attention, the content generation method further includes: Acquire first sample content of the first modality and second sample content of the second modality, encode the first sample content to obtain a first sample feature, and encode the second sample content to obtain a second sample feature; Inserting the second sample feature into the first sample feature and adding noise to obtain a noisy sample feature, calling the diffusion model to obtain a denoised sample feature based on the noisy sample feature, wherein the denoised sample feature includes a text probability distribution corresponding to the first modality and a denoised sub-feature corresponding to the second modality; Acquire a first label content of the first modality and a second label content of the second modality, encode the first label content to obtain a first label feature, and encode the second label content to obtain a second label feature; Mapping the text probability distribution to a continuous space, determining a first mean square error loss based on the mapped text probability distribution and the first label feature, and determining a second mean square error loss based on the denoising sub-feature and the second label feature; The first mean square error loss and the second mean square error loss are weightedly summed to obtain a target loss, and parameters of the diffusion model are adjusted based on the target loss.

11. The content generation method according to claim 10, characterized in that: Mapping the text probability distribution to a continuous space includes: Normalizing the text probability distribution; The first sample features are weighted averaged according to the normalized text probability distribution to map the text probability distribution to a continuous space.

12. A content generating device, characterized in that: include: A content acquisition module, used to acquire first input content of a first modality and second input content of a second modality; an encoding module, configured to encode the first input content to obtain a first target feature, encode the second input content to obtain a second target feature, and insert the second target feature into the first target feature to obtain a fused target feature, wherein the fused target feature includes a plurality of encoding tags; A noise adding module, configured to add noise to the fused target feature by masking a preset proportion of the coding marks in the fused target feature, so as to obtain a noisy target feature; A denoising module is used to input the noisy target features into a diffusion model for denoising based on bidirectional attention to obtain denoised target features, and generate first target content of the first modality and second target content of the second modality based on the denoised target features.

13. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the content generation method according to any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the content generation method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the content generation method according to any one of claims 1 to 11 is implemented.

Images