Multimedia resource generation method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to a multimedia resource generation method and device, electronic equipment and storage medium. The method comprises: obtaining a first text prompt word for multimedia resource generation and a reference multimedia resource; performing semantic extraction and multimedia detail extraction on the reference multimedia resource based on a double encoder to obtain first multimedia semantic features and first multimedia detail features; concatenating first text features corresponding to the first text prompt word, the first multimedia semantic features and the first multimedia detail features to obtain first sequence concatenation features; inputting the first sequence concatenation features into a target Transformer model for feature interaction processing to obtain first target interaction features; and inputting preset noise information and the first target interaction features into a target diffusion model for multimedia resource generation processing to obtain a first generated multimedia resource. According to the technical solutions provided by the present disclosure, more diverse multimedia resource generation scenarios can be adapted.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a method, device, electronic device, and storage medium for generating multimedia resources. Background Art

[0002] With the rapid development of artificial intelligence generated content (AIGC), multimodal controllable image generation has been widely used. In related technologies, it is generally chosen to discretize images into token sequences and incorporate them into the vocabulary of the language model for joint prediction; or to use continuous representation methods to achieve joint generation of images and text through continuous feature spaces. However, these will make it difficult to efficiently align image representation and text representation under a unified framework, resulting in low fidelity of generated multimedia resources (such as generated images) and inability to adapt to the increasingly rich multimedia resource generation scenarios. In addition, in the generation technology based on the diffusion model, an adapter (Adapter-based) structure is introduced to achieve the purpose of control by combining structural clues such as edge maps, depth information, and key points, but such methods that rely on independent adapters usually face problems such as task isolation, redundant parameters, and weak multi-condition combination capabilities. Summary of the Invention

[0003] The present disclosure provides a multimedia resource generation method, apparatus, electronic device, and storage medium, which can at least improve the fidelity of generated multimedia resources and adapt to more diverse multimedia resource generation scenarios. The technical solutions of the present disclosure are as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a method for generating multimedia resources is provided, comprising:

[0005] Obtaining a first text prompt word and a reference multimedia resource for generating a multimedia resource;

[0006] Performing semantic extraction and multimedia detail extraction on the reference multimedia resource based on a dual encoder to obtain a first multimedia semantic feature and a first multimedia detail feature;

[0007] splicing the first text feature, the first multimedia semantic feature, and the first multimedia detail feature corresponding to the first text prompt word to obtain a first sequence splicing feature;

[0008] Inputting the first sequence concatenation features into the target Transformer model for feature interaction processing to obtain a first target interaction feature;

[0009] The preset noise information and the first target interaction feature are input into a target diffusion model to perform multimedia resource generation processing to obtain a first generated multimedia resource.

[0010] In one possible implementation, the dual encoder includes a multimedia semantic encoder and a multimedia detail encoder; and the step of performing semantic extraction and multimedia resource detail extraction on the reference multimedia resource based on the dual encoder to obtain a first multimedia semantic feature and a first multimedia detail feature includes:

[0011] Inputting the reference multimedia resource into the multimedia semantic encoder to extract semantic features to obtain the first multimedia semantic features;

[0012] The reference multimedia resource is input into the multimedia detail encoder to extract the first multimedia detail feature to obtain the first multimedia detail feature.

[0013] In a possible implementation, the method further includes:

[0014] Get learnable vector features;

[0015] The step of splicing the first text feature, the first multimedia semantic feature, and the first multimedia detail feature corresponding to the first text prompt word to obtain a first sequence splicing feature includes:

[0016] The first text feature, the first multimedia semantic feature, the first multimedia detail feature, and the learnable vector feature are spliced ​​together to obtain the first sequence splicing feature.

[0017] In one possible implementation, inputting the first sequence concatenation feature into a target Transformer model for feature interaction processing to obtain a first target interaction feature includes:

[0018] Inputting the first sequence concatenated features into the target Transformer model for feature interaction processing, and outputting a first interaction feature;

[0019] Extracting an interaction feature corresponding to the learnable vector feature from the first interaction features as the first target interaction feature;

[0020] Among them, the first target interaction feature is used to summarize and represent the second interaction feature; the second interaction feature is the interaction feature in the first interaction feature corresponding to the first text feature, the first multimedia semantic feature and the first multimedia detail feature.

[0021] In a possible implementation, inputting the preset noise information and the first target interaction feature into a target diffusion model to perform multimedia resource generation processing to obtain the first generated multimedia resource includes:

[0022] Inputting the first target interaction feature into the connection layer model to perform feature dimension conversion to obtain multimedia generation condition feature;

[0023] The multimedia generation condition characteristics and the preset noise information are input into the target diffusion model to perform multimedia resource generation processing to obtain the first generated multimedia resource.

[0024] In a possible implementation, the method further includes:

[0025] Obtaining a second text feature corresponding to the second text prompt word;

[0026] Performing semantic extraction and multimedia detail extraction on the reference multimedia resource and the first generated multimedia resource based on the dual encoder to obtain a second multimedia semantic feature and a second multimedia detail feature;

[0027] splicing the second text feature, the second multimedia semantic feature, and the second multimedia detail feature to obtain a second sequence splicing feature;

[0028] Inputting the second sequence concatenation feature into the target Transformer model for feature interaction processing to obtain a second target interaction feature;

[0029] The preset noise information and the second target interaction feature are input into the target diffusion model to perform multimedia resource generation processing to obtain a second generated multimedia resource.

[0030] In a possible implementation, inputting the preset noise information and the first target interaction feature into a target diffusion model to perform multimedia resource generation processing to obtain the first generated multimedia resource includes:

[0031] Inputting the preset noise information and the first target interaction feature into the target diffusion model to perform multimedia resource generation processing, and outputting a first generated multimedia feature;

[0032] The first generated multimedia feature is input into a feature decoder for decoding to obtain the first generated multimedia resource.

[0033] In a possible implementation, the method further includes:

[0034] Obtaining sample prompt words, sample multimedia resources, and generating multimedia tag information;

[0035] Performing semantic extraction and multimedia detail extraction on the sample multimedia resource based on a dual encoder to obtain sample multimedia semantic features and sample multimedia detail features;

[0036] splicing the sample text features corresponding to the sample text prompt words, the sample multimedia semantic features, and the sample multimedia detail features to obtain a sample sequence splicing feature;

[0037] Input the sample sequence splicing features into a preset Transformer model for feature interaction processing to obtain sample interaction features;

[0038] Inputting the preset noise information and the sample interaction characteristics into a preset diffusion model to perform multimedia resource generation processing to obtain sample-generated multimedia information;

[0039] Generating multimedia information based on the sample and the generated multimedia tag information to obtain loss information;

[0040] Parameters of the preset Transformer model and the preset diffusion model are adjusted according to the loss information until an iteration end condition is met, and the preset Transformer model corresponding to the time when the iteration end condition is met is used as the target Transformer model, and the preset diffusion model corresponding to the time when the iteration end condition is met is used as the target diffusion model.

[0041] According to a second aspect of an embodiment of the present disclosure, there is provided a multimedia resource generating apparatus, comprising:

[0042] A multimodal information acquisition module is configured to acquire a first text prompt word for multimedia resource generation and a reference multimedia resource;

[0043] a dual encoding module configured to perform semantic extraction and multimedia detail extraction on the reference multimedia resource based on a dual encoder to obtain a first multimedia semantic feature and a first multimedia detail feature;

[0044] a feature splicing module configured to splice the first text feature, the first multimedia semantic feature, and the first multimedia detail feature corresponding to the first text prompt word to obtain a first sequence of spliced ​​features;

[0045] a feature interaction module configured to input the first sequence concatenated features into a target Transformer model for feature interaction processing to obtain a first target interaction feature;

[0046] The multimedia resource generation module is configured to execute multimedia resource generation processing by inputting preset noise information and the first target interaction feature into a target diffusion model to obtain a first generated multimedia resource.

[0047] In one possible implementation, the dual encoder includes a multimedia semantic encoder and a multimedia detail encoder; and the dual encoding module includes:

[0048] a semantic encoding unit configured to input the reference multimedia resource into the multimedia semantic encoder to extract semantic features and obtain the first multimedia semantic features;

[0049] The detail encoding unit is configured to input the reference multimedia resource into the multimedia detail encoder to extract the first multimedia detail feature, thereby obtaining the first multimedia detail feature.

[0050] In a possible implementation, the apparatus further includes:

[0051] A learnable vector feature acquisition module is configured to execute acquisition of learnable vector features;

[0052] The feature splicing module is further configured to splice the first text feature, the first multimedia semantic feature, the first multimedia detail feature and the learnable vector feature to obtain the first sequence splicing feature.

[0053] In a possible implementation, the feature interaction module includes:

[0054] a feature interaction unit configured to input the first sequence concatenated features into a target Transformer model for feature interaction processing, and output a first interaction feature;

[0055] an interaction feature extraction unit, configured to extract an interaction feature corresponding to the learnable vector feature from the first interaction feature as the first target interaction feature;

[0056] Among them, the first target interaction feature is used to summarize and represent the second interaction feature; the second interaction feature is the interaction feature in the first interaction feature corresponding to the first text feature, the first multimedia semantic feature and the first multimedia detail feature.

[0057] In a possible implementation, the multimedia resource generation module includes:

[0058] a feature dimension conversion unit configured to perform feature dimension conversion on the first target interaction feature input into a connection layer model to obtain a multimedia generation condition feature;

[0059] The multimedia resource generation unit is configured to input the multimedia generation condition characteristics and the preset noise information into the target diffusion model to perform multimedia resource generation processing to obtain the first generated multimedia resource.

[0060] In a possible implementation, the apparatus further includes:

[0061] A text feature acquisition module is configured to acquire a second text feature corresponding to a second text prompt word;

[0062] The dual encoding module is further configured to perform semantic extraction and multimedia detail extraction on the reference multimedia resource and the first generated multimedia resource based on the dual encoder to obtain a second multimedia semantic feature and a second multimedia detail feature;

[0063] The feature splicing module is further configured to perform splicing of the second text feature, the second multimedia semantic feature, and the second multimedia detail feature to obtain a second sequence splicing feature;

[0064] The feature interaction module is further configured to input the second sequence concatenation feature into the target Transformer model for feature interaction processing to obtain a second target interaction feature;

[0065] The multimedia resource generation module is further configured to input the preset noise information and the second target interaction feature into the target diffusion model to perform multimedia resource generation processing to obtain a second generated multimedia resource.

[0066] In a possible implementation, the multimedia resource generation module includes:

[0067] a multimedia feature generation acquisition unit configured to input the preset noise information and the first target interaction feature into the target diffusion model to perform multimedia resource generation processing and output a first generated multimedia feature;

[0068] The feature decoding unit is configured to input the first generated multimedia feature into a feature decoder for decoding to obtain the first generated multimedia resource.

[0069] In a possible implementation, the apparatus further includes:

[0070] A sample data acquisition module is configured to acquire sample prompt words, sample multimedia resources, and generate multimedia tag information;

[0071] The sample encoding module is further configured to perform semantic extraction and multimedia detail extraction on the sample multimedia resource based on the dual encoder to obtain sample multimedia semantic features and sample multimedia detail features;

[0072] a sample sequence splicing module configured to splice the sample text features corresponding to the sample text prompt words, the sample multimedia semantic features, and the sample multimedia detail features to obtain a sample sequence splicing feature;

[0073] A sample interaction feature acquisition module is configured to input the sample sequence splicing features into a preset Transformer model for feature interaction processing to obtain sample interaction features;

[0074] a sample-generated multimedia information acquisition module configured to input the preset noise information and the sample interaction characteristics into a preset diffusion model to perform multimedia resource generation processing to obtain sample-generated multimedia information;

[0075] a loss determination module configured to generate multimedia information based on the sample and generate multimedia tag information to obtain loss information;

[0076] A model parameter adjustment module is configured to perform parameter adjustment on the preset Transformer model and the preset diffusion model according to the loss information until an iteration end condition is met, and the preset Transformer model corresponding to the iteration end condition is used as the target Transformer model, and the preset diffusion model corresponding to the iteration end condition is used as the target diffusion model.

[0077] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a method as described in any one of the above-mentioned first aspects.

[0078] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any method described in the first aspect of the embodiment of the present disclosure.

[0079] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by a processor, enable a computer to execute any one of the methods according to the first aspect of the embodiment of the present disclosure.

[0080] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0081] Using a dual encoder, semantic and multimedia detail extraction is performed on the reference multimedia resource to obtain a first multimedia semantic feature and a first multimedia detail feature. The first text feature corresponding to the first text prompt, the first multimedia semantic feature, and the first multimedia detail feature are concatenated to obtain a first sequence concatenation feature. The first sequence concatenation feature is input into a target Transformer model for feature interaction processing to obtain a first target interaction feature. Preset noise information and the first target interaction feature are input into a target diffusion model for multimedia resource generation to obtain a first generated multimedia resource. In multimedia resource generation scenarios using text and reference multimedia resources, the target Transformer model is used to unify the multimodal conditional features to obtain an integrated conditional feature, which is then input into the diffusion model for multimedia resource generation. This architecture allocates the burden of conditional understanding and integration to the target Transformer model, while the burden of generating high-fidelity multimedia resources is allocated to the target diffusion model, achieving both division of labor and efficient collaboration. Combined with the dual encoder's extraction of multimedia details, the target diffusion model can achieve high-fidelity multimedia resource generation, thus adapting to a wider variety of multimedia resource generation scenarios.

[0082] Furthermore, using sequence concatenation, the target Transformer model aligns text, semantics, and detail features, ensuring that multimodal inputs are processed and fed into the generative model with unified parameters. This fully leverages the contextual learning capabilities of the attention mechanism. Furthermore, since all inputs are fused within the same target Transformer model, conflicts between different inputs are effectively prevented, supporting multiple rounds of editing interactions and the mutual promotion of multimodal tasks. Furthermore, by dynamically fusing multimodal conditions in the sequence feature dimension, the method eliminates the need for additional adaptation parameters, significantly reducing computational overhead and avoiding the parameter inflation problem of adapter methods. This method achieves lightweight deployment while ensuring generation quality.

[0083] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0085] Figure 1 It is a schematic diagram showing an application environment according to an exemplary embodiment.

[0086] Figure 2 The figure is a flowchart of a method for generating multimedia resources according to an exemplary embodiment.

[0087] Figure 3 A method flow architecture for generating multimedia resources according to an exemplary embodiment is shown. Figure 1 .

[0088] Figure 4 A method flow architecture for generating multimedia resources according to an exemplary embodiment is shown. Figure 2 .

[0089] Figure 5 The figure is a training flowchart of a target Transformer model and a target diffusion model according to an exemplary embodiment.

[0090] Figure 6 The figure is a block diagram of a multimedia resource generating device according to an exemplary embodiment.

[0091] Figure 7 The present invention is a block diagram of an electronic device for generating multimedia resources according to an exemplary embodiment.

[0092] Figure 8 The present invention is a block diagram of an electronic device for generating multimedia resources based on an exemplary embodiment. DETAILED DESCRIPTION

[0093] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0094] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0095] Artificial Intelligence (AI) is the theory, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0096] In recent years, with the research and progress of artificial intelligence technology, artificial intelligence technology has been widely used in many fields. The solutions provided in the embodiments of this application involve computer vision technology, machine learning / deep learning and other technologies, which are specifically illustrated by the following embodiments.

[0097] See also Figure 1 , Figure 1 is a schematic diagram showing an application environment according to an exemplary embodiment. Figure 1 As shown, the application environment may include a server 01 and a terminal 02 .

[0098] In an optional embodiment, server 01 can be used for processing multimedia resource generation. Specifically, server 01 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0099] In an optional embodiment, terminal 02 can be used in conjunction with server 01 to generate multimedia resources. For example, it can provide text prompt words (e.g., first text prompt words, second text prompt words), reference multimedia resources, display generated multimedia resources (e.g., first generated multimedia resources, second generated multimedia resources), etc. Specifically, terminal 02 can include, but is not limited to, electronic devices such as smartphones, desktop computers, tablet computers, laptops, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, and smart wearable devices. Optionally, the operating system running on the electronic device can include, but is not limited to, Android, iOS, Linux, Windows, etc.

[0100] In addition, it should be noted that Figure 1 What is shown is merely one application environment of the multimedia resource generation method provided by the present disclosure.

[0101] In the embodiments of this specification, the server 01 and the terminal 02 may be connected directly or indirectly via wired or wireless communication, and this application does not impose any limitation thereto.

[0102] It should be noted that the following figure shows a possible order of steps, which is not actually limited to this order. Some steps can be executed in parallel without relying on each other. The user information (including but not limited to user device information, user personal information, user behavior information, etc.) and data (including but not limited to data for display, data for training, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0103] Before introducing the method embodiments provided in the present application, a brief introduction is first given to the application scenarios, relevant terms or nouns that may be involved in the method embodiments of the present application to facilitate understanding by technical personnel in the field of the present application.

[0104] Transformer model: A deep learning model based on the attention mechanism, widely used in natural language processing and other machine learning tasks. For example, it can be composed of self-attention mechanisms, feedforward neural networks, and multi-head attention mechanisms.

[0105] Diffusion Models: A type of generative model that reconstructs (generates) data by removing noise (e.g., random noise, Gaussian noise) step by step.

[0106] Figure 2 FIG. 1 is a flowchart of a method for generating multimedia resources according to an exemplary embodiment. Figure 2 As shown, the following steps may be included.

[0107] In step S201 , a first text prompt word and a reference multimedia resource for multimedia resource generation are obtained.

[0108] In the embodiments of this specification, multimedia resources may refer to multimedia content, for example, including but not limited to images, videos, etc., which is not limited in this disclosure.

[0109] Accordingly, the type of reference multimedia resource can correspond to the multimedia resource to be generated. For example, the reference multimedia resource can be a reference image, and the multimedia resource to be generated can be a generated image. For example, the reference multimedia resource can be used as a control (or constraint or guide) for generating multimedia resources. Taking the reference image as an example, the visual details and structure of the generated image can be guided. For example, information such as the edges, contours, and layout of the reference image can help the model determine the structure of the generated image. Furthermore, the generated image can retain certain visual features of the reference image, such as color and texture.

[0110] The first text prompt word may refer to a text used to prompt the generation of multimedia resources. For example, the first text prompt word may be a word, a sentence, etc., such as "a cute kitten".

[0111] In one possible implementation, the first text prompt and reference multimedia resource may be uploaded by the user on a terminal. For example, a social application may include a multimedia resource creation page. The user may log in to the social application through a terminal and upload the first text prompt and reference multimedia resource on the multimedia resource creation page. Accordingly, the server may obtain the first text prompt and reference multimedia resource for multimedia resource generation.

[0112] For example, Figure 3 As shown, the process architecture of the multimedia resource generation method can be divided into an input layer, a processing layer, and an output layer. Taking image generation as an example, the input layer: in order to solve the multimodal image generation task, the initial input of the system can be the first text prompt word input by the user, one or more reference images. Optionally, since the target diffusion model requires input noise as the starting point for denoising, the model also needs to input preset noise information (such as a noise image). Processing layer: For the first text prompt word, a text encoder can be used to extract text features. For the reference image, a dual encoder is used for simultaneous encoding, and sequence splicing, a target Transformer model, and a target diffusion model can be used for image generation processing. Output layer: After being processed by the decoder, the latent variable features of the generated image will be parsed and mapped to the pixel space, and finally converted into clear image generation content that meets the input conditions, that is, the generated image. Optionally, multiple rounds of image generation processing can also be performed.

[0113] In step S203, semantic extraction and multimedia detail extraction are performed on the reference multimedia resource based on the dual encoder to obtain a first multimedia semantic feature and a first multimedia detail feature.

[0114] In the embodiments of this specification, a dual encoder may refer to an encoding module including two independent encoders, which can be used to extract two types of features. For example, the two features may include, but are not limited to, semantic features and visual features. In one example, the visual features may be visual detail features, such as texture features of an image. The two independent encoders can respectively extract semantic features and multimedia detail features.

[0115] In a possible implementation, the dual encoder may include a multimedia semantic encoder and a multimedia detail encoder, which may be used to extract semantic features and detail features, respectively. By decoupling the semantic and detail feature extraction paths, high-level semantic integration and low-frequency detail preservation in image generation tasks are collaboratively optimized. Figure 4 As shown in Figure 2, the multimedia semantic encoder can embed the input reference image into blocks to generate a token sequence. The multimedia detail encoder can perform convolution and pooling on the input reference image to obtain a token sequence with flattened detail features. This effectively blends visual features from different sources.

[0116] For example, a multimedia semantic encoder can use a Vision Transformer as the backbone network, modeling global contextual dependencies through a self-attention mechanism to capture high-level semantic information such as object structure and scene logic in the image. This can be paired with a target autoregressive Transformer generator, feeding the extracted semantic features (token sequence) into the autoregressive model to generate the main planning content in sequence, ensuring semantic consistency between the generated results and the input instructions (such as object layout and attribute constraints in text descriptions).

[0117] The multimedia detail encoder can be a neural network model based on a convolutional neural network (CNN) or dense residual blocks, extracting multi-scale texture features (such as edge sharpness, material texture, illumination gradient, and other low-frequency details) through local receptive fields. This can be paired with a subsequent target diffusion model to input multimedia detail features into the diffusion process, optimizing pixel-level reconstruction quality through iterative denoising and suppressing blur and distortion in the generated image.

[0118] Accordingly, the above-mentioned semantic extraction and multimedia resource detail extraction of the reference multimedia resource based on the dual encoder to obtain the first multimedia semantic feature and the first multimedia detail feature can include: inputting the reference multimedia resource into the multimedia semantic encoder to extract the semantic feature to obtain the first multimedia semantic feature (for example, the first image semantic feature). Furthermore, the reference multimedia resource can be input into the multimedia detail encoder to extract the first multimedia detail feature to obtain the first multimedia detail feature, for example, the first image detail feature (such as the first image texture feature, etc.). By using the dual encoder including the multimedia semantic encoder and the multimedia detail encoder, the accuracy of semantic understanding can be improved and the multimedia details can be retained, thereby making the fidelity of subsequent multimedia resource generation higher.

[0119] In step S205 , the first text feature, the first multimedia semantic feature, and the first multimedia detail feature corresponding to the first text prompt word are spliced ​​together to obtain a first sequence of spliced ​​features.

[0120] In one possible implementation, Figure 3 As shown, a text encoder can be used to encode the first text prompt word to obtain a first text feature. In one example, referring to Figure 4 The text encoder may include a text tokenizer and a text embedding layer for tokenization and vectorization. For example, the text tokenizer may be used to split the first text prompt word into discrete units (tokens). The embedding layer may also be used to convert the text discrete units (tokens) into trainable vectors, such as the first text feature.

[0121] Optionally, the first text prompt word may be in a natural language format. Based on this, the first text prompt word may be subjected to text preprocessing, for example, cleaning and standardization of the first text prompt word, such as removing special symbols and redundant spaces, and normalizing abbreviations. Furthermore, the first text prompt word may be encoded using a text encoder to obtain a first text feature.

[0122] In a possible implementation, the first text feature, the first multimedia semantic feature, and the first multimedia detail feature corresponding to the first text prompt word may be spliced ​​together, for example: Figure 3 The sequence features shown in are spliced ​​together to obtain the first sequence splicing feature. Figure 4 The sequence splicing shown can splice the first text feature, the first multimedia semantic feature and the first multimedia detail feature together to obtain a first sequence splicing feature.

[0123] In step S207, the first sequence splicing feature is input into the target Transformer model for feature interaction processing to obtain the first target interaction feature.

[0124] In a possible implementation, the first sequence splicing feature can be input into the target Transformer model for feature interaction processing to obtain the first target interaction feature.

[0125] In an example, the target Transformer model can include an autoregressive Transformer model, and the present disclosure does not limit this. Correspondingly, the first sequence splicing feature can be input into the autoregressive Transformer model for feature interaction processing, and the input conditions are mixed using the attention mechanism to integrate and extract key information to obtain the first target interaction feature.

[0126] Exemplarily, the autoregressive Transformer model can involve several key modules: the causal attention mechanism and 3D RoPE. Among them, the causal attention mechanism can perform information interaction in the sequence dimension, extract effective information, and integrate all input conditions. The causal attention mechanism ensures the temporal causality of the autoregressive generation process, that is, the prediction of the current Token only depends on historical information, avoiding leakage of future information. After converting the input sequence (such as the first sequence splicing feature) into Q, K, V feature matrices through a transformation matrix, perform the attention calculation operation:

[0127]

[0128] Among them, the elements of matrix M determine whether the attention score of position i to position j is masked. When i≥j (the current position i focuses on positions no more than its own position j and can only use the content of the generated and historical positions), the mask value is 0 and does not affect the attention calculation; when i<j (if the current position i focuses on the "future" position j, it is filled with -∞), after Softmax, the weights of these positions will approach 0, achieving the effect of "masking the future". Softmax() is an activation function used for normalization processing. Q refers to the query matrix, which represents a set of vectors for "querying" and is used to actively match and query information at other positions and is obtained by linear transformation of the input sequence. K refers to the key matrix, which is a set of vectors for "keys" and is used to calculate the similarity with Q. It is also linearly transformed from the input sequence and usually has the same dimension as Q. is the transpose of K, and after transposing, it performs matrix multiplication with Q to calculate the similarity scores between different positions. is the dimension (i.e., the vector length) of K (or Q because their dimensions match), and scales the result.

[0129] Through the attention mechanism, all tokens in the sequence interact with each other and extract effective information. The causal attention mechanism is more computationally efficient than the bidirectional attention mechanism.

[0130] Optionally, in order to effectively represent the position of each token in the sequence and allow the autoregressive Transformer model to perceive the position of each token, a 3D RoPE is used to decompose the position embedding into three components: time, height, and width to effectively model the spatial and temporal information of multimodal data. For example, for text input, the same position ID can be used for the three components of time, height, and width. For image input, the time ID can remain unchanged, and the height and width IDs can be assigned according to the spatial position within the image, helping the model understand the two-dimensional spatial position. For video input, the time ID can be incremented for each frame, and the height and width IDs are assigned like images, thereby capturing dynamic changes in time and space.

[0131] In step S209 , the preset noise information and the first target interaction feature are input into the target diffusion model to perform multimedia resource generation processing to obtain a first generated multimedia resource.

[0132] In one example, the target diffusion model can be a pre-trained diffusion model, which is not limited in this disclosure. Preset noise information and the first target interaction feature can be input into the target diffusion model to generate multimedia resources, thereby obtaining a first generated multimedia resource. Specifically, the first generated multimedia resource, such as a first generated image or a first generated video, can be directly output.

[0133] In a possible implementation, the output of the target diffusion model during the training process may be generated multimedia features, based on which the generated multimedia features may be further decoded using a feature decoder to obtain generated multimedia resources. Figure 3 The above-mentioned inputting the preset noise information and the first target interaction feature into the target diffusion model for multimedia resource generation processing to obtain the first generated multimedia resource may include: inputting the preset noise information and the first target interaction feature into the target diffusion model for multimedia resource generation processing, and outputting the first generated multimedia feature (for example Figure 3 Furthermore, the first generated multimedia feature can be input into a feature decoder for decoding to obtain the first generated multimedia resource (e.g. Figure 3 By setting the output of the target diffusion model to generate multimedia features, the diffusion model can focus on learning to generate representations of multimedia resources during training. This makes feature representation learning more accurate and processing simpler and more efficient.

[0134] In an optional implementation, a connection layer can be set between the target Transformer model and the diffusion model to achieve alignment of the output dimensions and input dimensions of the two models. This can improve the adaptability between models. Based on this, the above-mentioned inputting the preset noise information and the first target interaction feature into the target diffusion model for multimedia resource generation processing to obtain the first generated multimedia resource can include: inputting the first target interaction feature into the connection layer model for feature dimension conversion to obtain multimedia generation condition feature. Furthermore, the multimedia generation condition feature and the preset noise information can be input into the target diffusion model for multimedia resource generation processing to obtain the first generated multimedia resource. Wherein, the input dimension of the connection layer can be the output dimension of the target Transformer, and the output dimension can be the input dimension of the diffusion model. For example, referring to Figure 4 The diffusion model may be a flow matching diffusion model, which is not limited in the present disclosure.

[0135] By performing semantic extraction and multimedia detail extraction on the reference multimedia resource using a dual encoder, a first multimedia semantic feature and a first multimedia detail feature are obtained. The first text feature corresponding to the first text prompt word, the first multimedia semantic feature, and the first multimedia detail feature are concatenated to obtain a first sequence concatenation feature. The first sequence concatenation feature is input into a target Transformer model for feature interaction processing to obtain a first target interaction feature. Preset noise information and the first target interaction feature are input into a target diffusion model for multimedia resource generation processing to obtain a first generated multimedia resource. In multimedia resource generation scenarios using text and reference multimedia resources, the target Transformer model is used to unify the multimodal conditional features to obtain an integrated conditional feature, which is then input into the diffusion model for multimedia resource generation. This architecture allocates the pressure of conditional understanding and integration to the target Transformer model, while the pressure of high-fidelity image generation is allocated to the target diffusion model, achieving division of labor and efficient collaboration, and adapting to a wider variety of multimedia resource generation scenarios.

[0136] In addition, sequence splicing is used to align text, semantics, and detail features through the target Transformer model, ensuring that multimodal inputs are processed with unified parameters and input into the generative model, making full use of the contextual learning ability of the attention mechanism. At the same time, since all inputs are fused in the same target Transformer model, conflicts between different inputs can be effectively prevented, and multiple rounds of editing interactions and mutual promotion between multimodal tasks can be supported.

[0137] Reference Figure 3In one possible implementation, further multimedia resources can be generated based on the first generated multimedia resource, achieving multiple rounds of generation, so that the generation of multimedia resources can be easily adjusted, making the generation of multimedia resources more flexible and efficient. Based on this, the method may also include:

[0138] The second text feature corresponding to the second text prompt word is obtained. This process can refer to the processing method of the first text prompt word, and will not be repeated here.

[0139] Furthermore, based on the dual encoder, semantic extraction and multimedia detail extraction can be performed on the reference multimedia resource and the first generated multimedia resource to obtain a second multimedia semantic feature and a second multimedia detail feature; the second text feature, the second multimedia semantic feature and the second multimedia detail feature are spliced ​​to obtain a second sequence splicing feature; and the second sequence splicing feature can be input into the target Transformer model for feature interaction processing to obtain a second target interaction feature; and the preset noise information and the second target interaction feature can be input into the target diffusion model for multimedia resource generation processing to obtain a second generated multimedia resource. The specific processing method here can refer to the above steps S203~S209 and will not be repeated here.

[0140] like Figure 3 As shown, if the first multimedia resource generation process is considered as the i-th generation process, the multimedia resource generation based on the second text prompt word and the first multimedia resource generation can be considered as the i+1-th generation process. This allows multiple rounds of multimedia resource generation. The specific processing content of the second multimedia resource generation process can be referred to the first multimedia resource generation process, and will not be repeated here.

[0141] In an alternative embodiment, referring to Figure 4 , the learnable vector feature may or may not participate in sequence splicing. If it does not participate in sequence splicing, it can be as described above; if it participates in sequence splicing, it can be as described below. Accordingly, the method may further include: obtaining the learnable vector feature. Exemplarily, the learnable vector feature may be a token sequence of a preset length. The preset length may be less than the length of the token sequence after the first text feature, the first multimedia semantic feature, and the first multimedia detail feature are spliced ​​together, so that the first text feature, the first multimedia semantic feature, and the first multimedia detail feature can be compressed.

[0142] Correspondingly, the above-mentioned splicing of the first text feature, the first multimedia semantic feature and the first multimedia detail feature corresponding to the first text prompt word to obtain a first sequence splicing feature may include: splicing the first text feature, the first multimedia semantic feature, the first multimedia detail feature and the learnable vector feature to obtain a first sequence splicing feature.

[0143] Furthermore, the step of inputting the first sequence concatenated features into the target Transformer model for feature interaction processing to obtain the first target interaction feature may include: inputting the first sequence concatenated features into the target Transformer model for feature interaction processing, and outputting the first interaction feature. For example, the learnable vector feature may interact with other input tokens via a self-attention layer. Furthermore, an interaction feature corresponding to the learnable vector feature may be extracted from the first interaction feature as the first target interaction feature.

[0144] The first target interaction feature can be used to summarize and characterize the second interaction feature (i.e., compress the second interaction feature); the second interaction feature is the interaction feature in the first interaction feature that corresponds to the first text feature, the first multimedia semantic feature, and the first multimedia detail feature. Figure 4 For example, the sequence of the first text feature, the first multimedia semantic feature, and the first multimedia detail feature is 5000 tokens, and the learnable vector feature is 1000 tokens, so the first sequence splicing feature can be 6000 tokens. Correspondingly, the first interaction feature can be 6000 tokens, so that the interaction feature corresponding to the 1000 tokens of the learnable vector feature can be extracted from the 6000 tokens of the first interaction feature as the first target interaction feature, so that the first target interaction feature can realize the compressed representation of the first interaction feature. For example, if the learnable vector feature is 1000 tokens and is at the tail of the sequence splicing, the tail 1000 tokens can be extracted from the first interaction feature as the first target interaction feature.

[0145] By setting the learnable vector feature, the first text feature, the first multimedia semantic feature and the first multimedia detail feature can be learned, and the token length of the learnable vector feature can be customized so that the first target interaction feature can achieve a compressed representation of the first interaction feature, thereby improving the generation efficiency of multimedia resources.

[0146] Reference Figure 5 In a possible implementation, the method may further include:

[0147] In step S501, sample prompt words and sample multimedia resources are obtained and multimedia tag information is generated.

[0148] In the embodiments of this specification, the generated multimedia tag information may be generated multimedia resources or generated multimedia features annotated by sample data, which may be used as a target for generation learning.

[0149] In one example, a plurality of sample data and generated multimedia tag information corresponding to each sample data can be extracted from a large amount of data generated based on text prompt words and multimedia resources in a historical period. For example, a sample data may include a sample prompt word, a sample multimedia resource, and annotated generated multimedia tag information.

[0150] In step S503, semantic extraction and multimedia detail extraction are performed on the sample multimedia resource based on the dual encoder to obtain sample multimedia semantic features and sample multimedia detail features;

[0151] In step S505, the sample text features, sample multimedia semantic features, and sample multimedia detail features corresponding to the sample text prompt words are spliced ​​to obtain a sample sequence splicing feature;

[0152] In step S507, the sample sequence concatenation features are input into a preset Transformer model for feature interaction processing to obtain sample interaction features;

[0153] In step S509, the preset noise information and the sample interaction features are input into a preset diffusion model to perform multimedia resource generation processing to obtain sample generation multimedia information.

[0154] The specific implementation of steps S503 to S509 here can refer to the contents of the above steps S203 to S209, which will not be repeated here.

[0155] In step S511 , multimedia information and multimedia tag information are generated based on the sample to obtain loss information.

[0156] In one possible implementation, the loss between generating multimedia information from a sample and generating multimedia tag information can be calculated based on a preset loss function. For example, the preset loss function may include a DDPM (Denoising Diffusion Probabilistic Models) loss function, a stream matching loss function, etc., which is not limited in this disclosure.

[0157] In an alternative embodiment, referring to Figure 4The diffusion model can be a flow matching diffusion model. Accordingly, model training can be performed based on the flow matching loss function. This training can define a continuous transformation path from simple distributions (such as Gaussian noise) to complex data distributions by learning a time- and state-dependent velocity field (vector field), thereby achieving high-quality multimedia resource generation. Unlike traditional diffusion models, flow matching does not rely on simulating the diffusion process, but directly regresses to an ideal velocity field, making training more stable and sampling more efficient.

[0158] Exemplarily, the flow matching loss function can be as follows:

[0159]

[0160] in, It refers to the output of the flow-matched diffusion model, i.e., the predicted velocity field; It is the ideal velocity field (i.e., generating multimedia label information), representing the true flow speed from noise to data. Uniform sampling on the interval [0,1] can be called a time step; It can represent sample data (such as sample prompt words, sample multimedia resources), distributed from the middle sampling, Generally follow Changing data distribution, in flow matching, it describes the data characteristics of different "flow" stages. The model needs to learn Next, data The corresponding "flow" speed law is obtained by Sampling multiple samples allows the model to learn velocity fields that adapt to different distribution forms. stands for Computational Expectation.

[0161] In step S513, the parameters of the preset Transformer model and the preset diffusion model are adjusted according to the loss information until the iteration end condition is met. The preset Transformer model corresponding to the iteration end condition is used as the target Transformer model, and the preset diffusion model corresponding to the iteration end condition is used as the target diffusion model.

[0162] In one possible implementation, gradient calculation can be performed based on the loss information, thereby performing gradient backpropagation to adjust the parameters of the preset Transformer model and the preset diffusion model. After adjusting the model parameters, the process can return to step S501 for the next round of training iterations. After multiple iterations of training, if the iteration end condition is met, the preset Transformer model corresponding to the time when the iteration end condition is met can be used as the target Transformer model, and the preset diffusion model corresponding to the time when the iteration end condition is met can be used as the target diffusion model.

[0163] Exemplarily, the iteration termination condition may include but is not limited to a loss threshold, an iteration number threshold, etc., which is not limited in the present disclosure.

[0164] Compared to independent training of single-task branches, the training method in the embodiments of this specification achieves end-to-end training, effectively resolving conflicts when combining multiple conditions, and achieving joint training performance gains. Furthermore, compared to adapters that only adjust local features without optimizing the backbone network, this training method achieves full network adjustments, resulting in higher fidelity in multimedia resource generation.

[0165] In an optional implementation, if learnable vector features are involved in sequence concatenation, the initial learnable vector features can be adjusted and updated based on loss information during model training to obtain learnable vector features. This allows the model to gradually learn how to more effectively capture and express key information in the input sequence, ultimately forming a representation of the input sequence content. This mechanism enables the model to better align and fuse visual and textual information, improving multimodal understanding capabilities.

[0166] The technical solution of the embodiments of this specification abandons the traditional adapter-based approach and adopts a hybrid unified structure to achieve multimodal generation, thereby realizing a unified framework for multi-task multimedia resource generation, which can bring the following beneficial effects:

[0167] Enhance task compatibility and eliminate modal conflicts: Integrate multi-task control signals through the full attention mechanism, so that multimodal information such as text and images can interact and cooperate in the same learning process, solving the information fragmentation problem caused by independent training of traditional adapter methods and significantly improving the cross-condition fusion effect.

[0168] Reduce parameter redundancy and optimize computational efficiency: Dynamically integrate control conditions in the sequence dimension to avoid the parameter inflation problem of adapter methods. This eliminates the need to introduce additional adaptation parameters, significantly reducing computational overhead and enabling lightweight deployment while ensuring generation quality.

[0169] Improve generation quality and controllability: Learn cross-modal contextual relationships based on autoregressive Transformer to achieve more natural task fusion.

[0170] Enhanced multi-task generalization capabilities: The unified framework simultaneously supports heterogeneous tasks such as plain text generation and image conditional generation, and achieves cross-task generation capabilities by sharing parameters.

[0171] Achieving logical chain reasoning capabilities: Leveraging the variable-length processing characteristics of causal attention, this approach completes intent reasoning and information integration within a unified Transformer architecture, supporting complex creative scenarios. For example, it can be efficiently deployed in scenarios such as film and television special effects generation, game asset creation, and industrial animation production.

[0172] Native support for interactive editing: The serialization characteristics of the autoregressive Transformer in the hybrid architecture naturally adapt to the needs of multi-round editing interactions and expand creative flexibility.

[0173] In summary, the solution of the present invention breaks through the limitations of existing adapter methods in multimodal vision generation, realizes more efficient, higher quality, and more flexible controllable image generation, and provides more competitive technical support for film and television production, game animation and other fields.

[0174] Figure 6 FIG1 is a block diagram of a multimedia resource generation device according to an exemplary embodiment. Figure 6 , the apparatus may include:

[0175] The multimodal information acquisition module 601 is configured to acquire a first text prompt word and a reference multimedia resource for multimedia resource generation;

[0176] The dual encoding module 603 is configured to perform semantic extraction and multimedia detail extraction on the reference multimedia resource based on the dual encoder to obtain a first multimedia semantic feature and a first multimedia detail feature;

[0177] A feature splicing module 605 is configured to splice the first text feature, the first multimedia semantic feature, and the first multimedia detail feature corresponding to the first text prompt word to obtain a first sequence of spliced ​​features;

[0178] A feature interaction module 607 is configured to input the first sequence concatenated features into a target Transformer model for feature interaction processing to obtain a first target interaction feature;

[0179] The multimedia resource generation module 609 is configured to input the preset noise information and the first target interaction feature into the target diffusion model to perform multimedia resource generation processing to obtain a first generated multimedia resource.

[0180] By performing semantic extraction and multimedia detail extraction on the reference multimedia resource using a dual encoder, a first multimedia semantic feature and a first multimedia detail feature are obtained. The first text feature corresponding to the first text prompt word, the first multimedia semantic feature, and the first multimedia detail feature are concatenated to obtain a first sequence concatenation feature. The first sequence concatenation feature is input into a target Transformer model for feature interaction processing to obtain a first target interaction feature. Preset noise information and the first target interaction feature are input into a target diffusion model for multimedia resource generation processing to obtain a first generated multimedia resource. In multimedia resource generation scenarios using text and reference multimedia resources, the target Transformer model is used to unify the multimodal conditional features to obtain an integrated conditional feature, which is then input into the diffusion model for multimedia resource generation. This architecture allocates the pressure of conditional understanding and integration to the target Transformer model, while the pressure of high-fidelity image generation is allocated to the target diffusion model, achieving division of labor and efficient collaboration, and adapting to a wider variety of multimedia resource generation scenarios.

[0181] In addition, sequence splicing is used to align text, semantics, and detail features through the target Transformer model, ensuring that multimodal inputs are processed with unified parameters and input into the generative model, making full use of the contextual learning ability of the attention mechanism. At the same time, since all inputs are fused in the same target Transformer model, conflicts between different inputs can be effectively prevented, and multiple rounds of editing interactions and mutual promotion between multimodal tasks can be supported.

[0182] In one possible implementation, the dual encoder includes a multimedia semantic encoder and a multimedia detail encoder; and the dual encoding module includes:

[0183] a semantic encoding unit configured to input the reference multimedia resource into the multimedia semantic encoder to extract semantic features and obtain the first multimedia semantic features;

[0184] The detail encoding unit is configured to input the reference multimedia resource into the multimedia detail encoder to extract the first multimedia detail feature, thereby obtaining the first multimedia detail feature.

[0185] In a possible implementation, the apparatus further includes:

[0186] A learnable vector feature acquisition module is configured to execute acquisition of learnable vector features;

[0187] The feature splicing module is further configured to splice the first text feature, the first multimedia semantic feature, the first multimedia detail feature and the learnable vector feature to obtain the first sequence splicing feature.

[0188] In a possible implementation, the feature interaction module includes:

[0189] a feature interaction unit configured to input the first sequence concatenated features into a target Transformer model for feature interaction processing, and output a first interaction feature;

[0190] an interaction feature extraction unit, configured to extract an interaction feature corresponding to the learnable vector feature from the first interaction feature as the first target interaction feature;

[0191] Among them, the first target interaction feature is used to summarize and represent the second interaction feature; the second interaction feature is the interaction feature in the first interaction feature corresponding to the first text feature, the first multimedia semantic feature and the first multimedia detail feature.

[0192] In a possible implementation, the multimedia resource generation module includes:

[0193] a feature dimension conversion unit configured to perform feature dimension conversion on the first target interaction feature input into a connection layer model to obtain a multimedia generation condition feature;

[0194] The multimedia resource generation unit is configured to input the multimedia generation condition characteristics and the preset noise information into the target diffusion model to perform multimedia resource generation processing to obtain the first generated multimedia resource.

[0195] In a possible implementation, the apparatus further includes:

[0196] A text feature acquisition module is configured to acquire a second text feature corresponding to a second text prompt word;

[0197] The dual encoding module is further configured to perform semantic extraction and multimedia detail extraction on the reference multimedia resource and the first generated multimedia resource based on the dual encoder to obtain a second multimedia semantic feature and a second multimedia detail feature;

[0198] The feature splicing module is further configured to perform splicing of the second text feature, the second multimedia semantic feature, and the second multimedia detail feature to obtain a second sequence splicing feature;

[0199] The feature interaction module is further configured to input the second sequence concatenation feature into the target Transformer model for feature interaction processing to obtain a second target interaction feature;

[0200] The multimedia resource generation module is further configured to input the preset noise information and the second target interaction feature into the target diffusion model to perform multimedia resource generation processing to obtain a second generated multimedia resource.

[0201] In a possible implementation, the multimedia resource generation module includes:

[0202] a multimedia feature generation acquisition unit configured to input the preset noise information and the first target interaction feature into the target diffusion model to perform multimedia resource generation processing and output a first generated multimedia feature;

[0203] The feature decoding unit is configured to input the first generated multimedia feature into a feature decoder for decoding to obtain the first generated multimedia resource.

[0204] In a possible implementation, the apparatus further includes:

[0205] A sample data acquisition module is configured to acquire sample prompt words, sample multimedia resources, and generate multimedia tag information;

[0206] The sample encoding module is further configured to perform semantic extraction and multimedia detail extraction on the sample multimedia resource based on the dual encoder to obtain sample multimedia semantic features and sample multimedia detail features;

[0207] a sample sequence splicing module configured to splice the sample text features corresponding to the sample text prompt words, the sample multimedia semantic features, and the sample multimedia detail features to obtain a sample sequence splicing feature;

[0208] A sample interaction feature acquisition module is configured to input the sample sequence splicing features into a preset Transformer model for feature interaction processing to obtain sample interaction features;

[0209] a sample-generated multimedia information acquisition module configured to input the preset noise information and the sample interaction characteristics into a preset diffusion model to perform multimedia resource generation processing to obtain sample-generated multimedia information;

[0210] a loss determination module configured to generate multimedia information based on the sample and generate multimedia tag information to obtain loss information;

[0211] A model parameter adjustment module is configured to perform parameter adjustment on the preset Transformer model and the preset diffusion model according to the loss information until an iteration end condition is met, and the preset Transformer model corresponding to the iteration end condition is used as the target Transformer model, and the preset diffusion model corresponding to the iteration end condition is used as the target diffusion model.

[0212] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0213] Figure 7 This is a block diagram of an electronic device for generating multimedia resources according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The electronic device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for generating multimedia resources is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0214] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0215] Figure 8 This is a block diagram of an electronic device for generating multimedia resources based on an exemplary embodiment. The electronic device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The electronic device includes a processor, a memory, and a network interface connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for generating multimedia resources is implemented.

[0216] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present disclosure, and does not constitute a limitation on the electronic device to which the scheme of the present disclosure is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0217] In an exemplary embodiment, an electronic device is further provided, including: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the multimedia resource generation method in the embodiment of the present disclosure.

[0218] In an exemplary embodiment, a computer-readable storage medium is also provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the multimedia resource generation method of the disclosed embodiment. The computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, or the like.

[0219] In an exemplary embodiment, a computer program product containing instructions is also provided. When the computer program product is run on a computer, the computer is caused to execute the method for generating multimedia resources in the embodiment of the present disclosure.

[0220] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, which can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0221] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0222] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A multimedia resource generation method, characterized in that: include: Obtaining a first text prompt word and a reference multimedia resource for generating a multimedia resource; Performing semantic extraction and multimedia detail extraction on the reference multimedia resource based on a dual encoder to obtain a first multimedia semantic feature and a first multimedia detail feature; the first multimedia detail feature represents low-frequency details of the reference multimedia resource; splicing the first text feature, the first multimedia semantic feature, and the first multimedia detail feature corresponding to the first text prompt word to obtain a first sequence splicing feature; Inputting the first sequence concatenation features into the target Transformer model for feature interaction processing to obtain a first target interaction feature; Inputting the preset noise information and the first target interaction feature into the target diffusion model to perform multimedia resource generation processing to obtain a first generated multimedia resource; Obtaining a second text feature corresponding to the second text prompt word; Performing semantic extraction and multimedia detail extraction on the reference multimedia resource and the first generated multimedia resource based on the dual encoder to obtain a second multimedia semantic feature and a second multimedia detail feature; splicing the second text feature, the second multimedia semantic feature, and the second multimedia detail feature to obtain a second sequence splicing feature; Inputting the second sequence concatenation feature into the target Transformer model for feature interaction processing to obtain a second target interaction feature; The preset noise information and the second target interaction feature are input into the target diffusion model to perform multimedia resource generation processing to obtain a second generated multimedia resource.

2. The method according to claim 1, characterized in that The dual encoder includes a multimedia semantic encoder and a multimedia detail encoder; the step of extracting semantics and extracting multimedia resource details from the reference multimedia resource based on the dual encoder to obtain a first multimedia semantic feature and a first multimedia detail feature includes: Inputting the reference multimedia resource into the multimedia semantic encoder to extract semantic features to obtain the first multimedia semantic features; The reference multimedia resource is input into the multimedia detail encoder to extract the first multimedia detail feature to obtain the first multimedia detail feature.

3. The method according to claim 1, characterized in that The method further comprises: Get learnable vector features; The step of splicing the first text feature, the first multimedia semantic feature, and the first multimedia detail feature corresponding to the first text prompt word to obtain a first sequence splicing feature includes: The first text feature, the first multimedia semantic feature, the first multimedia detail feature, and the learnable vector feature are spliced ​​together to obtain the first sequence splicing feature.

4. The method according to claim 3, characterized in that Inputting the first sequence concatenation features into the target Transformer model for feature interaction processing to obtain first target interaction features includes: Inputting the first sequence concatenated features into the target Transformer model for feature interaction processing, and outputting a first interaction feature; Extracting an interaction feature corresponding to the learnable vector feature from the first interaction features as the first target interaction feature; Among them, the first target interaction feature is used to summarize and represent the second interaction feature; the second interaction feature is the interaction feature in the first interaction feature corresponding to the first text feature, the first multimedia semantic feature and the first multimedia detail feature.

5. The method according to claim 1, wherein The step of inputting the preset noise information and the first target interaction feature into a target diffusion model to perform multimedia resource generation processing to obtain a first generated multimedia resource includes: Inputting the first target interaction feature into the connection layer model to perform feature dimension conversion to obtain multimedia generation condition feature; The multimedia generation condition characteristics and the preset noise information are input into the target diffusion model to perform multimedia resource generation processing to obtain the first generated multimedia resource.

6. The method according to claim 1, characterized in that The step of inputting the preset noise information and the first target interaction feature into a target diffusion model to perform multimedia resource generation processing to obtain a first generated multimedia resource includes: Inputting the preset noise information and the first target interaction feature into the target diffusion model to perform multimedia resource generation processing, and outputting a first generated multimedia feature; The first generated multimedia feature is input into a feature decoder for decoding to obtain the first generated multimedia resource.

7. The method according to claim 1 or 6, characterized in that The method further comprises: Obtaining sample prompt words, sample multimedia resources, and generating multimedia tag information; Performing semantic extraction and multimedia detail extraction on the sample multimedia resource based on a dual encoder to obtain sample multimedia semantic features and sample multimedia detail features; Splicing the sample text features, the sample multimedia semantic features, and the sample multimedia detail features corresponding to the sample prompt words to obtain a sample sequence splicing feature; Input the sample sequence splicing features into a preset Transformer model for feature interaction processing to obtain sample interaction features; Inputting the preset noise information and the sample interaction characteristics into a preset diffusion model to perform multimedia resource generation processing to obtain sample-generated multimedia information; Generating multimedia information based on the sample and the generated multimedia tag information to obtain loss information; Parameters of the preset Transformer model and the preset diffusion model are adjusted according to the loss information until an iteration end condition is met, and the preset Transformer model corresponding to the time when the iteration end condition is met is used as the target Transformer model, and the preset diffusion model corresponding to the time when the iteration end condition is met is used as the target diffusion model.

8. A multimedia resource generating device, characterized in that: include: A multimodal information acquisition module is configured to acquire a first text prompt word for multimedia resource generation and a reference multimedia resource; a dual encoding module configured to perform semantic extraction and multimedia detail extraction on the reference multimedia resource based on a dual encoder to obtain a first multimedia semantic feature and a first multimedia detail feature; the first multimedia detail feature represents a low-frequency detail of the reference multimedia resource; a feature splicing module configured to splice the first text feature, the first multimedia semantic feature, and the first multimedia detail feature corresponding to the first text prompt word to obtain a first sequence of spliced ​​features; a feature interaction module configured to input the first sequence concatenated features into a target Transformer model for feature interaction processing to obtain a first target interaction feature; A multimedia resource generation module is configured to input preset noise information and the first target interaction feature into a target diffusion model to perform multimedia resource generation processing to obtain a first generated multimedia resource; A text feature acquisition module is configured to acquire a second text feature corresponding to a second text prompt word; The dual encoding module is further configured to perform semantic extraction and multimedia detail extraction on the reference multimedia resource and the first generated multimedia resource based on the dual encoder to obtain a second multimedia semantic feature and a second multimedia detail feature; The feature splicing module is further configured to perform splicing of the second text feature, the second multimedia semantic feature, and the second multimedia detail feature to obtain a second sequence splicing feature; The feature interaction module is further configured to input the second sequence splicing feature into the target Transformer model for feature interaction processing to obtain a second target interaction feature; The multimedia resource generation module is further configured to input the preset noise information and the second target interaction feature into the target diffusion model to perform multimedia resource generation processing to obtain a second generated multimedia resource.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the multimedia resource generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the multimedia resource generation method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, enable a computer to execute the multimedia resource generation method according to any one of claims 1 to 7.

Images