Method and device for understanding and generating multi-modal content

Abstract

The embodiment of the invention discloses a multi-modal content understanding and generating method and device. In the embodiment of the invention, image-text content input by a user is acquired, and the image-text content input by the user comprises an image part and a text part; generating a feature sequence according to the image-text content input by the user; and inputting the feature sequence into a pre-trained hybrid autoregressive and diffusion architecture model to generate target content, the target content including a target image and / or a target text. According to the method, the image-text content input by the user is processed by adopting the hybrid autoregression and diffusion architecture model, so that the quality and the generation efficiency of the target image and the target text can be improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a method and apparatus for multimodal content understanding and generation. Background Technology

[0002] With the rapid development of Large Language Models (LLMs), these models have been applied to various security risk identification scenarios. As the difficulty and complexity of security risk identification gradually increase, and the risks that are newly added and changed are numerous, rapid, and detailed, large language models need to cover many business domains and respond quickly to risk tasks of varying complexity, continuously iterating and optimizing. Therefore, a large amount of high-quality training data is required when training these large language models. The better the quality and the greater the quantity of the training data, the higher the performance of the large language model. The training data can be text and image data.

[0003] In existing technologies, text-to-image understanding models that use common image-to-text and text-to-text supervision methods generate text. However, these models do not pay enough attention to complex, hierarchical, and fine-grained images, resulting in poor text quality. Alternatively, common image-generating models based on diffusion architectures are limited by the understanding capabilities of their text encoders, making it difficult to achieve precise control over text-to-image conversion, resulting in poor-quality images. Therefore, the quality of text-to-image data generated by existing technologies is generally poor.

[0004] In conclusion, improving the quality and generation efficiency of text and image data is a problem that needs to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a method and apparatus for multimodal content understanding and generation, which uses a hybrid autoregressive and diffusion architecture model to process user-input image and text content, thereby improving the quality and generation efficiency of target images and target text.

[0006] In a first aspect, embodiments of the present invention provide a method for multimodal content understanding and generation, the method comprising: acquiring user-input image and text content, wherein the user-input image and text content includes an image portion and a text portion; generating a feature sequence based on the user-input image and text content; and inputting the feature sequence into a pre-trained hybrid autoregressive and diffusion architecture model to generate target content, wherein the target content includes a target image and / or target text.

[0007] Optionally, generating a feature sequence based on the user input content specifically includes: generating contrastive language image pre-trained CLIP features based on the image portion, and generating a text feature vector based on the text portion; concatenating the CLIP features, the text feature vector, and the learnable features to generate the feature sequence.

[0008] Optionally, generating contrastive language image pre-trained CLIP features based on the image portion specifically includes: inputting the image portion into a pre-trained visual encoder to generate the CLIP features.

[0009] Optionally, generating a text feature vector based on the text portion specifically includes: performing text segmentation on the text portion to determine multiple word units; mapping each word unit through an embedding layer to generate an intermediate text feature vector; and generating the text feature vector based on the multiple intermediate text feature vectors.

[0010] Optionally, the step of inputting the feature sequence into a pre-trained hybrid autoregressive and diffusion architecture model to generate target content specifically includes: inputting the feature sequence into the autoregressive model part of the pre-trained hybrid autoregressive and diffusion architecture model to generate a target feature sequence; and generating the target content based on the target feature sequence.

[0011] Optionally, the step of inputting the feature sequence into a pre-trained hybrid autoregressive and diffusion architecture model to generate target content specifically includes: inputting the feature sequence into the autoregressive model part of the pre-trained hybrid autoregressive and diffusion architecture model to generate a predicted CLIP feature sequence; inputting the predicted CLIP feature sequence into the diffusion model part of the hybrid autoregressive and diffusion architecture model to generate a target feature sequence; and generating the target content based on the target feature sequence.

[0012] Optionally, the training process of the hybrid autoregressive and diffusion architecture model includes: acquiring sample data, wherein the sample data includes historical input text and image content and historical output feature sequences; generating historical input feature sequences based on the historical input text and image content in the sample data; and training the hybrid autoregressive and diffusion architecture model based on the historical input feature sequences and the historical output feature sequences.

[0013] Optionally, the method further includes: optimizing the hybrid autoregressive and diffusion architecture model based on the loss function.

[0014] Optionally, the loss function includes cross-entropy loss, reconstruction loss, and self-feedback loss.

[0015] Secondly, embodiments of the present invention provide a multimodal content understanding and generation apparatus, the apparatus comprising: an acquisition unit for acquiring user-input image and text content, wherein the user-input image and text content includes an image portion and a text portion; a first generation unit for generating a feature sequence based on the user-input image and text content; a second generation unit for inputting the feature sequence into a pre-trained hybrid autoregressive and diffusion architecture model to generate a target feature sequence; and a third generation unit for generating target content based on the target feature sequence, wherein the target content includes a target image and / or target text.

[0016] Optionally, the first generation unit is specifically used to: generate contrastive language image pre-trained CLIP features based on the image portion, and generate text feature vectors based on the text portion; and concatenate the CLIP features, the text feature vectors, and the learnable features to generate the feature sequence.

[0017] Optionally, the first generation unit is specifically used to: input the image portion into a pre-trained visual encoder to generate the CLIP feature.

[0018] Optionally, the first generation unit is specifically used to: perform text segmentation on the text portion to determine multiple word units; map each word unit through an embedding layer to generate an intermediate text feature vector; and generate the text feature vector based on the multiple intermediate text feature vectors.

[0019] Optionally, the second generation unit is specifically used to: input the feature sequence into the autoregressive model part of a pre-trained hybrid autoregressive and diffusion architecture model to generate a target feature sequence; and generate the target content based on the target feature sequence.

[0020] Optionally, the second generation unit is specifically used to: input the feature sequence into the autoregressive model part of a pre-trained hybrid autoregressive and diffusion architecture model to generate a predicted CLIP feature sequence; input the predicted CLIP feature sequence into the diffusion model part of the hybrid autoregressive and diffusion architecture model to generate a target feature sequence; and generate the target content based on the target feature sequence.

[0021] Optionally, the training process of the hybrid autoregressive and diffusion architecture model includes: the acquisition unit is used to: acquire sample data, wherein the sample data includes historical input text and image content and historical output feature sequences; the first generation unit is further used to: generate historical input feature sequences based on the historical input text and image content in the sample data; the device further includes: a training unit, used to train the hybrid autoregressive and diffusion architecture model based on the historical input feature sequences and the historical output feature sequences.

[0022] Optionally, the apparatus further includes an optimization unit for optimizing the hybrid autoregressive and diffusion architecture model based on a loss function.

[0023] Optionally, the loss function includes cross-entropy loss, reconstruction loss, and self-feedback loss.

[0024] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in the first aspect or any one of the possible methods of the first aspect.

[0025] Fourthly, embodiments of the present invention provide a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method as described in the first aspect or any one of the possibilities of the first aspect.

[0026] In this embodiment of the invention, user-input image and text content is acquired, wherein the user-input image and text content includes an image portion and a text portion; a feature sequence is generated based on the user-input image and text content; the feature sequence is input into a pre-trained hybrid autoregressive and diffusion architecture model to generate target content, wherein the target content includes a target image and / or target text. By using the above method and employing a hybrid autoregressive and diffusion architecture model to process the user-input image and text content, the quality and generation efficiency of the target image and target text can be improved. Attached Figure Description

[0027] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which: Figure 1 This is a flowchart of a method for multimodal content understanding and generation in an embodiment of the present invention; Figure 2 This is a schematic diagram of a user inputting text and image content in an embodiment of the present invention; Figure 3 This is another schematic diagram of user input of graphic content in an embodiment of the present invention; Figure 4 This is a flowchart of a method for generating feature sequences according to an embodiment of the present invention; Figure 5 This is a flowchart of a training method for a hybrid autoregressive and diffusion architecture model in an embodiment of the present invention; Figure 6 This is a schematic diagram of a multimodal content understanding and generation device according to an embodiment of the present invention; Figure 7 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0028] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0029] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0030] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0031] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0032] In existing technologies, image-to-text and text-to-text supervision models, such as the Qwen-VL series, are commonly used for text generation. However, these models lack effective supervision from the visual side and do not pay enough attention to complex, hierarchical, and fine-grained images, resulting in poor text quality. Alternatively, common image-generating models based on diffusion architectures are limited by the understanding capabilities of their text encoders, making it difficult to achieve precise text-to-image control and producing poor-quality images. Therefore, the quality of image-to-text data generated by existing technologies is poor. Training large language models for security risk identification with such low-quality image-to-text data will prevent these models from handling highly complex risk scenarios and from quickly identifying and addressing new and changed risk points. Therefore, improving the quality and generation efficiency of image-to-text data, and consequently improving the performance of large language models for security risk identification, is a problem that needs to be solved.

[0033] In this embodiment of the invention, to address the aforementioned problems, a method for multimodal content understanding and generation is proposed, specifically as follows: Figure 1 As shown, the method includes: Step S101: Obtain the user-input text and image content.

[0034] The user-input graphic content includes both image and text components.

[0035] In one possible implementation, the user-input image and text content is heterogeneous and multimodal, that is, the image part and the text part of the user-input image and text content are input in a mixed manner. For example, the user-input image and text content is input first and then the text part, or first the text part and then the image part, or the image part and the text part are input alternately.

[0036] The following two specific examples illustrate the user-input text and image content: Example 1: The user input of text and image content is as follows Figure 2 As shown, first enter the text "I am going to this place", then enter the image, which is a landscape picture, and finally enter the text again, which is "Travel, please make a travel plan for me, and generate corresponding text and pictures". This is just an example, and the specific text and image content entered by the user will be determined according to the actual usage.

[0037] Example 2, the user input of text and image content is as follows: Figure 3 As shown, first input image 1, then input the text "Step 1", then input image 2, then input the text "Step 2", and so on. Input image n, then input the text "Step n". This is just an example. The specific image and text content input by the user will be determined according to the actual usage.

[0038] In one possible implementation, either an image or text can be input separately, depending on the actual usage.

[0039] Step S102: Generate a feature sequence based on the user-input text and image content.

[0040] In one possible implementation, generating a feature sequence based on the user input content specifically includes, for example: Figure 4 As shown, it includes the following steps: Step S401: Generate contrastive language image pre-trained CLIP features based on the image portion, and generate text feature vectors based on the text portion.

[0041] Specifically, the step of generating Contrastive Language-Image Pre-Training (CLIP) features based on the image portion includes: inputting the image portion into a pre-trained visual encoder to generate the CLIP features; wherein the CLIP features are features obtained after training a multimodal model based on contrastive learning, and are used to compare images and text in a unified vector space to achieve flexible alignment between images and text.

[0042] In one possible implementation, the visual encoder can be a Transformer-based visual model (Vision Transformer, ViT) that encodes the two-dimensional pixel information of the image portion into a set of semantically rich CLIP image features. The above process extracts the key semantic information of the image portion and transforms it into a unified serialization format aligned with the text feature vector.

[0043] In one possible implementation, generating a text feature vector based on the text portion specifically includes: performing text segmentation on the text portion to determine multiple word units; mapping each word unit through an embedding layer to generate an intermediate text feature vector; and generating the text feature vector based on the multiple intermediate text feature vectors.

[0044] Step S402: Concatenate the CLIP features, the text feature vector, and the learnable features to generate the feature sequence.

[0045] In one possible implementation, the CLIP features include text features and image features. The text features, image features, text feature vectors, and learnable features are concatenated sequentially to generate the feature sequence. The learnable features have the same dimension as the text features and image features, and are used to guide the hybrid autoregressive and diffusion architecture model to perform specific tasks or switch between different modalities. The learnable features are special vectors learned during training, primarily serving as conditions for subsequent image generation by the hybrid autoregressive and diffusion architecture model, providing clear contextual cues for the model.

[0046] In this embodiment of the invention, steps 401 and 402 convert the user-input text and image content into a unified feature sequence composed of the CLIP features, the text feature vector, and the learnable features, laying the foundation for subsequent processing of the hybrid autoregressive and diffusion architecture model.

[0047] Step S103: Input the feature sequence into a pre-trained hybrid autoregressive and diffusion architecture model to generate target content, wherein the target content includes target images and / or target text.

[0048] In one possible implementation, in response to the target content being target text, the step of inputting the feature sequence into a pre-trained hybrid autoregressive and diffusion architecture model to generate target content specifically includes: inputting the feature sequence into the autoregressive model part of the pre-trained hybrid autoregressive and diffusion architecture model to generate a target feature sequence; and generating the target content based on the target feature sequence. The autoregressive model part is composed of an autoregressive model, which is a sequence model. When predicting the next element in the sequence, the autoregressive model relies on all the elements that have already been generated. The autoregressive model performs well in tasks such as text generation.

[0049] In this embodiment of the invention, the autoregressive model predicts the next element in the sequence at each time step based on all the feature vectors it has previously received and generated. For the text generation task, the autoregressive model uses an autoregressive approach to predict text feature vectors word by word. The text feature vectors are the target feature sequence. Target text is generated based on the target feature sequence. The autoregressive model can be constructed based on the Transformer network.

[0050] In one possible implementation, in response to the target content being a target image, the step of inputting the feature sequence into a pre-trained hybrid autoregressive and diffusion architecture model to generate the target content specifically includes: inputting the feature sequence into the autoregressive model part of the pre-trained hybrid autoregressive and diffusion architecture model to generate a predicted CLIP feature sequence; inputting the predicted CLIP feature sequence into the diffusion model part of the hybrid autoregressive and diffusion architecture model to generate a target feature sequence; and generating the target content based on the target feature sequence; wherein the diffusion model part is composed of a diffusion model, which is a type of generative model that generates high-quality images from random noise through a stepwise denoising process, and the diffusion model has powerful capabilities in image generation and editing.

[0051] In this embodiment of the invention, for the target image generation task, the output of the hidden layer features of the autoregressive model part is the predicted CLIP feature sequence. The diffusion model part, which can also be called the feature generation network, is used to denoise the predicted CLIP feature sequence, remove Gaussian noise, and generate the target feature sequence. Finally, the target image is generated through the image generation network.

[0052] In this embodiment of the invention, the hybrid mechanism of autoregressive and diffusion models enables the hybrid autoregressive and diffusion architecture model to flexibly adapt to the generation characteristics of different modalities. The hybrid autoregressive and diffusion architecture model can also be called a unified model, which can simultaneously complete the understanding task and the generation task, rather than using multiple independent models.

[0053] In one possible implementation, the output of the hybrid autoregressive and diffusion architecture model is a target feature sequence. This target feature sequence can be used by different task modules to perform diverse tasks. For example, the target feature sequence can be used by the image generation network to generate a target image, or by the text generation network to generate target text. The image generation network and the text generation network can be considered as a whole with the hybrid autoregressive and diffusion architecture model. The image generation network and the text generation network can also be part set after the hybrid autoregressive and diffusion architecture model, together forming the unified model, which is determined according to the actual situation.

[0054] In this embodiment of the invention, due to the introduction of the autoregressive architecture, the unified model can fully utilize complete historical context information, including deep semantic associations in the text-image interleaved input, to achieve cross-modal context awareness when generating any new content. Specifically, the unified model can generate coherent descriptions in the text modality or semantically related images in the image modality based on the input image content and text instructions, ensuring the logical coherence and contextual relevance of the output. The above mechanism enables it to deeply understand the complex relationships between text and images, achieving true unified multimodal understanding and generation.

[0055] In one possible implementation, the training process of the hybrid autoregressive and diffusion architecture model is as follows: Figure 5 As shown, it specifically includes: Step S501: Obtain sample data.

[0056] The sample data includes historical input text and image content and historical output feature sequences.

[0057] Step S502: Generate a historical input feature sequence based on the historical input image and text content in the sample data.

[0058] Step S503: Train the hybrid autoregressive and diffusion architecture model based on the historical input feature sequence and the historical output feature sequence.

[0059] In one possible implementation, the hybrid autoregressive and diffusion architecture model can be optimized according to a loss function, wherein the loss function includes cross-entropy loss, reconstruction loss, and self-feedback loss.

[0060] In one possible implementation, the cross-entropy loss The reconstruction loss used to understand the task. The self-feedback loss used for generating tasks This is the real-time feedback given to the output of the hybrid autoregressive and diffusion architecture model.

[0061] In this embodiment of the invention, the total loss is used for parameter optimization when optimizing the hybrid autoregressive and diffusion architecture model, wherein the total loss... .

[0062] In one possible implementation, the cross-entropy loss The formula is as follows: = ; The reconstruction loss The formula is as follows: = ; The self-feedback loss The formula is as follows: = ; Among them, the For the user's image input, the For the user's text input, the As learnable features, the Denotes the parameterized function of the unified model, where, The set of parameters to be optimized; the cross-entropy loss By calculating the predicted word probabilities and the actual word distribution using the model. The model parameters are optimized based on the difference between the values, where P represents the conditional probability; the reconstruction loss... By calculating the difference between the image generated by the unified model and the original image, the unified model is driven to finely reproduce the real image, thereby ensuring the fidelity of the generated image; wherein, the The generation module is based on learnable features The predicted "velocity field", the Defined from the noise floor To real images The target transformation trajectory; the self-feedback loss The unified model achieves self-improvement by setting different reward values. Here, r is any single value between [0,1], representing a comprehensive evaluation of the overall output image and text quality of the unified model. A higher r value indicates better overall output quality. R is a vector of corresponding reward values ​​for the output text and output image of the unified model, with each pixel ranging from [-1,1]. For the output text, the unified model evaluates accuracy and formatting and provides corresponding reward values. For the output image, the unified model evaluates detail satisfaction and aesthetic quality and obtains reward values. Detail satisfaction specifically refers to the similarity between the generated image and the required text. Element-by-element multiplication is performed to achieve fine-grained reward allocation; the aforementioned Let be the loss function, representing the common cross-entropy loss or diffusion loss, which can simultaneously optimize the text target. and image target .

[0063] In one possible implementation, when r·R > 0, the unified model is encouraged to retain the current generated result because the generated result is considered to be of good quality or correct; when r·R < 0, the unified model is encouraged to change the current generated result because the generated result is considered to be of poor quality or incorrect; through the introduction of the self-feedback loss, the unified model is able to self-improve its generated content without the need for additional manually labeled data.

[0064] In this embodiment of the invention, assuming the true word distribution is "The place described in the picture is Lake A, generate a travel plan for place B for you...", the model calculates the word probability predicted by the user's input text "I am going to travel to this place, please make a travel plan for me, and generate the corresponding text and picture". Based on the above true word distribution and the word probability predicted by the model, the cross-entropy loss is calculated, and then the parameters of the unified model are optimized.

[0065] In this embodiment of the invention, the cross-entropy loss, reconstruction loss, and self-feedback loss simultaneously optimize the understanding network (i.e., the autoregressive model) and the generative network. The generative network includes a feature generation network and an image generation network, and the feature generation network is constructed by a diffusion model. This achieves mutual promotion between understanding and generation, and ensures that the unified model can continuously optimize its output content.

[0066] In one possible implementation, images and text are interleaved into a unified model, and the input to the unified model is also interleaved with images and text. This input-output method lays the foundation for the introduction of visual reasoning tasks. Through self-evaluation, the unified model can verify its own generated results, forming an internal feedback loop for iterative optimization. Through task decomposition, the unified model can break down complex instructions. For example, when receiving the instruction "create a travel plan containing images and text," the model will first plan the text itinerary, and then generate corresponding scenic images step by step according to the itinerary content, rather than trying... Figure 1 All tasks can be completed in one go. For image generation with complex instructions, the unified model can also adopt the approach of first generating content 1, then generating content 2, and then merging content 1 and content 2 to achieve more controllable image generation.

[0067] In this embodiment of the invention, the unified model can quickly generate high-quality new risk image-text pairs based on new risk knowledge, high-difficulty risk knowledge, and general knowledge. By training a large number of high-quality new risk image-text pairs, the large language model in the risk control scenario can handle the challenges of high risk identification difficulty, rapid emergence of new risk points, and the need for rapid response and continuous iterative optimization.

[0068] Through the above embodiments, the unified model integrates autoregressive and diffusion models, breaking down the barriers between understanding and generation tasks through collaborative training. The unified model simultaneously optimizes both the understanding and generation networks, allowing the training of the generation network to feed back into the understanding network, prompting it to focus more on fine-grained information. Meanwhile, the understanding network provides more precise semantic guidance for generation, forming a self-optimizing technical loop. The visual reasoning module, as an internal supervision and control mechanism, enables the unified model to perform self-evaluation and task decomposition, thereby achieving more controllable and expected outputs, especially for complex image and text generation tasks. The unified model's fine-grained understanding capability can identify subtle risk points that traditional models struggle to capture, while its controllable generation capability can quickly and cost-effectively generate large amounts of risky image and text data, providing a large number of high-quality training samples for risk domain large language models.

[0069] In this embodiment of the invention, a device for multimodal content understanding and generation is provided, such as... Figure 6As shown, the system specifically includes: an acquisition unit 601, a first generation unit 602, a second generation unit 603, and a third generation unit 604; wherein, the acquisition unit 601 is used to acquire user-input image and text content, wherein the user-input image and text content includes an image part and a text part; the first generation unit 602 is used to generate a feature sequence based on the user-input image and text content; the second generation unit 603 is used to input the feature sequence into a pre-trained hybrid autoregressive and diffusion architecture model to generate a target feature sequence; the third generation unit 604 is used to generate target content based on the target feature sequence, wherein the target content includes a target image and / or target text.

[0070] Further, the first generation unit is specifically used to: generate contrastive language image pre-trained CLIP features based on the image portion, and generate text feature vectors based on the text portion; and concatenate the CLIP features, the text feature vectors, and the learnable features to generate the feature sequence.

[0071] Furthermore, the first generation unit is specifically used to: input the image portion into a pre-trained visual encoder to generate the CLIP feature.

[0072] Further, the first generation unit is specifically used to: perform text segmentation on the text portion to determine multiple word units; map each word unit through an embedding layer to generate an intermediate text feature vector; and generate the text feature vector based on the multiple intermediate text feature vectors.

[0073] Furthermore, the second generation unit is specifically used to: input the feature sequence into the autoregressive model part of a pre-trained hybrid autoregressive and diffusion architecture model to generate a target feature sequence; and generate the target content based on the target feature sequence.

[0074] Further, the second generation unit is specifically used to: input the feature sequence into the autoregressive model part of the pre-trained hybrid autoregressive and diffusion architecture model to generate a predicted CLIP feature sequence; input the predicted CLIP feature sequence into the diffusion model part of the hybrid autoregressive and diffusion architecture model to generate a target feature sequence; and generate the target content based on the target feature sequence.

[0075] Furthermore, the training process of the hybrid autoregressive and diffusion architecture model includes: the acquisition unit is used to: acquire sample data, wherein the sample data includes historical input text and image content and historical output feature sequences; the first generation unit is also used to: generate historical input feature sequences based on the historical input text and image content in the sample data; the device further includes: a training unit, used to train the hybrid autoregressive and diffusion architecture model based on the historical input feature sequences and the historical output feature sequences.

[0076] Furthermore, the device also includes an optimization unit for optimizing the hybrid autoregressive and diffusion architecture model based on a loss function.

[0077] Furthermore, the loss function includes cross-entropy loss, reconstruction loss, and self-feedback loss.

[0078] Figure 7 This is a schematic diagram of the structure of the electronic device described in an embodiment of the present invention. Figure 7 As shown, it includes a general computer hardware architecture, which includes at least a processor 701 and a memory 702. The processor 701 and the memory 702 are connected via a bus 703. The memory 702 is adapted to store instructions or programs executable by the processor 701. The processor 701 can be a standalone microprocessor or a collection of one or more microprocessors. Thus, the processor 701 executes the instructions stored in the memory 702 to perform the method flow of the embodiments of the present invention as described above, thereby realizing data processing and control of other devices. The bus 703 connects the above-mentioned components together, and also connects the above-mentioned components to a display controller 704, a display device, and an input / output (I / O) device 705. The input / output (I / O) device 705 can be a mouse, keyboard, modem, network interface, touch input device, motion-sensing input device, printer, and other devices known in the art. Typically, the input / output device 705 is connected to the system via an input / output (I / O) controller 706.

[0079] The instructions stored in memory 702 are executed by at least one processor 701 to: acquire user-input image and text content, wherein the user-input image and text content includes an image portion and a text portion; generate a feature sequence based on the user-input image and text content; input the feature sequence into a pre-trained hybrid autoregressive and diffusion architecture model to generate target content, wherein the target content includes a target image and / or target text.

[0080] Specifically, the electronic device includes: one or more processors 701 and a memory 702. Figure 7 Take a processor 701 as an example. The processor 701 and the memory 702 can be connected via a bus or other means. Figure 7 Taking a bus connection as an example, memory 702, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Processor 701 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in memory 702, thereby realizing the aforementioned method for determining multimodal content understanding and generation.

[0081] Memory 702 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store an option list, etc. Furthermore, memory 702 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 702 may optionally include memory remotely located relative to processor 701, and these remote memories may be connected to external devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0082] One or more modules are stored in memory 702, and when executed by one or more processors 701, they execute the multimodal content understanding and generation method in any of the above method embodiments.

[0083] As those skilled in the art will recognize, various aspects of the embodiments of the present invention can be implemented as a system, method, or computer program product. Therefore, various aspects of the embodiments of the present invention can take the form of a completely hardware implementation, a completely software implementation (including firmware, resident software, microcode, etc.), or an implementation combining software and hardware aspects, which may generally be referred to herein as a "circuit," "module," or "system." Furthermore, various aspects of the embodiments of the present invention can take the form of a computer program product implemented in one or more computer-readable media having computer-readable program code implemented thereon.

[0084] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, (but not limited to) an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media will include: an electrical connection having one or more wires, a portable computer floppy disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the context of embodiments of the present invention, a computer-readable storage medium can be any tangible medium capable of containing or storing a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0085] Computer-readable signal media may include propagated digital signals having computer-readable program code implemented therein, such as in baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and can communicate, propagate, or transmit a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0086] Program code implemented on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, or any suitable combination thereof.

[0087] Computer program code for performing operations relating to various aspects of embodiments of the present invention can be written in any combination of one or more programming languages, including: object-oriented programming languages ​​such as Java, Smalltalk, C++, etc.; and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code can be executed as a standalone software package entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet provided by an Internet service provider).

[0088] The flowchart illustrations and / or block diagrams of the methods, apparatus (systems), and computer program products according to embodiments of the present invention describe various aspects of the embodiments of the present invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine such that the instructions (executed via the processor of the computer or other programmable data processing apparatus) create means for implementing the functions / actions specified in the flowchart and / or block diagram blocks or blocks.

[0089] These computer program instructions may also be stored in a computer-readable medium that can direct a computer, other programmable data processing apparatus or other means to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of writing that includes instructions that implement the functions / actions specified in flowchart and / or block diagram blocks or blocks.

[0090] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operable steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide for implementing the functions / actions specified in flowchart and / or block diagram blocks or blocks.

[0091] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0092] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding access points are provided for users to choose to authorize or refuse processing. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

Claims

1. A method for multimodal content understanding and generation, characterized in that, The method includes: Obtain user-inputted image and text content, wherein the user-inputted image and text content includes an image portion and a text portion; Generate a feature sequence based on the user-input text and image content; The feature sequence is input into a pre-trained hybrid autoregressive and diffusion architecture model to generate target content, wherein the target content includes target images and / or target text.

2. The method according to claim 1, characterized in that, The step of generating a feature sequence based on the user input content specifically includes: Generate contrastive language image pre-trained CLIP features based on the image portion, and generate text feature vectors based on the text portion; The CLIP features, the text feature vector, and the learnable features are concatenated to generate the feature sequence.

3. The method according to claim 2, characterized in that, The step of generating contrastive language image pre-trained CLIP features based on the image portion specifically includes: The image portion is input into a pre-trained visual encoder to generate the CLIP features.

4. The method according to claim 2, characterized in that, The step of generating a text feature vector based on the text portion specifically includes: The text portion is processed by word segmentation to determine multiple word units; Each of the aforementioned lexical units is mapped through an embedding layer to generate an intermediate text feature vector; The text feature vector is generated based on the multiple intermediate text feature vectors.

5. The method according to claim 1, characterized in that, The step of inputting the feature sequence into a pre-trained hybrid autoregressive and diffusion architecture model to generate target content specifically includes: The feature sequence is input into the autoregressive model part of a pre-trained hybrid autoregressive and diffusion architecture model to generate the target feature sequence; The target content is generated based on the target feature sequence.

6. The method according to claim 1, characterized in that, The step of inputting the feature sequence into a pre-trained hybrid autoregressive and diffusion architecture model to generate target content specifically includes: The feature sequence is input into the autoregressive model part of a pre-trained hybrid autoregressive and diffusion architecture model to generate a predicted CLIP feature sequence. The predicted CLIP feature sequence is input into the diffusion model part of the hybrid autoregressive and diffusion architecture model to generate the target feature sequence. The target content is generated based on the target feature sequence.

7. The method according to claim 1, characterized in that, The training process of the hybrid autoregressive and diffusion architecture model includes: Acquire sample data, wherein the sample data includes historical input text and image content and historical output feature sequences; Generate a historical input feature sequence based on the historical input text and image content in the sample data; The hybrid autoregressive and diffusion architecture model is trained based on the historical input feature sequence and the historical output feature sequence.

8. The method according to claim 7, characterized in that, The method further includes: The hybrid autoregressive and diffusion architecture model is optimized based on the loss function.

9. The method according to claim 8, characterized in that, The loss function includes cross-entropy loss, reconstruction loss, and self-feedback loss.

10. A device for multimodal content understanding and generation, characterized in that, The device includes: The acquisition unit is used to acquire user-input image and text content, wherein the user-input image and text content includes an image part and a text part; The first generation unit is used to generate a feature sequence based on the user-input text and image content; The second generation unit is used to input the feature sequence into a pre-trained hybrid autoregressive and diffusion architecture model to generate the target feature sequence; The third generation unit is used to generate target content based on the target feature sequence, wherein the target content includes target image and / or target text.

11. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-9.

Images