Multi-modal data processing method and apparatus, electronic device, computer-readable storage medium, and computer program product

Abstract

Disclosed in the present application are a multi-modal data processing method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring a reference image and a reference text; extracting a reference visual feature of the reference image; by means of a multi-modal large language model, determining an embedding of the reference text, an embedding of a start mark of the reference visual feature, an embedding of the reference visual feature, and an embedding of an end mark of the reference visual feature; on the basis of the multi-modal large language model, splicing the embedding of the reference text, the embedding of the start mark, the embedding of the reference visual feature, and the embedding of the end mark into a target embedding sequence, performing attention processing on the basis of the embedding of the start mark, the embedding of the end mark, and an embedding selected by a sliding window in the target embedding sequence, and outputting a predicted sequence; and generating a predicted image and a predicted text on the basis of the predicted sequence.

Description

Multi-modal data processing method and device, electronic device, computer readable storage medium and computer program product

[0001] Cross-reference to related applications

[0002] This application is based on the Chinese patent application No. 2024107494580, filed on June 11, 2024, and claims priority to the Chinese patent application, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the technical field of computers, and in particular to a multi-modal data processing method, device, electronic device, computer readable storage medium and computer program product. BACKGROUND

[0004] With the significant progress of image generation and open-form text generation technology, multi-modal data processing of image data and text data has become an increasingly attractive field. This task requires a full understanding of the interrelated image data and text data, and since the text data not only has abstract and narrative properties, but is also closely intertwined with complex image data, the performance of the model is usually degraded during the reasoning process when continuous multiple rounds of multi-modal data processing are required, which affects the generation quality of multi-modal content. SUMMARY

[0005] The following is a summary of the subject matter described in detail in the embodiments of the present application. This summary is not intended to limit the scope of protection of the claims.

[0006] The embodiments of the present application provide a multi-modal data processing method, device, electronic device, computer readable storage medium and computer program product, which can effectively improve the performance of a multi-modal large language model, and further improve the generation quality of multi-modal content.

[0007] In one aspect, the embodiments of the present application provide a multi-modal data processing method applied to an electronic device, comprising:

[0008] obtaining a reference image and a reference text, wherein the reference text is used to describe the content in the reference image;

[0009] extracting a reference visual feature of the reference image;

[0010] determining, by a multi-modal large language model, an embedding of the reference text, an embedding of a start token of the reference visual feature, an embedding of the reference visual feature, and an embedding of an end token of the reference visual feature;

[0011] based on the multimodal large language model, concatenating the embedding of the reference text, the embedding of the start token, the embedding of the reference visual feature, and the embedding of the end token into a target embedding sequence, performing attention processing based on the embedding of the start token, the embedding of the end token, and the embedding of the embedding in the target embedding sequence selected by a sliding window, and outputting a prediction sequence;

[0012] generating a predicted image and a predicted text based on the prediction sequence, wherein the predicted text is used to describe the content in the predicted image.

[0013] In another aspect, the embodiments of the present application also provide a multimodal data processing apparatus, comprising:

[0014] an acquisition module configured to acquire a reference image and a reference text, wherein the reference text is used to describe the content in the reference image;

[0015] a first processing module configured to extract a reference visual feature of the reference image, and determine, through a multimodal large language model, an embedding of the reference text, an embedding of a start token of the reference visual feature, an embedding of the reference visual feature, and an embedding of an end token of the reference visual feature;

[0016] a second processing module configured to, based on the multimodal large language model, concatenate the embedding of the reference text, the embedding of the start token, the embedding of the reference visual feature, and the embedding of the end token into a target embedding sequence, perform attention processing based on the embedding of the start token, the embedding of the end token, and the embedding of the embedding in the target embedding sequence selected by a sliding window, and output a prediction sequence;

[0017] a generation module configured to generate a predicted image and a predicted text based on the prediction sequence, wherein the predicted text is used to describe the content in the predicted image.

[0018] In another aspect, the embodiments of the present application also provide an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the multimodal data processing method described above when executing the computer program.

[0019] In another aspect, the embodiments of the present application also provide a computer readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the multimodal data processing method described above.

[0020] In another aspect, an embodiment of the present application also provides a computer program product, which comprises a computer program stored in a computer readable storage medium. A processor of a computer device reads the computer program from the computer readable storage medium, and the processor executes the computer program to enable the computer device to perform the multi-modal data processing method described above.

[0021] The embodiments of the present application have at least the following beneficial effects: by obtaining a reference image and a reference text used to describe the content in the reference image, extracting a reference visual feature of the reference image, adding a start mark and an end mark at both ends of the reference visual feature, the reference visual feature and the reference text can be distinguished based on the start mark and the end mark, and on this basis, the reference text and the reference visual feature with the start mark and the end mark added at both ends are input into the multi-modal large language model. By embedding the text and the reference visual feature into the same semantic space, the multi-modal large language model can better understand the relationship between the text and the image, so as to generate a prediction result that conforms to the semantics. By jointly embedding multiple embeddings, the multi-modal large language model can better capture and utilize context information, while optimizing the generation of text and image, and improving the accuracy of prediction. Based on the prediction sequence output by the multi-modal large language model, the prediction image and the prediction text used to describe the prediction image are generated based on the prediction sequence. During inference, the embeddings in the start mark, the end mark, and the embeddings selected by the sliding window in the target embedding sequence are subjected to attention processing. By limiting the range of attention calculation, the sliding window mechanism significantly reduces the computational complexity, so that the model can maintain high inference speed in long sequence generation. Therefore, even if continuous multi-modal data processing is performed for multiple rounds, the multi-modal large language model can focus on the information carried by the embeddings in the start mark, the end mark, and the embeddings selected by the sliding window in the target embedding sequence through attention processing, thereby reducing the perplexity of the multi-modal large language model during inference, effectively improving the performance of the multi-modal large language model, and further improving the generation quality of multi-modal content.

[0022] Other features and advantages of the embodiments of the present application will be described in the following description, and some will become apparent from the description, or will be learned from the practice of the embodiments of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings are included to provide a further understanding of the technical solutions of the embodiments of the present application, and constitute a part of the specification, and are used to explain the technical solutions of the embodiments of the present application together with the embodiments of the present application, and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0024] FIG. 1 is a schematic diagram of an optional implementation environment provided by an embodiment of the present application;

[0025] FIG. 2 is an optional flow diagram of a multi-modal data processing method according to an embodiment of the present application;

[0026] FIG. 3 is an optional visual diagram of attention processing according to an embodiment of the present application;

[0027] FIG. 4 is an optional flow diagram of obtaining predicted visual features according to an embodiment of the present application;

[0028] FIG. 5 is an optional flow diagram of obtaining embedding of reference visual features according to an embodiment of the present application;

[0029] FIG. 6 is an optional flow diagram of training a multi-modal large language model according to an embodiment of the present application;

[0030] FIG. 7 is an optional flow diagram of jointly training a feature extractor and the diffusion model according to an embodiment of the present application;

[0031] FIG. 8 is an optional first flow diagram of retraining the diffusion model according to an embodiment of the present application;

[0032] FIG. 9 is an optional second flow diagram of retraining the diffusion model according to an embodiment of the present application;

[0033] FIG. 10 is an optional distribution diagram of attention score distribution according to an embodiment of the present application;

[0034] FIG. 11 is an optional column diagram of FID score according to an embodiment of the present application;

[0035] FIG. 12 is an optional pie diagram of style consistency comparison result according to an embodiment of the present application;

[0036] FIG. 13 is an optional pie diagram of narrative appeal comparison result according to an embodiment of the present application;

[0037] FIG. 14 is an optional pie diagram of image-text consistency comparison result according to an embodiment of the present application;

[0038] FIG. 15 is an optional structural diagram of a multi-modal data processing apparatus according to an embodiment of the present application;

[0039] FIG. 16 is a partial structural block diagram of a terminal according to an embodiment of the present application;

[0040] FIG. 17 is a partial structural block diagram of a server according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application are further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present application and not to limit the embodiments of the present application.

[0042] It should be noted that in the embodiments of the present application, when it is necessary to perform relevant processing according to data related to the characteristics of the target object, such as target object attribute information or attribute information set, the permission or consent of the target object will be obtained first, and the collection, use and processing of the data will comply with relevant laws, regulations and standards. Among them, the target object can be a user. In addition, when the embodiments of the present application need to obtain target object attribute information, the separate permission or separate consent of the target object will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the target object, the necessary target object related data for enabling the embodiments of the present application to normally run will be obtained.

[0043] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as processing circuitry or memory) or a combination thereof. Similarly, one processor (or multiple processors or memory) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the functions of the module or unit.

[0044] In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, some key terms used by the embodiments of the present application are explained first:

[0045] 1) Cloud technology refers to a kind of hosting technology that unifies a series of resources such as hardware, software and network in a wide area network or local area network to realize data calculation, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology and application technology based on cloud computing business model application, which can form a resource pool, and can be used on demand, flexibly and conveniently. Cloud computing technology will become an important support. The background service of the technical network system needs a large amount of calculation and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, every item may have its own identification mark in the future, which needs to be transmitted to the background system for logical processing. Different levels of data will be processed separately, and various industry data will need strong system support, which can only be realized through cloud computing.

[0046] 2) Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is the design principle and implementation method of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-training model technology, operation / interaction system, mechatronics, etc. Among them, the pre-training model is also called large model, basic model, which can be widely used in downstream tasks of various directions of artificial intelligence after fine-tuning. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning, etc.

[0047] 3) Machine Learning (ML) is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is a subject that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0048] 4) Reference image refers to visual data used for comparison, analysis or generation of target in computer vision and multimedia applications. It is a two-dimensional digital matrix, where each element (pixel) represents a color or intensity value. Reference images can be various types of images, including but not limited to RGB color images, grayscale images, binary images, depth images, thermal imaging images, etc. The sources of reference images are diverse, including natural scene photos, art works, medical images, satellite images, animation or movie screenshots, etc.

[0049] 5) Reference text, is the text data used to describe or annotate visual content, such as for describing reference images. It can be a description, label, title, explanation, story, dialogue or any other form of text about the content of the image, aiming to provide semantic understanding, contextual information or emotional expression of the visual data. Reference text can be manually written or generated through natural language processing technology.

[0050] 6) Visual features, are abstract information extracted from images through computer vision algorithms, used to describe the content, structure or attributes of images. These features can be low-level (such as color, texture, edge, shape) or high-level (such as object category, scene type, action, expression, etc.). Visual features are usually represented as numerical vectors or matrices, and can be used for image recognition, classification, retrieval, generation, etc.

[0051] 7) Start token, is a special symbol or token used in sequence data processing, used to indicate the start of a sequence. In natural language processing and sequence generation tasks, the start token is often used to prompt the model to start generating or processing a new sequence. The start token is usually a preset symbol or character, for example, the start token is preset as <boi>.

[0052] 8) End token, a special symbol or token used in sequence data processing to indicate the end of a sequence. The end token is used to tell the model that the generation or processing of a sequence has been completed. In generation tasks, the end token can be used to stop further generation process, ensuring that the generated sequence has appropriate length and completeness. The end token is usually a preset symbol or character, for example, the end token is preset as <eoi>.

[0053] 9)Predicted token, which is the basic unit used in text analysis and generation processes, representing the smallest language unit considered by the language model when predicting the next token. The predicted token can be a complete word, a single character, a subword, or other granularity of language units, depending on the used tokenization strategy and the architecture of the model.

[0054] With the significant progress of image generation and open-form text generation technology, multi-modal data processing of image data and text data has become an increasingly attractive field. This task requires a full understanding of the interrelated image data and text data, and since the text data not only has abstract and narrative properties, but is also closely intertwined with complex image data, during the inference process, when continuous multiple rounds of multi-modal data processing are required, it usually leads to a decline in the performance of the model, affecting the generation quality of multi-modal content.

[0055] Based on this, the embodiments of the present application provide a multi-modal data processing method and device, electronic equipment and storage medium, which can effectively improve the performance of the multi-modal large language model, and further improve the generation quality of multi-modal content.

[0056] Referring to FIG. 1, FIG. 1 is a schematic diagram of an optional implementation environment provided by the embodiments of the present application, which includes a terminal 101 and a server 102, wherein the terminal 101 and the server 102 are connected through a communication network.

[0057] Exemplarily, the server 102 can obtain a reference image and a reference text sent by the terminal 101, wherein the reference text is used to describe the content in the reference image; extract a reference visual feature of the reference image, and input the reference text and the reference visual feature with a start mark and an end mark added at both ends into a multi-modal large language model; determine a target embedding sequence obtained by splicing an embedding of the reference text, an embedding of the start mark, an embedding of the reference visual feature, and an embedding of the end mark based on the multi-modal large language model, perform attention processing based on the embedding of the start mark, the embedding of the end mark, and the embedding selected by a sliding window in the target embedding sequence, and output a predicted sequence; generate a predicted image and a predicted text based on the predicted sequence, wherein the predicted text is used to describe the content in the predicted image; and the server 102 sends the predicted image and the predicted text to the terminal 101.

[0058] The server 102 extracts reference visual features of the reference image by obtaining the reference image and reference text describing the content in the reference image, adds a start tag and an end tag at both ends of the reference visual features, which can facilitate subsequent differentiation between the reference visual features and the reference text based on the start tag and the end tag, and on this basis, inputs the reference text and the reference visual features with the start tag and the end tag added at both ends into the multi-modal large language model, outputs a prediction sequence based on the multi-modal large language model, and generates a predicted image and predicted text describing the predicted image based on the prediction sequence. Since the embeddings of the start tag, the end tag, and the embeddings selected by the sliding window in the target embedding sequence are processed by attention during inference, even if multiple rounds of multi-modal data processing are performed, the multi-modal large language model can focus on the information carried by the embeddings of the start tag, the end tag, and the embeddings selected by the sliding window in the target embedding sequence through attention processing, thereby reducing the perplexity during inference of the multi-modal large language model, effectively improving the performance of the multi-modal large language model, and further improving the generation quality of multi-modal content.

[0059] Taking an artistic creation assistance scenario as an example, the terminal is used to collect reference images and reference text of artistic creation, and the server generates predicted images and predicted text that meet the requirements of artistic creation based on the reference images and the reference text, serving as an artistic creation assistance tool.

[0060] Taking an educational training scenario as an example, the terminal is used to collect reference images and reference text, and the server generates predicted images and predicted text for making teaching materials and training content based on the reference images and the reference text.

[0061] Taking a social media scenario as an example, the terminal is used to collect reference images and reference text, and the server generates predicted images and predicted text for providing personalized content recommendations based on the reference images and the reference text.

[0062] Taking an advertising scenario as an example, the terminal is used to collect reference images and reference text, and the server generates predicted images and predicted text that meet the business requirements set by the advertiser based on the reference images and the reference text.

[0063] The server 102 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc. basic cloud computing services. In addition, the server 102 can also be a node server in a blockchain network.

[0064] The terminal 101 can be a mobile phone, a computer, a smart voice interaction device, a smart household appliance, a vehicle-mounted terminal, etc., but is not limited thereto. The terminal 101 and the server 102 can be directly or indirectly connected through wired or wireless communication, and the embodiments of the present application are not limited thereto.

[0065] The method provided by the embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, intelligent transportation, and assisted driving scenarios.

[0066] Referring to FIG. 2, FIG. 2 is an optional flowchart of a multi-modal data processing method provided by the embodiments of the present application. The multi-modal data processing method can be executed by a server, or can be executed by a terminal, or can be executed by a server in cooperation with a terminal. The multi-modal data processing method includes but is not limited to the following steps 201 to 204.

[0067] Step 201: Obtain a reference image and a reference text.

[0068] The reference image is visual modal data, and the reference image refers to a two-dimensional digital matrix composed of pixels. The reference image can be an RGB image, that is, each pixel point corresponds to three color channels. The reference image can also be a grayscale image, a binary image, a depth image, etc., and the embodiments of the present application are not limited thereto. The source of the reference image can be diverse, for example, the reference image can be an image from an animation clip, a movie screenshot, a photographic work, etc.

[0069] The reference text is text modal data, and the reference text is used to describe the content in the reference image, that is, the reference text is a description text of the reference image. For example, the reference text can be a text describing the visual elements, scene details, emotional atmosphere, or story of the reference image. For example, when the reference image is an animation clip, the reference text can be a subtitle text, a script text, a data text, etc. related to the animation clip. It can be seen that the reference text provides more detailed information for the reference image, and the use of the reference text can effectively deepen and expand the understanding of the reference image.

[0070] Step 202: Extract a reference visual feature of the reference image, and input the reference text and the reference visual feature with a start tag and an end tag added at both ends into a multi-modal large language model.

[0071] The reference visual features are more abstract and informative features extracted from the reference image. The reference visual features can include one visual feature vector or multiple visual feature vectors of the same dimension. For example, the reference visual features include 256 visual feature vectors of 768 dimensions. The reference visual features can be obtained by feature extraction on the reference image using a feature extractor. For example, the feature extractor includes a convolutional neural network (CNN), a generative adversarial network, a contrastive language-image pre-training (CLIP) model, etc. The reference visual features can also be obtained by any of the following processing methods:

[0072] Manual feature extraction: Use manually designed feature descriptors such as Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), and Histogram of Oriented Gradient (HOG). These methods describe image content by calculating local features of the image.

[0073] Deep learning features: In addition to convolutional neural networks, other deep learning models such as Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), and self-attention mechanism can also be used. These models can capture global and local features of the image.

[0074] Pre-trained models: Use pre-trained models such as those on ImageNet. These models have learned rich image features and can be directly used for feature extraction.

[0075] Transfer learning: Use transfer learning on the basis of pre-trained models to extract visual features for specific tasks. This method can utilize existing knowledge to improve the accuracy of feature extraction.

[0076] Unsupervised learning: Use unsupervised learning methods such as autoencoder, Variational AutoEncoder (VAE), and Generative Adversarial Network (GAN) to learn image features without labels.

[0077] Multi-task learning: through multi-task learning, multiple related tasks such as image classification, object detection, semantic segmentation, etc. are learned at the same time, so as to extract more comprehensive and robust visual features.

[0078] For example, in the self-attention mechanism, the reference visual feature of the reference image can be obtained by performing the following processing: converting the reference image into a feature map through a convolutional layer, the feature map including low-level features of the reference image such as edges, textures, etc.; flattening the feature map into a sequence of feature vectors, each feature vector in the sequence of feature vectors corresponding to a position or region in the reference image; for each feature vector in the sequence of feature vectors, performing self-attention calculation on the feature vector to determine the weight of the feature vector, the weight representing the correlation degree between the current feature vector and other feature vectors; and performing weighted summation on multiple feature vectors based on the weight to obtain the reference visual feature.

[0079] The embodiments of the present application extract the reference visual feature of the reference image through the self-attention mechanism, which can consider the information of the entire reference image when processing the feature vector of each position, thereby capturing the global context relationship. Through the weight distribution, more attention is given to the local details in the reference image, improving the accuracy of feature extraction.

[0080] The start marker is usually a preset symbol or character, which is used to indicate the start of the sequence and can be located at the first position of the sequence. For example, the start marker is preset as <boi>, and similarly, an end marker is also a predetermined symbol or character for indicating the end of the sequence, which can be located at the last position of the sequence, for example, the end marker is predetermined as <eoi>It can be seen that the start mark and the end mark can be regarded as text modal data; the two ends of the reference visual feature refer to the beginning end and the ending end of the reference visual feature, and the start mark and the end mark are added at the two ends of the reference visual feature, for example, the start mark is added at the beginning end of the reference visual feature, and the end mark is added at the ending end of the reference visual feature, and the start mark, the reference visual feature and the end mark can exist in a combined form.

[0081] The multi-modal refers to information from different senses or sources, such as vision, hearing, touch, etc., and the multi-modal large language model is an extended form of the large language model. Compared with the large language model (LLM) that can only process input data corresponding to the text modal, the multi-modal large language model can not only process input data corresponding to the text modal, but also process input data corresponding to other modalities in addition to the text modal, for example, other modalities include visual modal, audio modal, etc.

[0082] Based on this, for the reference image belonging to the visual modal data, the reference image needs to be mapped into a fixed-dimensional reference visual feature first, and then the reference visual feature with the start mark and the end mark added at the two ends is input into the multi-modal large language model. The multi-modal large language model can distinguish the reference visual feature and the reference text based on the start mark and the end mark. The reference text, the start mark and the end mark are all input data corresponding to the text modal, and the multi-modal large language model can effectively process the reference text, the start mark and the end mark. The reference visual feature is input data corresponding to the visual modal, and the multi-modal large language model can also effectively process the reference visual feature.

[0083] In an example, when the reference text and the reference visual feature with the start mark and the end mark added at the two ends are input into the multi-modal large language model, the data can be input in multiple ways, for example, the reference text and the reference visual feature with the start mark and the end mark added at the two ends are input into the multi-modal large language model as a batch of data; for another example, the reference text and the reference visual feature with the start mark and the end mark added at the two ends are input into the multi-modal large language model in sequence; when the data is input in sequence, the reference text can be input into the multi-modal large language model first, or the reference visual feature with the start mark and the end mark added can be input into the multi-modal large language model first, which is not limited in the embodiments of the present application.

[0084] Step 203: determining, based on the multi-modal large language model, a target embedding sequence spliced by the embedding of the reference text, the embedding of the start token, the embedding of the reference visual feature, and the embedding of the end token, and performing attention processing on the embedding of the start token, the embedding of the end token, and the embedding selected by the sliding window in the target embedding sequence to obtain a predicted sequence.

[0085] The multi-modal large language model can determine the embedding of the corresponding modal data through different embedding modules. For example, the multi-modal large language model includes a word embedding module and a graph embedding module. The word embedding module is used to process input data corresponding to the text modality. The reference text, the start token, and the end token are embedded by the word embedding module, respectively, to determine the embedding of the reference text, the embedding of the start token, and the embedding of the end token. The embedding of the reference text includes the embedding of each word in the reference text.

[0086] For example, each word in the reference text, the start token, and the end token are recorded as word pieces in a vocabulary. Each word piece in the vocabulary is assigned a unique index. The embedding of each word in the reference text, the embedding of the start token, and the embedding of the end token can be determined by the vocabulary. The word embedding module maintains a word embedding matrix that stores the word embedding of each word piece. The word embedding module can query the corresponding word embedding in the word embedding matrix based on the index, i.e., obtain the embedding of the reference text, the embedding of the start token, and the embedding of the end token.

[0087] The graph embedding module is used to process input data corresponding to the visual modality. The reference visual feature is embedded by the graph embedding module to determine the embedding of the reference visual feature. When the reference visual feature includes a visual feature vector, the embedding of the reference visual feature is the embedding of the visual feature vector. When the reference visual feature includes multiple visual feature vectors, the embedding of the reference visual feature includes the embedding of each visual feature vector. The dimension of the embedding of the visual feature vector is the same as that of the word embedding mentioned above.

[0088] In some embodiments, the embedding of the reference text, the embedding of the start token, the embedding of the reference visual feature, and the embedding of the end token can be directly added in a preset order to obtain the target embedding sequence. The preset order can be: the embedding of the reference text, the embedding of the start token, the embedding of the reference visual feature, and the embedding of the end token. The multi-modal large language model can also assign weights to the embedding of the reference text, the embedding of the start token, the embedding of the reference visual feature, and the embedding of the end token. The embedding of the reference text, the embedding of the start token, the embedding of the reference visual feature, and the embedding of the end token are weighted and summed based on the weights to obtain the target embedding sequence.

[0089] For example, in the case of direct splicing, the embedding a of the reference text, the embedding b of the start marker, the embedding c of the reference visual feature, and the embedding d of the end marker are added to obtain the target embedding sequence a+b+c+d. For example, in the case of weighted sum by assigning weights, the weight of the embedding a of the reference text is w1, the weight of the embedding b of the start marker is w2, the weight of the embedding c of the reference visual feature is w3, and the weight of the embedding d of the end marker is w4, and the target embedding sequence can be represented as a*w1+b*w2+c*w3+d*w4.

[0090] According to the weight distribution, the proportion of different modal information can be flexibly adjusted according to the needs of the task, so that the target embedding sequence is more in line with the requirements of the task. The process of weighted sum makes the information of different modalities effectively fused, and enhances the understanding ability of the model to multi-modal data. The weight distribution mechanism helps to maintain the semantic consistency of text and visual information, so that the target embedding sequence can better reflect the semantic content of the input data.

[0091] The target embedding sequence is an input sequence of the multi-modal large language model, the prediction sequence is an output sequence of the multi-modal large language model, and the output sequence is composed of all output data of the multi-modal large language model; the sliding window is used to determine the attention range of the multi-modal large language model, and the attention range refers to the specific range of the multi-modal large language model in the input sequence when reasoning the output data; when the multi-modal data is processed for a plurality of rounds by the multi-modal large language model, each time an output data is reasoned, the embedding corresponding to the output data is added at the end of the target embedding sequence, and the next output data is reasoned based on the new target embedding sequence. The sliding window will slide with the length of the target embedding sequence, and the last embedding of the target embedding sequence is always the embedding selected by the sliding window. The size of the sliding window is usually fixed, for example, the size of the sliding window is set to 5, assuming that the length of the target embedding sequence is 30, the first time the embedding selected by the sliding window includes the last 5 embeddings in the target embedding sequence, and the second time the length of the target embedding sequence is updated to 31, at this time the embedding selected by the sliding window includes the last 5 embeddings in the new target embedding sequence.

[0092] Based on this, the attention scope of a multimodal large language model includes not only the embeddings selected by the sliding window, but also the embeddings of the start and end tags. The multimodal large language model can process and focus on the information carried by the embeddings of the start and end tags, as well as the embeddings selected by the sliding window in the target embedding sequence. When processing the target embedding sequence, compared to only focusing on the embeddings selected by the sliding window, retaining the embeddings of the start and end tags expands the model's memory, allowing the model to maintain attention on the key information carried by the embeddings of the start and end tags, avoiding information loss due to the sliding of the window. This helps the multimodal large language model maintain a uniform and stable attention distribution when processing long sequences, thereby reducing the perplexity of the multimodal large language model during reasoning and effectively improving its performance. For example, when using a multimodal large language model for story generation, it can improve the coherence of the generated content and maintain consistency in style and characters.

[0093] It is worth noting that the embedding of the start marker and the embedding of the end marker can also be selected by the sliding window. Here, the attention range includes the embedding of the start marker and the embedding of the end marker. For example, the attention range includes the embedding of the start marker and the embedding of the end marker that are not selected by the sliding window. The number of embeddings of the start marker or the number of embeddings of the end marker that are not selected by the sliding window can be zero. This application embodiment does not limit this.

[0094] Perplexity is a measure of the uncertainty of a model's inference of output data given an input sequence. Lower perplexity indicates better predictive performance, while higher perplexity indicates worse predictive performance. Perplexity is often used as an important metric for evaluating the performance of multimodal large language models.

[0095] For example, perplexity can be determined by performing the following process: For each word in the sequence, calculate the predicted probability of the word given all preceding words, and determine the logarithm of the predicted probability; determine the sum of the logarithms of all words, and determine the ratio of the sum to the length of the sequence to obtain the average log probability; determine the negative of the average log probability, and raise the negative to the power of 2 to obtain the perplexity. For example, for each word n ​​in the sequence... i The predicted probability is p i The logarithm of the predicted probability is log2p i If there are N words in total, then the sum of the logarithms of all words is: The average log probability is The negative of the mean logarithm is Confusion level

[0096] In addition, since the size of the sliding window is fixed, and the total number of the embedded start tokens and the embedded end tokens is relatively small, the attention range of the multi-modal large language model is relatively narrow, which can effectively reduce the memory usage and significantly improve the inference efficiency compared to global attention processing on the target embedding sequence.

[0097] Step 204: generating a predicted image and a predicted text based on the predicted sequence.

[0098] The predicted text is used to describe the content in the predicted image, and the predicted text is usually text that does not include special tokens. The predicted image is visual modality data, and the predicted text is text modality data; the predicted image is an image associated with the reference image, for example, when using a multi-modal large language model to generate a story, the generated predicted image is an image of the next scene of the reference image, for example, when using a multi-modal large language model to generate a step, the generated predicted image is an image of the next step of the reference image, for example, when using a multi-modal large language model to enhance content, the generated predicted image is an image similar to the reference image.

[0099] For example, the special token includes at least one of the following:

[0100] Start Token: usually used to mark the beginning of the sequence, for example, when generating text or image description, the start token can help the model know the starting point of the generation process.

[0101] End Token: used to mark the end of the sequence, to ensure that the model knows when to stop generating.

[0102] Separator Token: used to separate different parts of input data, for example, in multi-modal input, the separator token can be used to separate text and visual features.

[0103] Mask Token: in some pre-training tasks, such as the Masked Language Model task in BERT, the Mask Token is used to mask some words in the input sequence, and the model needs to predict the masked words.

[0104] Class Token: in some models, the class token is used to represent the classification information of the entire sequence, for example, in an image classification task, the class token can be used to represent the class of the image.

[0105] Position Token: used to represent the position information of each element in the sequence, which is very important for models that need to consider sequential information.

[0106] Based on this, since the reference text and the reference image are alternated, the multi-modal large language model can learn the content format of the alternating text modal and visual modal, output a prediction sequence for generating alternating text modal data and visual modal data, and further generate alternating predicted images and predicted texts based on the prediction sequence, the predicted images and the predicted texts belong to multi-modal content; the multi-modal large language model can pay attention to the information carried by the embedding of the start token, the embedding of the end token, and the embedding selected by the sliding window in the target embedding sequence through attention processing, thereby reducing the perplexity of the multi-modal large language model during inference, effectively improving the performance of the multi-modal large language model, and further improving the generation quality of the multi-modal content, which can obtain rich text content and diverse visual content, so that the predicted text and the predicted image are closely intertwined.

[0107] For example, the prediction sequence is composed of all output data of the multi-modal large language model, and the length of the prediction sequence increases every time the multi-modal large language model infers a new output data. The stop condition of the multi-modal large language model can be set, and the multi-modal large language model can continuously infer new output data until the stop condition is reached, for example, the stop condition is that the length of the prediction sequence reaches a preset length threshold. Since the length of the prediction sequence can be adjusted, the alternating predicted images and predicted texts can be single round or multiple rounds, which is not limited in the embodiments of the present application.

[0108] In some embodiments, the prediction sequence includes a prediction token and a prediction visual feature, and the attention processing is performed based on the embedding of the start token, the embedding of the end token, and the embedding selected by the sliding window in the target embedding sequence to output the prediction sequence, for example, the prediction embedding can be generated based on the attention processing of the embedding of the start token, the embedding of the end token, and the embedding selected by the sliding window in the target embedding sequence; the prediction embedding belonging to the text type is classified to output the prediction token; and the prediction embedding belonging to the visual type is mapped to output the prediction visual feature.

[0109] The prediction token is a basic unit in text analysis, and can be a word, a character, a subword, etc.; and the prediction visual feature is in a visual feature space and is used to indicate more abstract and more informative features of an image.

[0110] Based on this, in the inference process of multiple rounds in succession, the multimodal large language model generates a prediction embedding in each round, and through attention processing, the multimodal large language model can consider the information carried by the embedding of the start token, the embedding of the end token, and the embedding selected by the sliding window in the target embedding sequence when generating the prediction embedding, thereby improving the relevance and coherence of the generated content. In addition, the multimodal large language model can generate prediction embeddings belonging to the text type and prediction embeddings belonging to the visual type. The prediction embedding belonging to the text type refers to the prediction embedding through which the output data corresponding to the text modality can be determined, and the prediction embedding belonging to the visual type refers to the prediction embedding through which the output data corresponding to the visual modality can be determined. When generating the prediction embedding belonging to the text type, the prediction embedding belonging to the text type is classified, and the prediction token corresponding to the text modality can be output. When generating the prediction embedding belonging to the visual type, the prediction embedding belonging to the visual type is mapped, and the prediction visual feature corresponding to the visual modality can be output.

[0111] For example, the multimodal large language model is internally provided with a Transformer decoding layer. After the embedding of the start token, the embedding of the end token, and the embedding selected by the sliding window in the target embedding sequence are input into the Transformer decoding layer, the Transformer decoding layer can perform self-attention processing on the input data to generate a prediction embedding. When the multimodal large language model processes multimodal data, it can alternately generate a set of prediction embeddings belonging to the text type and a set of prediction embeddings belonging to the visual type, and then alternately output a set of prediction tokens corresponding to the text modality and a set of prediction visual features corresponding to the visual modality.

[0112] In some embodiments, referring to FIG. 3, which is an optional visual schematic diagram of attention processing provided by an embodiment of the present application, a head token is added at the head end of the reference text. The target embedding sequence is determined based on the embedding of the reference text, the embedding of the start token, the embedding of the end token, and the embedding of the reference visual feature, and attention processing is performed based on the embedding of the start token, the embedding of the end token, and the embedding selected by the sliding window in the target embedding sequence to output a prediction sequence. For example, the target embedding sequence is determined based on the embedding of the head token, the embedding of the reference text, the embedding of the start token, the embedding of the end token, and the embedding of the reference visual feature, and attention processing is performed based on the embedding of the head token, the embedding of the start token, the embedding of the end token, and the embedding selected by the sliding window in the target embedding sequence to generate a prediction embedding. The prediction embedding belonging to the text type is classified to output a prediction token, and the prediction embedding belonging to the visual type is mapped to output a prediction visual feature.

[0113] Wherein, the head end mark is preset as a symbol or character, for example, the head end mark is preset as <bos>The first end of the reference text can be added with one or more first end markers, which are not limited in the embodiments of the present application.

[0114] Based on this, through attention processing, the multi-modal large language model can consider the embedding of the first end marker, the embedding of the start marker, the embedding of the end marker, and the information carried by the embedding selected by the sliding window in the target embedding sequence when generating the prediction embedding, which can further improve the relevance and coherence of the generated content.

[0115] In some embodiments, the self-attention processing is performed based on the embedding of the start marker, the embedding of the end marker, and the embedding selected by the sliding window in the target embedding sequence to generate the prediction embedding. For example, the embedding of the start marker, the embedding of the end marker, and the embedding selected by the sliding window in the target embedding sequence can be spliced into a to-be-processed embedding sequence; a first key matrix and a first value matrix are constructed based on the to-be-processed embedding sequence; a first query matrix is constructed based on the embedding of the last position in the to-be-processed embedding sequence; and the attention processing is performed based on the first query matrix, the first key matrix, and the first value matrix to generate the prediction embedding.

[0116] The to-be-processed embedding sequence is spliced from the embedding of the start marker, the embedding of the end marker, and the embedding selected by the sliding window in the target embedding sequence according to the order in the target embedding sequence, the first key matrix is the product of the to-be-processed embedding sequence and a learnable first key parameter matrix, the first value matrix is the product of the to-be-processed embedding sequence and a learnable first value parameter matrix, the first key matrix and the first value matrix both include the hidden state of each embedding in the to-be-processed embedding sequence, the first query matrix is the product of the embedding of the last position in the to-be-processed embedding sequence and a learnable first query parameter matrix, and the first query matrix includes the hidden state of the embedding of the last position in the to-be-processed embedding sequence. The hidden state can be a vector representation processed by a self-attention layer and a feedforward neural network layer, and the vector representation includes the implicit understanding of the model for each position in the input sequence.

[0117] For example, assume that the to-be-processed embedding sequence includes 50 embeddings, each embedding in the to-be-processed embedding sequence has a dimension of 1*768, the dimensions of the first query parameter matrix, the first key parameter matrix, and the first value parameter matrix are all 768*768, the to-be-processed embedding sequence multiplied by the first key parameter matrix obtains a first key matrix with a dimension of 50*768, the to-be-processed embedding sequence multiplied by the first value parameter matrix obtains a first value matrix with a dimension of 50*768, and the embedding of the last position in the to-be-processed embedding sequence multiplied by the first query parameter matrix obtains a first query matrix with a dimension of 1*768.

[0118] Based on this, since the first query matrix indicates the information that needs to be focused on at present, the first query matrix is specifically used to find the information most relevant to the embedding at the last position in the to-be-processed embedding sequence, the first key matrix and the first value matrix both contain the information of each embedding in the to-be-processed embedding sequence, the first key matrix is used for the multi-modal large language model to determine the importance of each embedding in the to-be-processed embedding sequence to the embedding at the last position in the to-be-processed embedding sequence, the first value matrix is used to generate a predicted embedding according to the importance, and the multi-modal large language model performs attention processing on the first query matrix, the first key matrix and the first value matrix to generate the predicted embedding. Since the source of the first query matrix is the same as the source of the first key matrix and the first value matrix, in fact, the first query matrix, the first key matrix and the first value matrix are self-attention processed to generate the predicted embedding. When generating the predicted embedding, the multi-modal large language model can pay more attention to important information in the to-be-processed embedding sequence and ignore secondary information, efficiently utilize and integrate input information, and enhance the ability of the model to understand and generate complex outputs.

[0119] It should be noted that after the to-be-processed embedding sequence is determined, the to-be-processed embedding sequence is loaded into the memory and attention processing is performed. Before attention processing, the to-be-processed embedding sequence is positionally encoded based on the relative positions of each embedding in the to-be-processed embedding sequence in the memory to obtain a position encoding vector corresponding to each embedding in the to-be-processed embedding sequence, and then each embedding in the to-be-processed embedding sequence is adjusted based on the position encoding vector. This can provide the multi-modal large language model with relative position information of each embedding in the to-be-processed embedding sequence, so that the multi-modal large language model can learn richer content semantic representations. In addition, since the size of the sliding window is fixed and the total number of the embedding of the start token and the embedding of the end token is relatively small, the attention range of the multi-modal large language model is relatively narrow. Compared with loading the entire target embedding sequence into the memory for global attention processing, the memory usage can be effectively reduced, and the inference efficiency can be significantly improved.

[0120] In some embodiments, the predicted embedding is generated by performing self-attention processing on the embedding of the start token, the embedding of the end token and the embeddings selected by the sliding window in the target embedding sequence, for example, the embedding of the start token, the embedding of the end token and the embeddings selected by the sliding window in the target embedding sequence can be spliced into a to-be-processed embedding sequence; the first key matrix and the first value matrix are constructed based on the to-be-processed embedding sequence; the first query matrix is constructed based on the embedding at the last position in the to-be-processed embedding sequence; and the predicted embedding is generated by performing attention processing based on the first query matrix, the first key matrix and the first value matrix.

[0121] For example, the predicted embedding can be determined in various ways. The first way of determining the predicted embedding is described in detail below.

[0122] In some embodiments, the attention processing is performed based on the first query matrix, the first key matrix and the first value matrix to generate the prediction embedding. For example, the attention score matrix can be obtained based on the product of the transpose of the first query matrix and the first key matrix, the attention score matrix is adjusted based on a scaling factor, the adjusted attention score matrix is normalized to obtain an attention weight matrix, where the attention weight matrix includes an attention weight corresponding to each embedding in the embedding sequence to be processed; and the prediction embedding is generated based on the product of the adjusted attention weight matrix and the first value matrix.

[0123] When the attention weight corresponding to an embedding in the embedding sequence to be processed is larger, it means that the importance of the embedding to the embedding at the last position in the embedding sequence to be processed is higher, and vice versa.

[0124] Therefore, when generating the prediction embedding, the multi-modal large language model can pay more attention to important information in the embedding sequence to be processed and ignore secondary information through the attention weight matrix, efficiently utilize and integrate the input information, and enhance the ability of the model to understand and generate complex outputs.

[0125] For example, the calculation formula of the prediction embedding is as shown in formula (1):

[0126] Where O l is the prediction embedding, Q l is the first query matrix, K l is the first key matrix, is the transpose of the first key matrix, is the scaling factor, d k is the dimension of each embedding in the embedding sequence to be processed, is the adjusted attention score matrix, and softmax is a normalization function, means that the adjusted attention score matrix is normalized, and V l is the first value matrix.

[0127] The second way of determining the prediction embedding is described in detail below.

[0128] In some embodiments, the attention processing is performed based on the first query matrix, the first key matrix and the first value matrix to generate the prediction embedding. For example, the attention weight matrix can be determined based on the first query matrix and the first key matrix, where the attention weight matrix includes respective attention weights of respective embeddings in the sequence of embeddings to be processed. The respective attention weights of the embedding of the start token and the embedding of the end token are adjusted to be greater than or equal to a preset weight threshold. The prediction embedding is generated based on the product of the adjusted attention weight matrix and the first value matrix.

[0129] In the process of generating the prediction embedding, after determining the attention weight matrix, the multimodal large language model needs to determine whether the attention weight corresponding to the embedding of the start token is greater than or equal to the weight threshold. When the attention weight corresponding to the embedding of the start token is less than the weight threshold, the attention weight corresponding to the embedding of the start token is adjusted so that the attention weight corresponding to the embedding of the start token is greater than or equal to the weight threshold. Similarly, the multimodal large language model also needs to determine whether the attention weight corresponding to the embedding of the end token is greater than or equal to the weight threshold. When the attention weight corresponding to the embedding of the end token is less than the weight threshold, the attention weight corresponding to the embedding of the end token is adjusted so that the attention weight corresponding to the embedding of the end token is greater than or equal to the weight threshold.

[0130] Based on this, in the attention weight matrix, the respective attention weights of the embedding of the start token and the embedding of the end token are ensured to be greater than or equal to the weight threshold. Since the respective attention weights of the embedding of the start token and the embedding of the end token are relatively high, when generating the prediction embedding, the multimodal large language model can keep paying attention to the key information carried by the embedding of the start token and the embedding of the end token through the attention weight matrix, avoid information loss caused by the sliding of the sliding window, and help the multimodal large language model to keep the attention distribution uniform and stable when processing long sequences, thereby reducing the perplexity of the multimodal large language model when reasoning, effectively improving the performance of the multimodal large language model, and also paying more attention to important information in the sequence of embeddings to be processed while ignoring secondary information, efficiently utilizing and integrating input information, and enhancing the ability of the model to understand and generate complex outputs.

[0131] Exemplarily, when the embedding corresponding to the starting mark has an attention weight of 0.2, the weight threshold is 0.25, the target difference between the attention weight corresponding to the embedding of the starting mark and the weight threshold is -0.05, the attention weight corresponding to the embedding of the starting mark needs to be increased by 0.05, then the adjustment amount of the attention weight corresponding to the remaining embeddings in the embedding sequence to be processed is obtained by multiplying the target difference and the proportion between the attention weights corresponding to the remaining embeddings in the embedding sequence to be processed, and then the attention weights corresponding to the remaining embeddings in the embedding sequence to be processed are adjusted based on the adjustment amount, so as to ensure that the sum of the attention weights corresponding to each embedding in the embedding sequence to be processed is equal to one.

[0132] In some embodiments, the predicted embedding belonging to the visual type is mapped to obtain a predicted visual feature, for example, a second key matrix and a second value matrix can be constructed according to the predicted embedding belonging to the visual type; a plurality of first learnable vectors are obtained, and a second query matrix is constructed according to the plurality of first learnable vectors, wherein the total dimension of the plurality of first learnable vectors is the same as the dimension of the reference visual feature; attention processing is performed based on the second query matrix, the second key matrix and the second value matrix to obtain the predicted visual feature.

[0133] The second key matrix is the product of the predicted embedding belonging to the visual type and a learnable second key parameter matrix, the second value matrix is the product of the predicted embedding belonging to the visual type and a learnable second value parameter matrix, the second key matrix and the second value matrix both include the hidden state of the predicted embedding belonging to the visual type, and the second query matrix is the product of the plurality of first learnable vectors and a learnable second query parameter matrix, and the second query matrix includes the hidden state of the plurality of first learnable vectors.

[0134] Exemplarily, the learnable second key parameter matrix and the learnable second value parameter matrix can be obtained by performing the following processing: obtaining an initialized deep learning model, an initialized second key parameter matrix and an initialized second value parameter matrix; predicting an input sequence through the initialized deep learning model to obtain a predicted output; determining a loss based on the input sequence and a true label of the input sequence; determining a gradient for the initialized second key parameter matrix and the initialized second value parameter matrix based on the loss through a back propagation algorithm, and updating the values of the initialized second key parameter matrix and the initialized second value parameter matrix according to the gradient by calling an optimizer until the model converges, thereby obtaining the learnable second key parameter matrix and the learnable second value parameter matrix.

[0135] In general, the second key matrix and the second value matrix are constructed by a plurality of continuous prediction embeddings of the visual type, assuming that the second key matrix and the second value matrix are both constructed by 64 continuous prediction embeddings of the visual type, the dimension of the prediction embedding is 1*768, the dimension of the reference visual feature is 256*768, the number of the first learnable vectors is 256, the dimension of each first learnable vector is 1*768, the total dimension of the 256 first learnable vectors is 256*768, the second query matrix is constructed by 256 first learnable vectors, the dimensions of the second query parameter matrix, the second key parameter matrix and the second value parameter matrix are all 768*768, then the second key matrix with a dimension of 64*768 is obtained by multiplying the 64 prediction embeddings of the visual type with the second key parameter matrix, the second value matrix with a dimension of 64*768 is obtained by multiplying the 64 prediction embeddings of the visual type with the second value parameter matrix, the second query matrix with a dimension of 256*768 is obtained by multiplying the 256 first learnable vectors with the second query parameter matrix, and the prediction visual feature with a dimension of 256*768 is obtained by performing attention processing based on the second query matrix, the second key matrix and the second value matrix.

[0136] Here, the above "performing attention processing based on the second query matrix, the second key matrix and the second value matrix to obtain the prediction visual feature" can be determined by performing the following processing: determining a candidate weight matrix based on the second query matrix and the second key matrix, wherein the candidate weight matrix includes the attention weight corresponding to each embedding in the prediction sequence; and generating the prediction visual feature based on the product of the candidate weight matrix and the second value matrix.

[0137] Based on this, the prediction embedding output by the multi-modal large language model is in a text embedding space, and the prediction embedding has language information. In order to ensure the effective generation of the prediction image, it is necessary to map the prediction embedding of the visual type to obtain a prediction visual feature in a visual feature space. For example, a second query matrix constructed by a plurality of first learnable vectors, and a second key matrix and a second value matrix constructed by prediction embeddings of the visual type, and then performing attention processing on the second query matrix, the second key matrix and the second value matrix to obtain the prediction visual feature. Since the source of the second query matrix is different from the source of the second key matrix and the second value matrix, in fact, cross-attention processing is performed on the second query matrix, the second key matrix and the second value matrix to obtain the prediction visual feature, so that the prediction visual feature is aligned with the plurality of first learnable vectors in the visual feature space, and the prediction embedding of the visual type is mapped to the prediction visual feature with visual information.

[0138] For example, referring to FIG. 4, FIG. 4 is an optional flow diagram for obtaining a predicted visual feature according to an embodiment of the present application. The multimodal large language model is internally provided with N stacked first query Transformer layers, the first query Transformer layer including a first self-attention sublayer, a first cross-attention sublayer, and a first feedforward sublayer. The second Transformer layer is used to map the predicted embedding belonging to the visual type. After the first learnable vector and the predicted embedding belonging to the visual type are input into the first query Transformer layer, the first self-attention sublayer can perform self-attention processing on the first learnable vector to obtain a first hidden vector. The first cross-attention sublayer can perform cross-attention processing on the first hidden vector and the predicted embedding belonging to the visual type to obtain a second hidden vector. The first feedforward sublayer is used to map the second hidden vector to obtain the predicted visual feature.

[0139] In addition, the first query Transformer layer can be pre-trained in combination with a text Transformer layer. The text Transformer layer is used to encode the text. During the pre-training process, the output of the first query Transformer layer is aligned with the output of the text Transformer layer. By pre-training the first query Transformer layer, the first learnable vector with visual information can be learned, so that the first query Transformer layer can filter useful language information from the predicted embedding through cross-attention processing, and convert the useful language information from the text embedding space to the visual feature space, which can effectively improve the quality of the predicted image.

[0140] It is worth noting that, similar to aligning the output space of the multimodal large language model with the visual feature space, the input space of the multimodal large language model needs to be aligned with the visual feature space when processing the embedding of the reference visual feature, that is, mapping the embedding of the reference visual feature. The mapping process of the reference visual feature is described in detail below.

[0141] In some embodiments, the determination of the target embedding sequence spliced by the embedding of the reference text, the embedding of the start token, the embedding of the reference visual feature, and the embedding of the end token based on the multi-modal large language model can be based on the multi-modal large language model, constructing a third key matrix and a third value matrix according to the reference visual feature; obtaining a plurality of second learnable vectors pre-trained, constructing a third query matrix according to the plurality of second learnable vectors, wherein the second learnable vector has the same dimension as the embedding of the reference text; performing attention processing based on the third query matrix, the third key matrix, and the third value matrix to obtain the embedding of the reference visual feature; and splicing the embedding of the reference text, the embedding of the start token, the embedding of the reference visual feature, and the embedding of the end token to obtain the target embedding sequence.

[0142] Among them, since the reference text, the start token and the end token are all input data corresponding to the text modality, the embedding of the reference text, the embedding of the start token and the embedding of the end token can be determined by embedding processing the reference text, the start token and the end token respectively. When embedding processing the reference text, the reference text needs to be segmented first, and then the segmentation result of the reference text is embedded. The embedding of the reference text, the embedding of the start token and the embedding of the end token are all in the text embedding space, and the dimensions of the embedding of the reference text, the embedding of the start token and the embedding of the end token are consistent.

[0143] Among them, the third key matrix is the product of the reference visual feature and the learnable third key parameter matrix, the third value matrix is the product of the reference visual feature and the learnable third value parameter matrix, the third key matrix and the third value matrix both include the hidden state of the reference visual feature, the third query matrix is the product of the plurality of second learnable vectors and the learnable third query parameter matrix, and the third query matrix includes the hidden state of the plurality of second learnable vectors.

[0144] Exemplarily, the third key matrix and the third value matrix are constructed by the reference visual feature with a dimension of 256*768, the number of the second learnable vectors is 64, the dimension of each second learnable vector is 1*768, the dimension of each second learnable vector is the same as the dimension of the embedding of the reference text, the third query matrix is constructed by the 64 second learnable vectors, the dimensions of the third query parameter matrix, the third key parameter matrix and the third value parameter matrix are all 768*768, then the third key matrix with a dimension of 256*768 is obtained by multiplying the reference visual feature and the third key parameter matrix, the third value matrix with a dimension of 256*768 is obtained by multiplying the reference visual feature and the third value parameter matrix, the third query matrix with a dimension of 64*768 is obtained by multiplying the 64 second learnable vectors and the third query parameter matrix, the attention processing is performed based on the third query matrix, the third key matrix and the third value matrix, and the embedding of the reference visual feature with a dimension of 1*768 is obtained, therefore, when the predicted token output by the multi-modal large language model is the start token, the first to 64th predicted embeddings generated subsequently are used to determine the predicted visual feature, and the 65th predicted embedding generated subsequently is used to determine the predicted token.

[0145] Based on this, the reference visual feature extracted from the reference image is in the visual feature space, the reference visual feature has visual information, in order to ensure that the multi-modal large language model can effectively process the input data, it is necessary to map the reference visual feature to obtain the embedding of the reference visual feature in the text embedding space, for example, the third query matrix constructed by the plurality of second learnable vectors, and the third key matrix and the third value matrix constructed by the reference visual feature, then the attention processing is performed on the third query matrix, the third key matrix and the third value matrix, since the source of the third query matrix is different from the source of the third key matrix and the third value matrix, in fact, the cross-attention processing is performed on the third query matrix, the third key matrix and the third value matrix to obtain the embedding of the reference visual feature, so that the embedding of the reference visual feature is aligned with the second learnable vector in the text embedding space, the reference visual feature is mapped to the embedding of the reference visual feature with language information, so that the embedding of the reference text, the embedding of the start token, the embedding of the reference visual feature and the embedding of the end token are all in the text embedding space, which can effectively reduce the burden of the multi-modal large language model to learn and align the visual modality and the text modality, thereby alleviating the forgetting problem.

[0146] For example, referring to FIG. 5, FIG. 5 is an optional flow diagram for obtaining an embedding of a reference visual feature according to an embodiment of the present application. The multi-modal large language model is internally provided with N stacked second query Transformer layers, the second query Transformer layer including a second self-attention sub-layer, a second cross-attention sub-layer, and a second feed-forward sub-layer. The second Transformer layer is configured to map the reference visual feature. After the plurality of second learnable vectors and the reference visual feature are input into the second query Transformer layer, the second self-attention sub-layer is configured to perform self-attention processing on the plurality of second learnable vectors to obtain a plurality of third hidden vectors. The second cross-attention sub-layer is configured to perform cross-attention processing on the plurality of third hidden vectors and the reference visual feature to obtain a plurality of fourth hidden vectors. The second feed-forward sub-layer is configured to map the plurality of fourth hidden vectors to obtain the embedding of the reference visual feature.

[0147] In addition, the second query Transformer layer can be pre-trained in combination with an image Transformer layer. The image Transformer layer is configured to encode an image. During the pre-training process, the output of the second query Transformer layer is aligned with the output of the image Transformer layer. By pre-training the second query Transformer layer, a plurality of second learnable vectors with language information can be learned. The second query Transformer layer can filter useful visual information from the predicted embedding through cross-attention processing and convert the useful visual information from a visual feature space to a text embedding space. This can effectively reduce the burden of the multi-modal large language model to learn to align the visual modality and the text modality.

[0148] In some embodiments, referring to FIG. 6, FIG. 6 is an optional flow diagram for generating a predicted image according to an embodiment of the present application. A predicted image and a predicted text are generated based on a predicted sequence. For example, the predicted text can be generated based on a predicted token. A randomly generated first noise image is obtained. The first noise image is input into a diffusion model. The predicted visual feature is used as a diffusion condition of the diffusion model to denoise the first noise image, thereby generating the predicted image.

[0149] The first noise image is randomly generated, and the pixels of the first noise image have randomness. The size of the first noise image can be the same as the size of the reference image, i.e., the height of the first noise image is the same as the height of the reference image, and the width of the first noise image is the same as the width of the reference image. In this way, the size of the predicted image generated based on the first noise image is the same as the size of the reference image.

[0150] The diffusion model has two main processes: a forward process and a backward process. The forward process is also called a diffusion process. In the forward process, the diffusion model gradually adds Gaussian noise to an image until the image becomes completely random noise. For example, the diffusion process as a whole can be a parameterized Markov chain. The backward process is also called an inverse diffusion process. In the backward process, the diffusion model uses a series of Markov chains to gradually remove prediction noise at each time step, thereby recovering data from Gaussian noise. Therefore, the diffusion model can denoise a first noisy image based on a predicted visual feature in the backward process, thereby generating a predicted image, i.e., a predicted image that satisfies the predicted visual feature is obtained through denoising processing of the diffusion model.

[0151] For example, the diffusion model includes a Unet network. The Unet network can denoise a noisy image in the backward process. For any feature extractor in the Unet network, cross-attention processing is performed on the output of the current feature extractor based on the predicted visual feature to obtain cross-attention features. The cross-attention features are input to the next feature extractor. Therefore, for each feature extractor in the Unet network, cross-attention processing is performed on the output of the current feature extractor using the predicted visual feature. Cross-attention processing can help the diffusion model pay more attention to key areas in the image and improve the accuracy of denoising. The feature extractor of the Unet network can be connected to a cross-attention layer. In the cross-attention layer, query features are determined based on the output of the current feature extractor, key features and value features are determined based on the predicted visual feature, and cross-attention features are determined by the query features, the key features, and the value features.

[0152] Based on this, the diffusion model is used to gradually remove prediction noise of the first noisy image through multiple time steps, thereby recovering a high-quality and detailed predicted image. Based on the global attention mechanism, the predicted visual feature is injected into the diffusion model at each time step when the prediction noise is predicted. This is equivalent to regarding the predicted visual feature as a global latent code, i.e., a global feature. Through cross-attention processing, the diffusion model considers the predicted visual feature in each generation process to ensure that the predicted image matches the predicted visual feature, effectively improving the accuracy of the predicted image and thereby improving the quality of the predicted image.

[0153] In some embodiments, before the reference text is input to the multi-modal large language model with the start token and the end token added at both ends of the reference visual feature, the multi-modal data processing method further comprises: obtaining a first sample image, a first sample text, a target image, and a target text, wherein the first sample text is used to describe the content in the first sample image, the target text is used to describe the content in the target image, and the target image is an image associated with the first sample image; extracting a first visual feature of the first sample image, and inputting the first sample text and the first visual feature with the start token and the end token added at both ends to the multi-modal large language model for inference; determining a first loss of the multi-modal large language model for inference based on the target text, and determining a second loss of the multi-modal large language model for inference based on the target image; and training the multi-modal large language model based on the first loss and the second loss.

[0154] Similar to the reference image, the first sample image and the target image are both visual modal data, and similar to the reference text, the first sample text and the target text are both text modal data. The target image is associated with the first sample image, for example, when a story is generated using the multi-modal large language model, the target image is an image of the first sample image in the next scene, for another example, when a step is generated using the multi-modal large language model, the target image is an image of the first sample image in the next step, and for another example, when content is enhanced using the multi-modal large language model, the target image is an image similar to the first sample image.

[0155] For example, referring to FIG. 7, FIG. 7 is an optional flowchart of training a multi-modal large language model according to an embodiment of the present application.

[0156] Before the multi-modal large language model is used to generate the predicted text and the predicted image, the multi-modal large language model needs to be trained. During the model training process, the first sample image is feature-extracted to obtain a first visual feature, and the first sample text and the first visual feature with the start token and the end token added at both ends are input to the trained multi-modal large language model for inference. Similar to the generation process of the predicted text, the multi-modal large language model can output corresponding training tokens, and then generate corresponding training text. Similarly, similar to the generation process of the predicted image, the multi-modal large language model can output corresponding training visual features.

[0157] Therefore, by determining the first loss based on the target text and the training text, and by extracting the target visual feature of the target image and then determining the second loss based on the target visual feature and the training visual feature, the multi-modal large language model can be trained based on the first loss and the second loss, thereby improving the prediction accuracy of the multi-modal large language model.

[0158] For example, the first loss can be calculated by inputting the target text and the training text into a cross-entropy loss function, and the second loss can be calculated by inputting the target visual feature and the training visual feature into a cosine similarity loss function. In addition, the first loss can also be calculated in other ways, and the second loss can also be calculated in other ways. The loss function used to determine the first loss and the second loss can also include any one of the following: a mean square error loss function, a multi-label classification loss function, and a triplet loss function. The embodiments of the present application are not limited in this regard.

[0159] In some embodiments, during the model training process, the multi-modal large language model generates an output sequence by performing the task of predicting the next embedding. The first sample text, the start token, the first visual feature, the end token, the target text, and the target image are input into the multi-modal large language model after being spliced. Since the target text and the target image are both label data, inputting the target text and the target image into the multi-modal large language model allows the multi-modal large language model to generate training text and training images without waiting for the embedding of a certain position before generating the embedding of the next position. This enables parallel training, improves training efficiency, and makes it easier for the multi-modal large language model to converge during the training process.

[0160] In some embodiments, the first loss and the second loss are backpropagated to update the parameters of the multi-modal large language model. The process of calculating the first loss and the second loss and updating the parameters is iterated multiple times until the loss converges, the iteration process is stopped, and the trained multi-modal large language model is obtained.

[0161] For example, backpropagation is achieved through a backpropagation algorithm, which calculates the gradient of each neuron from the output layer to the input layer and updates the weights and biases of the neurons based on the gradient. Gradient descent is used to continuously update the parameters to reduce the loss value. Gradient descent can use various gradient descent algorithms, such as batch gradient descent, stochastic gradient descent, adaptive gradient descent, and momentum gradient descent.

[0162] In some embodiments, the multi-modal large language model is trained based on the first loss and the second loss. For example, the first loss and the second loss can be weighted and summed based on loss weights to obtain a model loss, and the multi-modal large language model is trained based on the model loss. Based on this, by determining an appropriate model loss, the training effect of the multi-modal large language model can be improved.

[0163] For example, the loss weights can be determined in various manners. In one manner of determining the loss weights, the loss weights are preset. For example, the loss weights corresponding to the first loss and the second loss are both set to 1. For another example, the loss weight corresponding to the first loss is set to 1, and the loss weight corresponding to the second loss is set to 10.

[0164] Another manner of determining the loss weights is described below.

[0165] In some embodiments, the multi-modal large language model is trained based on the first loss and the second loss. For example, the first loss and the second loss can be concatenated and input into a regression model to obtain the loss weights corresponding to the first loss and the second loss, respectively, where the regression model is trained jointly with the multi-modal large language model. The model loss is obtained by weighted summation of the first loss and the second loss based on the loss weights, and the multi-modal large language model is trained based on the model loss.

[0166] For example, the weight of the first loss L1 is The weight of the second loss L2 is The model loss obtained by weighted summation of the first loss and the second loss is

[0167] Based on this, the first loss and the second loss are concatenated and input into a regression model, which can predict the loss weights corresponding to the first loss and the second loss, respectively. The loss weights are used to represent the contribution of the first loss and the second loss to the model loss. The model loss is obtained by weighted summation of the first loss and the second loss based on the loss weights, which can adaptively adjust the first loss and the second loss, and can avoid overfitting. By jointly training the regression model and the multi-modal large language model, the regression model can more accurately predict the loss weights, improve the training effect of the multi-modal large language model, and thus improve the prediction accuracy of the multi-modal large language model.

[0168] In some embodiments, referring to FIG. 8, which is a first optional flowchart for jointly training a feature extractor and a diffusion model according to an embodiment of the present application. The first visual feature is extracted by the feature extractor. Before extracting the first visual feature of the first sample image, the multi-modal data processing method further includes: obtaining a second sample image, inputting the second sample image into the feature extractor, and extracting a second visual feature of the second sample image; obtaining a randomly generated second noise image, inputting the second noise image into the diffusion model, and performing denoising processing on the second noise image by taking the second visual feature as a diffusion condition of the diffusion model to generate a reconstructed image; determining a third loss based on the difference between the reconstructed image and the second sample image, and jointly training the feature extractor and the diffusion model based on the third loss.

[0169] wherein, similar to the reference image, the second sample image is also visual modality data, the feature extractor is configured to perform feature extraction on the second sample image, and the second visual feature of a more abstract and more informative is extracted from the second sample image; the second noise image is randomly generated, and the pixels of the second noise image have randomness. The size of the second noise image can be the same as the size of the second sample image, i.e., the height of the second noise image is the same as the height of the second sample image, and the width of the second noise image is the same as the width of the second sample image, so that the reconstructed image generated based on the second noise image is the same size as the second sample image.

[0170] wherein, the diffusion model is used to remove the predicted noise of the second noise image step by step through multiple time steps, so as to recover the second sample image with high quality and rich details; based on the global attention mechanism, the second visual feature is injected into the diffusion model when predicting the noise at each time step, which is equivalent to taking the second visual feature as a global latent code, i.e., a global feature, and through cross-attention processing, the diffusion model considers the second visual feature in each generation process to ensure that the second sample image matches the second visual feature.

[0171] Based on this, the feature extractor is used to perform feature extraction on the second sample image, which is equivalent to tokenization of the second sample image, and the second visual feature is used as a diffusion condition of the diffusion model to generate the reconstructed image, which is equivalent to de-tokenization of the second visual feature. The third loss is determined through the difference between the reconstructed image and the second sample image, and then the feature extractor and the diffusion model are jointly trained based on the third loss, which ensures efficient cooperation between the feature extractor and the diffusion model, so that the feature extractor learns to extract more meaningful second visual features from the second sample image, and the diffusion model can use the second visual feature as a guide for denoising processing, which helps to generate more accurate reconstructed images in the denoising process, and can improve the feature extraction capability of the feature extractor and the image reconstruction capability of the diffusion model at the same time.

[0172] For example, the third loss can be calculated by inputting the reconstructed image and the second sample image into a mean square error loss function, in addition to which, the third loss can also be calculated by other means. The loss function for determining the third loss includes but is not limited to any one of the following: a cross-entropy loss function, a multi-label classification loss function, and a triplet loss function, which are not limited by the embodiments of the present application.

[0173] In some embodiments, the feature extractor can be frozen, the first visual feature is extracted by the feature extractor, and before the first visual feature of the first sample image is extracted, the multi-modal data processing method further comprises: obtaining a second sample image, inputting the second sample image into the feature extractor, extracting a second visual feature of the second sample image; obtaining a randomly generated second noise image, inputting the second noise image into the diffusion model, performing denoising processing on the second noise image by taking the second visual feature as the diffusion condition of the diffusion model to generate a reconstructed image; determining a third loss based on the difference between the reconstructed image and the second sample image, and training the diffusion model based on the third loss. Based on this, the image reconstruction capability of the diffusion model can be improved.

[0174] Here, the feature extractor and the diffusion model can be trained in stages. In the first stage, the parameters of the diffusion model are frozen, and a fourth visual feature of the reconstructed image is extracted. A fifth loss is determined based on the second visual feature and the fourth visual feature. The parameters of the feature extractor are updated based on the fifth loss to obtain a trained feature extractor. In the second stage, the parameters of the trained feature extractor are frozen, the parameters of the diffusion model are updated based on the third loss to obtain a trained diffusion model, and the trained feature extractor and the trained diffusion model are fine-tuned based on the third loss until both the feature extractor and the diffusion model converge.

[0175] The embodiments of the present application train in stages. In each of the first two stages, only one model is trained, and each model can be optimized individually on its specific task, which can effectively utilize computing resources. Training each model individually can more easily understand the role and performance of each model, and in the joint training stage, the contribution of each model to the final result can be more clearly seen.

[0176] In some embodiments, referring to FIG. 9, FIG. 9 is an optional second flowchart provided by the embodiments of the present application for retraining the diffusion model; after the multi-modal large language model is trained based on the first loss and the second loss, the multi-modal data processing method further comprises: obtaining a third sample image, a second sample text, and a label image, wherein the second sample text is used to describe the content in the third sample image, and the label image is an image associated with the third sample image; extracting a third visual feature of the third sample image, inputting the second sample text and the third visual feature with a start token and an end token added at both ends into the trained multi-modal large language model for inference; obtaining a randomly generated third noise image, inputting the third noise image into the trained diffusion model, and performing denoising processing on the third noise image by taking the visual feature output by the trained multi-modal large language model as the diffusion condition of the diffusion model to generate an associated image; determining a fourth loss according to the associated image and the label image, and retraining the diffusion model based on the fourth loss.

[0177] The third sample image and the label image are both visual modal data similar to the reference image, and the second visual feature refers to more abstract and more informative features extracted from the third sample image; the third noise image is randomly generated, and the pixels of the third noise image have randomness. The size of the third noise image can be the same as the size of the third sample image, that is, the height of the third noise image is the same as the height of the third sample image, and the width of the third noise image is the same as the width of the third sample image, so that the reconstructed image generated based on the third noise image is the same size as the third sample image.

[0178] The label image is an image associated with the third sample image. For example, when a multi-modal large language model is used to generate a story, the label image is an image of the next scene of the third sample image. For another example, when a multi-modal large language model is used to generate a step, the label image is an image of the next step of the third sample image. For another example, when a multi-modal large language model is used to enhance content, the label image is an image similar to the third sample image.

[0179] The diffusion model is used to gradually remove the predicted noise of the third noise image through multiple time steps. Starting from the last time step, the noise is gradually reduced, thereby recovering the third sample image with high quality and rich details. Based on the global attention mechanism, the second visual feature is injected into the diffusion model when predicting the noise at each time step, which is equivalent to taking the second visual feature as a global latent code, i.e., a global feature. Through cross-attention processing, the attention weight between the second visual feature and the global latent code is calculated. According to the calculated attention weight, the information of the global latent code is fused into the second visual feature. Through the second visual feature fused with the information of the global latent code, the noise of the current time step is predicted, and the predicted noise is removed from the third noise image to obtain the third sample image after denoising. This ensures that the third sample image matches the second visual feature.

[0180] Based on this, feature extraction is performed on the third sample image to obtain a third visual feature. The second sample text and the third visual feature with a start tag and an end tag added at both ends are input into the trained multi-modal large language model for reasoning. Similar to the output process of the predicted visual feature, the multi-modal large language model can output a corresponding visual feature. Then, the visual feature output by the multi-modal large language model is used as a diffusion condition of the diffusion model to perform denoising processing on the third noisy image to generate a related image. The fourth loss is determined based on the related image and the label image. The diffusion model is trained again based on the fourth loss, and the diffusion model can generate a related image closer to the label image. In other words, the diffusion model is fine-tuned to better align the output space of the multi-modal large language model with the visual feature space, thereby enhancing the style consistency of the generated image.

[0181] For example, the fourth loss can be calculated by inputting the related image and the label image into a mean square error loss function. In addition, the fourth loss can also be calculated in other ways. The loss function used to determine the fourth loss includes, but is not limited to, any one of the following: a cross-entropy loss function, a multi-label classification loss function, and a triplet loss function. The embodiments of the present application are not limited in this regard.

[0182] In some embodiments, the first sample image, the first sample text, the target image, and the target text are obtained. The plurality of continuous video frames and the caption text corresponding to each video frame can be obtained. The text description model is called to generate the description text corresponding to each video frame. A prompt instruction for prompting content enhancement with reference to the description text is constructed. The prompt instruction, each caption text, and the corresponding description text are concatenated and input into the target large language model. The target large language model is used to enhance the content of each caption text to obtain the enhanced text corresponding to each video frame. The last video frame is determined as the target image, and the remaining video frames are determined as the first sample image. The enhanced text corresponding to the target image is determined as the target text, and the enhanced text corresponding to the first sample image is determined as the first sample text.

[0183] The plurality of continuous video frames can be obtained from video sources such as animations, movies, and television programs. For example, in an animation, a picture can be taken according to a continuous scene, or a picture can be taken at a fixed time interval. The plurality of continuous pictures obtained in this way are used as the plurality of continuous video frames. Each video frame can be an RGB image, a grayscale image, a binary image, a depth image, or the like. The embodiments of the present application are not limited in this regard.

[0184] The subtitle text refers to the textual display of spoken dialogue, sound effects, musical cues, and other auditory information in video content such as animations, movies, and television programs. The subtitle text is usually synchronized with the audio in the multimedia content, allowing viewers to obtain the corresponding content by reading the subtitle text in the absence of sound or when additional understanding is required, thereby improving the viewing experience of the video content.

[0185] The text description model can perform target detection on the video frame to obtain a target detection result including a detected object. The text description model can also perform feature extraction on the video frame to obtain picture features including scene features and action features of the detected object. Then, the text description model can perform text prediction on the video frame based on the target detection result and the picture features to obtain a description text corresponding to the video frame. The description text is used to represent background information such as characters, environment, and events of the video frame.

[0186] The target large language model belongs to a large language model. The large language model is a deep learning model trained using a large amount of text data and can generate natural language text or understand the meaning of language text. The large language model generally uses a recurrent neural network (RNN) or a variant such as a long short-term memory network (LSTM) and a gated recurrent unit (GRU) to capture context information in a text sequence, thereby realizing natural language text generation, language model evaluation, text classification, sentiment analysis, and other tasks. In the field of natural language processing, the large language model has been widely applied, such as speech recognition, machine translation, automatic summarization, dialogue systems, intelligent question answering, and the like.

[0187] For example, the prompt instruction can be "Please refer to the following description text to enhance the content of the subtitle text". The prompt instruction, each subtitle text, and the corresponding description text are concatenated and input into the target large language model. Assuming that the number of description texts and subtitle texts is two, the model input form of the target large language model can be "description text 1, subtitle text 1, description text 2, subtitle text 2". The model input form of the target large language model can also be "description text 1, description text 2, subtitle text 1, subtitle text 2". The target large language model can fuse the background information carried by the description text into the subtitle text, thereby achieving content enhancement of the subtitle text.

[0188] Based on this, first, the video frames and the corresponding subtitle texts are obtained, then the description texts of the video frames are generated, and then the content of the subtitle texts is enhanced based on the description texts by using the target large language model to obtain enhanced texts. Then, the video frames and the corresponding enhanced texts are used as image-text pairs to construct a data set, the data set contains high-resolution video frames and high-quality enhanced texts, and the data set can contain a large number of image-text pairs. Then, the last video frame in the plurality of continuous video frames is determined as a target image, and the remaining video frames in the plurality of continuous video frames are determined as first sample images, so that the target image is an image associated with the first sample images, and the target image is used to indicate the next scene of the first sample images. On this basis, the enhanced text corresponding to the target image is determined as a target text, and the enhanced text corresponding to the first sample image is determined as a first sample text, so as to ensure that the target text is used to describe the content in the target image, and the first sample text is used to describe the content in the first sample image. The multi-modal large language model is trained by using the first sample image, the first sample text, the target image, and the target text, so that the multi-modal large language model can effectively process story generation, step generation and other tasks, and the generation quality of the multi-modal large language model for multi-modal content is improved.

[0189] The complete process of the multi-modal data processing method will be described in detail below, which specifically includes a training phase and an inference phase.

[0190] The training phase will be described in detail below, which can be divided into three training sub-phases. The following takes a server (for example, the server 102) as an example to illustrate.

[0191] In the first training sub-phase, the diffusion model and the feature extractor are pre-trained.

[0192] First, a second sample image is obtained, and the second sample image is input into the feature extractor to extract second visual features of the second sample image.

[0193] Then, a second noise image generated randomly is obtained, and the second noise image is input into the diffusion model. The second visual features are used as diffusion conditions of the diffusion model to perform denoising processing on the second noise image to generate a reconstructed image.

[0194] Then, a third loss is determined based on the difference between the reconstructed image and the second sample image, and the feature extractor and the diffusion model are jointly trained based on the third loss.

[0195] In the second training sub-phase, the multi-modal large language model is trained.

[0196] First, a plurality of continuous video frames and the corresponding subtitle texts of each video frame are obtained.

[0197] Then, a text description model is called to generate a description text corresponding to each video frame.

[0198] Then, a prompt instruction for prompting content enhancement with reference to the description text is constructed, and the prompt instruction, each subtitle text and the corresponding description text are spliced and input into a target large language model. The target large language model is used to enhance the content of each subtitle text, and an enhanced text corresponding to each video frame is obtained.

[0199] Then, the last video frame is determined as a target image, the remaining video frames are determined as first sample images, the enhanced text corresponding to the target image is determined as a target text, and the enhanced text corresponding to the first sample images is determined as first sample texts. The first sample texts are used to describe the content in the first sample images, and the target text is used to describe the content in the target image. The target image is an image associated with the first sample images.

[0200] Then, a first visual feature of the first sample image is extracted, and the first sample text and the first visual feature with a start mark and an end mark added at both ends are input into the multi-modal large language model for inference.

[0201] Then, a first loss of the multi-modal large language model for inference is determined based on the target text, and a second loss of the multi-modal large language model for inference is determined based on the target image.

[0202] Then, the first loss and the second loss are spliced and input into a regression model for prediction to obtain loss weights corresponding to the first loss and the second loss respectively. The regression model is trained jointly with the multi-modal large language model.

[0203] Then, the first loss and the second loss are weighted and summed based on the loss weights to obtain a model loss, and the multi-modal large language model is trained based on the model loss.

[0204] In the third training sub-stage, the diffusion model is fine-tuned.

[0205] First, a third sample image, a second sample text and a label image are obtained. The second sample text is used to describe the content in the third sample image, and the label image is an image associated with the third sample image.

[0206] Then, a third visual feature of the third sample image is extracted, and the second sample text and the third visual feature with a start mark and an end mark added at both ends are input into the trained multi-modal large language model for inference.

[0207] Then, a third noise image randomly generated is obtained, the third noise image is input to the trained diffusion model, and the visual feature output by the trained multi-modal large language model is taken as a diffusion condition of the diffusion model to perform denoising processing on the third noise image, so as to generate a correlation image.

[0208] Then, a fourth loss is determined according to the correlation image and the label image, and the diffusion model is trained again based on the fourth loss.

[0209] The inference phase is described in detail below, and the following takes a server (for example, the server 102) as an execution subject for example.

[0210] First, a reference image and a reference text are obtained, where the reference text is used to describe the content in the reference image.

[0211] Then, a reference visual feature of the reference image is extracted, and the reference text and the reference visual feature with a start token and an end token added at both ends are input to the multi-modal large language model.

[0212] Then, based on the multi-modal large language model, a third key matrix and a third value matrix are constructed according to the reference visual feature.

[0213] Then, a plurality of second learnable vectors pre-trained are obtained, and a third query matrix is constructed according to the plurality of second learnable vectors, where the dimension of the second learnable vector is the same as the dimension of the embedding of the reference text.

[0214] Then, attention processing is performed based on the third query matrix, the third key matrix and the third value matrix to obtain an embedding of the reference visual feature.

[0215] Then, the embedding of the reference text, the embedding of the start token, the embedding of the reference visual feature and the embedding of the end token are spliced to obtain a target embedding sequence.

[0216] Then, the embedding of the start token, the embedding of the end token and the embedding selected by the sliding window in the target embedding sequence are spliced into a to-be-processed embedding sequence.

[0217] Then, a first key matrix and a first value matrix are constructed based on the to-be-processed embedding sequence.

[0218] Then, a first query matrix is constructed based on the embedding of the last position in the to-be-processed embedding sequence.

[0219] Then, an attention weight matrix is determined based on the first query matrix and the first key matrix, where the attention weight matrix includes the attention weight of each embedding in the to-be-processed embedding sequence.

[0220] Then, the starting token embedding or the ending token embedding is adjusted to be greater than or equal to a preset weight threshold according to the corresponding attention weight.

[0221] Then, the predicted embedding is generated based on the product of the adjusted attention weight matrix and the first value matrix.

[0222] Then, the predicted embedding belonging to the text type is classified, and the predicted token is output.

[0223] Then, the second key matrix and the second value matrix are constructed according to the predicted embedding belonging to the visual type.

[0224] Then, a plurality of first learnable vectors are obtained, and a second query matrix is constructed according to the plurality of first learnable vectors, wherein the total dimension of the plurality of first learnable vectors is the same as the dimension of the reference visual feature.

[0225] Then, the predicted visual feature is obtained by performing attention processing based on the second query matrix, the second key matrix and the second value matrix.

[0226] Then, the predicted text is generated based on the predicted token.

[0227] Then, a first noise image randomly generated is obtained, the first noise image is input into the diffusion model, and the predicted visual feature is used as a diffusion condition of the diffusion model to perform denoising processing on the first noise image, thereby generating a predicted image.

[0228] Based on this, by obtaining a reference image and a reference text used to describe the content in the reference image, extracting a reference visual feature of the reference image, and adding a starting token and an ending token at both ends of the reference visual feature, the reference visual feature and the reference text can be distinguished based on the starting token and the ending token in the subsequent process. On this basis, the reference text and the reference visual feature with the starting token and the ending token added at both ends are input into the multi-modal large language model, a predicted sequence is output based on the multi-modal large language model, and a predicted image and a predicted text used to describe the predicted image are generated based on the predicted sequence. Since the starting token embedding, the ending token embedding and the embedding selected by the sliding window in the target embedding sequence are used for attention processing during inference, even if multiple rounds of multi-modal data processing are performed, the multi-modal large language model can focus on the information carried by the starting token embedding, the ending token embedding and the embedding selected by the sliding window in the target embedding sequence through attention processing, thereby reducing the perplexity during inference of the multi-modal large language model, effectively improving the performance of the multi-modal large language model, and further improving the generation quality of the multi-modal content.

[0229] The principle of attention construction of the multi-modal large language model is described in detail below.

[0230] For example, referring to FIG. 10, FIG. 10 is an optional distribution diagram of the attention score distribution provided by the embodiment of the present application.

[0231] The test embedding sequence is obtained by splicing the embedding of the start token, the embedding of the test text, the embedding of the start token, the embedding of the end token, and the embedding of the test visual feature. The test visual feature is obtained by feature extraction on the test image. When the multi-modal large language model is used to perform global attention processing on the test embedding sequence, the query matrix and the key matrix are determined by each embedding in the test embedding sequence. Then, the attention score distribution of all embeddings is determined according to the product of the transpose of the query matrix and the key matrix.

[0232] Based on this, as shown in FIG. 10, the attention score distribution of all embeddings is plotted in the same distribution diagram. The abscissa is the index of the key matrix, which is used to indicate the corresponding embedding. The ordinate is the index of the query matrix, which is used to indicate the corresponding embedding. As can be seen, most queries mainly focus on the embedding of the start token, the embedding of the punctuation, the embedding of the start token, and the embedding of the end token. Since the norm of the embedding of the punctuation is very small, the contribution of the embedding of the punctuation to the reasoning result is low. Therefore, when constructing the attention of the multi-modal large language model, only the information carried by the embedding of the start token, the embedding of the end token, and the embedding selected by the sliding window in the target embedding sequence needs to be focused on by attention processing. This can reduce the perplexity of the multi-modal large language model during reasoning, effectively improve the performance of the multi-modal large language model, and further improve the generation quality of the multi-modal content. It can also effectively reduce the memory usage and significantly improve the reasoning efficiency.

[0233] The evaluation method of the multi-modal large language model provided by the embodiment of the present application and other models is described in detail below.

[0234] In the first evaluation method, the latent diffusion model (Latent Diffusion Model, LDM), the story generation model (StoryGen), and the multi-modal large language model provided by the embodiment of the present application are used to generate images, respectively. The evaluation indicators of each model in FID and CLIP score can be obtained, as shown in Table 1 below:

[0235] Table 1

[0236] Wherein, FID is the full name of Frechet Inception Distance, FID score is a measure of the distance between the feature vectors of real images and generated images, the lower the FID score, the smaller the distance between the feature vectors of real images and generated images, and the higher the quality of the images generated by the model; CLIP is the full name of Contrastive Language-Image Pre-training, and the CLIP score is used to evaluate the correlation between images and texts, and the higher the CLIP score, the higher the correlation between the generated images and the generated texts, and the higher the quality of the images generated by the model.

[0237] It can be seen that the FID score of the multi-modal large language model provided by the embodiment of the application is the lowest, and the CLIP score is the highest, so the image quality generated by the multi-modal large language model provided by the embodiment of the application is the highest, and the performance of the multi-modal large language model provided by the embodiment of the application is the best.

[0238] In the second evaluation method, the multi-modal large language model provided by the embodiment of the application and the image-text generation model provided by the related art are subjected to multiple comparative tests, each comparative test can generate test images and test texts, and then the performance of the two models is determined based on the results of each comparative test. Among them, in each comparative test, the input content of the multi-modal large language model and the image-text generation model is consistent, so as to ensure the reliability of the comparison result.

[0239] Then, in each comparative test, the style consistency score of the two models needs to be calculated, the style consistency score is used to represent the style similarity of the generated content and the input content, the higher the style consistency score, the higher the style similarity of the generated content and the input content, and vice versa, the lower the style consistency score, the lower the style similarity of the generated content and the input content, on this basis, in all comparative tests, the probability that the style consistency score of the multi-modal large language model is greater than the style consistency score of the image-text generation model is calculated to obtain the winning probability; the probability that the style consistency score of the multi-modal large language model is equal to the style consistency score of the image-text generation model is calculated to obtain the tie probability; and the probability that the style consistency score of the multi-modal large language model is less than the style consistency score of the image-text generation model is calculated to obtain the failure probability.

[0240] Then, in each comparative test, the narrative attraction of the generated content of the two models can be determined, for example, the narrative attraction can be determined by the voting results of the audience on the generated content, or the narrative attraction can be determined based on the comment results of the generated content, and on this basis, in all comparative tests, the probability that the narrative attraction of the multi-modal large language model is greater than the narrative attraction of the image-text generation model is obtained as the winning probability, the probability that the narrative attraction of the multi-modal large language model is equal to the narrative attraction of the image-text generation model is obtained as the tie probability, and the probability that the narrative attraction of the multi-modal large language model is less than the narrative attraction of the image-text generation model is obtained as the failure probability.

[0241] Then, in each comparative test, the narrative attraction of the generated content of the two models can be determined, for example, the narrative attraction can be determined by the voting results of the audience on the generated content, or the narrative attraction can be determined based on the comment results of the generated content, and on this basis, in all comparative tests, the probability that the narrative attraction of the multi-modal large language model is greater than the narrative attraction of the image-text generation model is obtained as the winning probability, the probability that the narrative attraction of the multi-modal large language model is equal to the narrative attraction of the image-text generation model is obtained as the tie probability, and the probability that the narrative attraction of the multi-modal large language model is less than the narrative attraction of the image-text generation model is obtained as the failure probability.

[0242] For example, referring to FIG. 11, FIG. 11 is an optional columnar schematic diagram of the FID score provided by the embodiment of the application.

[0243] It can be seen that the FID score of the multi-modal large language model provided by the embodiment of the application is low, so the image quality generated by the multi-modal large language model provided by the embodiment of the application is high.

[0244] For example, referring to FIG. 12, FIG. 12 is an optional pie-shaped schematic diagram of the style consistency comparison result provided by the embodiment of the application.

[0245] It can be seen that the probability that the style consistency score of the multi-modal large language model provided in the embodiment of the present application is greater than the style consistency score of the image text generation model is 16.1%, that is, the winning probability is 16.1%, the probability that the style consistency score of the multi-modal large language model provided in the embodiment of the present application is equal to the style consistency score of the image text generation model is 71.7%, that is, the probability of a tie is 71.7%, and the probability that the style consistency score of the multi-modal large language model provided in the embodiment of the present application is greater than the style consistency score of the image text generation model is 12.2%, that is, the failure probability is 12.2%. It can be seen that the style consistency of the multi-modal large language model provided in the embodiment of the present application is better than the style consistency of the image text generation model provided in the related art.

[0246] For example, referring to FIG. 13, FIG. 13 is an optional pie-shaped schematic diagram provided by the embodiment of the present application for describing the narration attraction comparison result.

[0247] It can be seen that the probability that the narration attraction of the multi-modal large language model provided in the embodiment of the present application is greater than the narration attraction of the image text generation model is 51.7%, that is, the winning probability is 51.7%, the probability that the narration attraction of the multi-modal large language model provided in the embodiment of the present application is equal to the narration attraction of the image text generation model is 12.8%, that is, the probability of a tie is 12.8%, and the probability that the narration attraction of the multi-modal large language model provided in the embodiment of the present application is greater than the narration attraction of the image text generation model is 35.6%, that is, the failure probability is 35.6%. It can be seen that the narration attraction of the multi-modal large language model provided in the embodiment of the present application is significantly better than the narration attraction of the image text generation model provided in the related art.

[0248] For example, referring to FIG. 14, FIG. 14 is an optional pie-shaped schematic diagram provided by the embodiment of the present application for describing the image-text consistency comparison result.

[0249] It can be seen that the probability that the image-text consistency score of the multi-modal large language model provided in the embodiment of the present application is greater than the image-text consistency score of the image text generation model is 47.8%, that is, the winning probability is 47.8%, the probability that the image-text consistency score of the multi-modal large language model provided in the embodiment of the present application is equal to the image-text consistency score of the image text generation model is 35.0%, that is, the probability of a tie is 35.0%, and the probability that the image-text consistency score of the multi-modal large language model provided in the embodiment of the present application is greater than the image-text consistency score of the image text generation model is 17.2%, that is, the failure probability is 17.2%. It can be seen that the image-text consistency of the multi-modal large language model provided in the embodiment of the present application is significantly better than the image-text consistency of the image text generation model provided in the related art.

[0250] Therefore, compared with the image text generation model provided by the related art, the FID score of the multi-modal large language model provided by the embodiments of the present application is lower, the style consistency is better, the narrative appeal is better, and the image-text consistency is better, so the performance of the multi-modal large language model provided by the embodiments of the present application is better.

[0251] In the third evaluation method, in the multi-modal large language model, the attention construction method adopted by the multi-modal data processing method provided by the embodiments of the present application and other attention construction methods are used for reasoning respectively, and evaluation indexes of different attention construction methods can be obtained, as shown in Table 2 below:

[0252] Table 2

[0253] It can be seen that, compared with the method of using global attention, the multi-modal data processing method provided by the embodiments of the present application can effectively reduce the memory usage and significantly reduce the reasoning time, thereby improving the reasoning efficiency; compared with the method of using sliding window, the multi-modal data processing method provided by the embodiments of the present application can significantly reduce the FID score and improve the CLIP score, effectively improve the performance of the multi-modal large language model, and further improve the generation quality of the multi-modal content.

[0254] For example, referring to FIG. 14, FIG. 14 is an optional pie-shaped schematic diagram of the image-text consistency comparison result provided by the embodiments of the present application.

[0255] It can be seen that the multi-modal data processing method provided by the embodiments of the present application can be applied to various scenes.

[0256] For example, in the field of education, the step image-text related to education can be expanded, which can include experimental step image-text, knowledge point image-text, etc. The reference image and the reference text of the first step are obtained, the reference visual features of the reference image are extracted, the reference text and the reference visual features with start marks and end marks added at both ends are input into the multi-modal large language model, the predicted sequence is output based on the multi-modal large language model, and then the predicted image and the predicted text of the subsequent step are generated based on the predicted sequence, so as to expand the step image-text. The predicted image and the predicted text can maintain the style consistency, the content appeal, and the image-text consistency, thereby improving the overall quality of the step image-text.

[0257] For example, in the field of entertainment, the story content can be expanded, a reference image and a reference text of a certain plot are obtained, reference visual features of the reference image are extracted, the reference text and the reference visual features with start and end markers added at both ends are input into the multi-modal large language model, a prediction sequence is output based on the multi-modal large language model, and predicted images and predicted texts of a subsequent plot are generated based on the prediction sequence, so as to expand the story content. The predicted images and the predicted texts can maintain style consistency, content appeal, and image-text consistency, thereby improving the overall quality of the story content.

[0258] It can be understood that, although each step in each of the above flowcharts is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified in this embodiment, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the above flowcharts can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0259] Referring to FIG. 15, FIG. 15 is an optional structure schematic diagram of a multi-modal data processing apparatus provided by the embodiment of the application. The multi-modal data processing apparatus 1500 comprises:

[0260] The acquisition module 1501 is configured to acquire a reference image and a reference text, wherein the reference text is used to describe the content in the reference image.

[0261] The first processing module 1502 is configured to extract reference visual features of the reference image, and input the reference text and the reference visual features with start and end markers added at both ends into the multi-modal large language model.

[0262] The second processing module 1503 is configured to determine a target embedding sequence obtained by splicing the embedding of the reference text, the embedding of the start marker, the embedding of the reference visual features, and the embedding of the end marker based on the multi-modal large language model, perform attention processing on the embedding selected by the sliding window in the start marker, the end marker and the target embedding sequence, and output a prediction sequence.

[0263] The generation module 1504 is configured to generate predicted images and predicted texts based on the prediction sequence, wherein the predicted texts are used to describe the content in the predicted images.

[0264] In some embodiments, the predicted sequence includes a predicted token and a predicted visual feature, and the second processing module 1503 is further configured to: perform attention processing based on the embedding of the start token, the embedding of the end token, and the embeddings in the target embedding sequence selected by the sliding window, to generate a predicted embedding; classify the predicted embedding belonging to the text type, to output the predicted token; and map the predicted embedding belonging to the visual type, to output the predicted visual feature.

[0265] In some embodiments, the second processing module 1503 is further configured to: splice the embedding of the start token, the embedding of the end token, and the embeddings in the target embedding sequence selected by the sliding window into a to-be-processed embedding sequence; construct a first key matrix and a first value matrix based on the to-be-processed embedding sequence; construct a first query matrix based on the embedding of the last position in the to-be-processed embedding sequence; and perform attention processing based on the first query matrix, the first key matrix, and the first value matrix, to generate a predicted embedding.

[0266] In some embodiments, the second processing module 1503 is further configured to: determine an attention weight matrix based on the first query matrix and the first key matrix, wherein the attention weight matrix includes an attention weight corresponding to each embedding in the to-be-processed embedding sequence; adjust the attention weight corresponding to the embedding of the start token or the embedding of the end token to be greater than or equal to a preset weight threshold; and generate the predicted embedding based on the product of the adjusted attention weight matrix and the first value matrix.

[0267] In some embodiments, the second processing module 1503 is further configured to: determine an attention score matrix based on the product of the transpose of the first query matrix and the first key matrix; adjust the attention score matrix based on a scaling factor; and normalize the adjusted attention score matrix to obtain an attention weight matrix.

[0268] In some embodiments, the second processing module 1503 is further configured to: construct a second key matrix and a second value matrix according to the predicted embedding belonging to the visual type; obtain a plurality of first learnable vectors pre-trained, and construct a second query matrix according to the plurality of first learnable vectors, wherein the total dimension of the plurality of first learnable vectors is the same as the dimension of the reference visual feature; and perform attention processing based on the second query matrix, the second key matrix, and the second value matrix, to obtain the predicted visual feature.

[0269] In some embodiments, the second processing module 1503 is further configured to: determine a candidate weight matrix based on the second query matrix and the second key matrix, wherein the candidate weight matrix includes an attention weight corresponding to each embedding in the predicted sequence; and generate the predicted visual feature based on the product of the candidate weight matrix and the second value matrix.

[0270] In some embodiments, the generation module 1504 is further configured to generate predicted text based on the predicted tokens, obtain a first randomly generated noise image, input the first noise image into a diffusion model, and denoise the first noise image based on the predicted visual features as diffusion conditions of the diffusion model to generate a predicted image.

[0271] In some embodiments, the second processing module 1503 is further configured to construct a third key matrix and a third value matrix based on the multi-modal large language model and the reference visual features, obtain a plurality of second learnable vectors pre-trained, and construct a third query matrix based on the plurality of second learnable vectors, wherein the second learnable vectors have the same dimension as the embedding of the reference text.

[0272] In some embodiments, the multi-modal data processing apparatus further comprises a first training module (not shown in the figure), which is configured to obtain a first sample image, a first sample text, a target image, and a target text, wherein the first sample text is used to describe the content in the first sample image, the target text is used to describe the content in the target image, and the target image is an image associated with the first sample image; extract a first visual feature of the first sample image, input the first sample text and the first visual feature with a start token and an end token added at both ends into the multi-modal large language model for inference; determine a first loss of the multi-modal large language model for inference based on the target text, and determine a second loss of the multi-modal large language model for inference based on the target image; and train the multi-modal large language model based on the first loss and the second loss.

[0273] In some embodiments, the first training module is further configured to input the first loss and the second loss after splicing into a regression model for prediction to obtain loss weights corresponding to the first loss and the second loss respectively, wherein the regression model is trained jointly with the multi-modal large language model; weight the first loss and the second loss based on the loss weights to obtain a model loss, and train the multi-modal large language model based on the model loss.

[0274] In some embodiments, the multi-modal data processing apparatus further comprises a second training module (not shown in the figure) configured to: obtain a second sample image, input the second sample image into the feature extractor, and extract a second visual feature of the second sample image; obtain a randomly generated second noise image, input the second noise image into the diffusion model, perform denoising processing on the second noise image by taking the second visual feature as a diffusion condition of the diffusion model, and generate a reconstructed image; determine a third loss based on a difference between the reconstructed image and the second sample image, and perform joint training on the feature extractor and the diffusion model based on the third loss.

[0275] In some embodiments, the second training module is further configured to: freeze the parameters of the diffusion model, extract a fourth visual feature of the reconstructed image, determine a fifth loss based on the second visual feature and the fourth visual feature, update the parameters of the feature extractor based on the fifth loss to obtain a trained feature extractor; freeze the parameters of the trained feature extractor, update the parameters of the diffusion model based on the third loss to obtain a trained diffusion model; and fine-tune the trained feature extractor and the trained diffusion model based on the third loss.

[0276] In some embodiments, the multi-modal data processing apparatus further comprises a third training module (not shown in the figure) configured to: obtain a third sample image, a second sample text, and a label image, wherein the second sample text is used to describe the content in the third sample image, and the label image is an image associated with the third sample image; extract a third visual feature of the third sample image, input the second sample text and the third visual feature with start and end markers added at both ends into the trained multi-modal large language model for inference; obtain a randomly generated third noise image, input the third noise image into the trained diffusion model, perform denoising processing on the third noise image by taking the visual feature output by the trained multi-modal large language model as a diffusion condition of the diffusion model, and generate an associated image; determine a fourth loss according to the associated image and the label image, and perform training on the diffusion model again based on the fourth loss.

[0277] In some embodiments, the first training module is further configured to: obtain a plurality of continuous video frames and corresponding subtitle texts of each video frame; call a text description model to perform text generation processing on each video frame respectively to generate description texts corresponding to each video frame; construct a prompt instruction for prompting content enhancement with reference to the description texts, splice the prompt instruction, each subtitle text, and the corresponding description text to input a target large language model, and perform content enhancement on each subtitle text based on the target large language model to obtain enhanced texts corresponding to each video frame; determine the last video frame as a target image, the remaining video frames as first sample images, the enhanced text corresponding to the target image as a target text, and the enhanced texts corresponding to the first sample images as first sample texts.

[0278] The multi-modal data processing apparatus 1500 and the multi-modal data processing method are based on the same inventive concept. By obtaining a reference image and a reference text for describing the content in the reference image, extracting a reference visual feature of the reference image, adding a start mark and an end mark at both ends of the reference visual feature, the reference visual feature and the reference text can be distinguished based on the start mark and the end mark subsequently. On this basis, the reference text and the reference visual feature with the start mark and the end mark added at both ends are input into a multi-modal large language model, a prediction sequence is output based on the multi-modal large language model, and a prediction image and a prediction text for describing the prediction image are generated based on the prediction sequence. Since attention processing is performed based on the embedding of the start mark, the embedding of the end mark, and the embedding selected by the sliding window in the target embedding sequence during inference, even if multi-modal data processing is performed for multiple rounds in succession, the multi-modal large language model can focus on the information carried by the embedding of the start mark, the embedding of the end mark, and the embedding selected by the sliding window in the target embedding sequence through attention processing, thereby reducing the perplexity during inference of the multi-modal large language model, effectively improving the performance of the multi-modal large language model, and further improving the generation quality of multi-modal content.

[0279] The electronic device for performing the multi-modal data processing method provided in the embodiments of the present application can be a terminal. Referring to FIG. 16, FIG. 16 is a partial structural block diagram of a terminal provided in the embodiments of the present application, which includes a camera assembly 1610, a first memory 1620, an input unit 1630, a display unit 1640, a sensor 1650, an audio circuit 1660, a wireless fidelity (WiFi) module 1670, a first processor 1680, and a first power supply 1690, and the like. Those skilled in the art can understand that the structure of the terminal shown in FIG. 16 does not constitute a limitation on the terminal, and can include more or fewer components than those shown, or combine certain components, or different component arrangements.

[0280] The camera assembly 1610 can be used to capture images or videos. Optionally, the camera assembly 1610 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is disposed on the front panel of the terminal, and the rear-facing camera is disposed on the back of the terminal. In some embodiments, the rear-facing camera is at least two, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function of the main camera and the depth-of-field camera, the panorama shooting and VR (Virtual Reality) shooting function of the main camera and the wide-angle camera, or other fusion shooting functions.

[0281] The first memory 1620 can be used to store software programs and modules, and the first processor 1680 executes various functional applications and data processing of the terminal by running the software programs and modules stored in the first memory 1620.

[0282] The input unit 1630 can be used to receive input digital or character information, and generate key signal input related to the setting and function control of the terminal. For example, the input unit 1630 can include a touch panel 1631 and other input devices 1632.

[0283] The display unit 1640 can be used to display input information or provided information and various menus of the terminal. The display unit 1640 can include a display panel 1641.

[0284] The audio circuit 1660, the speaker 1661, and the microphone 1662 can provide an audio interface.

[0285] The first power supply 1690 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery.

[0286] The number of sensors 1650 can be one or more, which includes but is not limited to an acceleration sensor, a gyroscope sensor, a pressure sensor, an optical sensor, and the like. Among them:

[0287] The acceleration sensor can detect the acceleration size in three coordinate axes of the coordinate system established by the terminal. For example, the acceleration sensor can be used to detect the components of the gravitational acceleration in three coordinate axes. The first processor 1680 can control the display unit 1640 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor. The acceleration sensor can also be used for game or user motion data collection.

[0288] The gyroscope sensor can detect the body direction and rotation angle of the terminal, and the gyroscope sensor can collect 3D actions of the user on the terminal in cooperation with the acceleration sensor. The first processor 1680 can realize the following functions according to the data collected by the gyroscope sensor: motion sensing (such as changing the UI according to the tilt operation of the user), image stabilization when shooting, game control, and inertial navigation.

[0289] The pressure sensor can be arranged in at least one of the side frame of the terminal and the lower layer of the display unit 1640. When the pressure sensor is arranged in the side frame of the terminal, the holding signal of the user on the terminal can be detected, and the left-hand or right-hand recognition or shortcut operation is performed by the first processor 1680 according to the holding signal collected by the pressure sensor. When the pressure sensor is arranged in the lower layer of the display unit 1640, the first processor 1680 controls the operable control on the UI interface according to the pressure operation of the user on the display unit 1640. The operable control includes at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0290] The optical sensor is used to collect the ambient light intensity. In an embodiment, the first processor 1680 can control the display brightness of the display unit 1640 according to the ambient light intensity collected by the optical sensor. For example, when the ambient light intensity is high, the display brightness of the display unit 1640 is increased; when the ambient light intensity is low, the display brightness of the display unit 1640 is decreased. In another embodiment, the first processor 1680 can also dynamically adjust the shooting parameters of the camera assembly 1610 according to the ambient light intensity collected by the optical sensor.

[0291] In the embodiment, the first processor 1680 included in the terminal can execute the multi-modal data processing method of the foregoing embodiment.

[0292] The electronic device provided by the embodiment of the present application for executing the multi-modal data processing method described above can also be a server. Referring to FIG. 17, FIG. 17 is a partial structural block diagram of a server provided by the embodiment of the present application. The server can have a large difference due to different configurations or performances, and can include one or more than one second processor 1710 and a second memory 1730, and one or more than one storage medium 1740 (for example, one or more than one mass storage device) storing an application program 1743 or data 1742. Among them, the second memory 1730 and the storage medium 1740 can be temporary storage or persistent storage. The program stored in the storage medium 1740 can include one or more than one module (not shown in the figure), and each module can include a series of instruction operations in the server. In some embodiments, the second processor 1710 can be arranged to communicate with the storage medium 1740, and execute a series of instruction operations in the storage medium 1740 on the server.

[0293] The server can also include at least one of one or more second power supplies 1720, one or more wired or wireless network interfaces 1750, one or more input / output interfaces 1760, one or more operating systems 1741, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0294] The second processor 1710 in the server can be configured to execute the multi-modal data processing method.

[0295] The embodiments of the present application also provide a computer readable storage medium, which is configured to store a computer program. The computer program is configured to execute the multi-modal data processing method of the above-mentioned embodiments.

[0296] The embodiments of the present application also provide a computer program product, which includes a computer program stored in a computer readable storage medium. A processor of a computer device reads the computer program from the computer readable storage medium, and executes the computer program, so that the computer device executes the multi-modal data processing method.

[0297] The terms "first", "second", "third", "fourth" etc. (if any) in the specification and above drawings of the present application are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units need not be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0298] It should be understood that in the embodiments of the present application, "at least one" refers to one or more, "multiple" refers to two or more. "At least one of the two", used to describe the association relationship of the associated objects, means that there can be three relationships, for example, "at least one of A and B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. "At least one of the following" or the like refers to any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b or c, can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0299] It should be understood that in the description of the embodiments of the present application, the meaning of multiple (or multiple) is more than two, greater than, less than, more than, etc. It is understood that the number is not included, and the above, below, etc. It is understood to include the number.

[0300] In several embodiments provided by the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division, and actual implementation can have another division manner. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0301] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0302] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0303] If the integrated unit is implemented in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the embodiments of the method of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0304] It should also be appreciated that the various embodiments provided by the present application can be combined arbitrarily to achieve different technical effects.

[0305] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.< / bos> < / eoi> < / boi> < / eoi> < / boi>

Claims

1. A multi-modal data processing method applied to an electronic device, comprising: obtaining a reference image and a reference text, wherein the reference text is used to describe the content in the reference image; extracting a reference visual feature of the reference image; determining, by a multi-modal large language model, an embedding of the reference text, an embedding of a start token of the reference visual feature, an embedding of the reference visual feature, and an embedding of an end token of the reference visual feature; based on the multi-modal large language model, concatenating the embedding of the reference text, the embedding of the start token, the embedding of the reference visual feature, and the embedding of the end token into a target embedding sequence, performing attention processing on the embedding of the start token, the embedding of the end token, and the embedding selected by a sliding window in the target embedding sequence to output a prediction sequence; generating a prediction image and a prediction text based on the prediction sequence, wherein the prediction text is used to describe the content in the prediction image.

2. The multi-modal data processing method of claim 1, wherein, The prediction sequence includes a prediction token and a prediction visual feature, and the attention processing on the embedding of the start token, the embedding of the end token, and the embedding selected by the sliding window in the target embedding sequence to output the prediction sequence includes: performing attention processing on the embedding of the start token, the embedding of the end token, and the embedding selected by the sliding window in the target embedding sequence to generate a prediction embedding; classifying the prediction embedding belonging to a text type to output the prediction token; mapping the prediction embedding belonging to a visual type to output the prediction visual feature.

3. The multi-modal data processing method of claim 2, wherein, The attention processing on the embedding of the start token, the embedding of the end token, and the embedding selected by the sliding window in the target embedding sequence to generate a prediction embedding includes: concatenating the embedding of the start token, the embedding of the end token, and the embedding selected by the sliding window in the target embedding sequence into a to-be-processed embedding sequence; constructing a first key matrix and a first value matrix based on the to-be-processed embedding sequence; constructing a first query matrix based on the embedding of the last position in the to-be-processed embedding sequence; performing attention processing based on the first query matrix, the first key matrix, and the first value matrix to generate a prediction embedding.

4. The multi-modal data processing method of claim 3, wherein, The attention processing based on the first query matrix, the first key matrix, and the first value matrix to generate a prediction embedding includes: determining an attention weight matrix based on the first query matrix and the first key matrix, wherein the attention weight matrix includes an attention weight corresponding to each embedding in the to-be-processed embedding sequence; adjusting the attention weight corresponding to the embedding of the start token or the embedding of the end token to be greater than or equal to a preset weight threshold; generating a prediction embedding based on the product of the adjusted attention weight matrix and the first value matrix.

5. The multi-modal data processing method of claim 4, wherein, The determination of the attention weight matrix based on the first query matrix and the first key matrix includes: determining an attention score matrix based on the product of the transpose of the first query matrix and the first key matrix; The attention score matrix is adjusted based on a scaling factor, and the adjusted attention score matrix is normalized to obtain an attention weight matrix.

6. The multi-modal data processing method of claim 2, wherein, The mapping of the prediction embedding belonging to the visual type obtains a prediction visual feature, including: A second key matrix and a second value matrix are constructed according to the prediction embedding belonging to the visual type; A plurality of first learnable vectors are obtained, and a second query matrix is constructed according to the plurality of first learnable vectors, wherein the total dimension of the plurality of first learnable vectors is the same as the dimension of the reference visual feature; Attention processing is performed based on the second query matrix, the second key matrix and the second value matrix to obtain the prediction visual feature.

7. The multi-modal data processing method of claim 6, wherein the attention processing based on the second query matrix, the second key matrix and the second value matrix to obtain the prediction visual feature includes: A candidate weight matrix is determined based on the second query matrix and the second key matrix, wherein the candidate weight matrix includes an attention weight corresponding to each embedding in the prediction sequence; The prediction visual feature is generated based on the product of the candidate weight matrix and the second value matrix.

8. The multi-modal data processing method of claim 2, wherein, The prediction image and the prediction text are generated based on the prediction sequence, including: The prediction text is generated based on the prediction token; A first noise image is randomly generated, the prediction visual feature is taken as a diffusion condition of a diffusion model, and the first noise image is denoised based on the diffusion model to generate a prediction image.

9. The multi-modal data processing method of any one of claims 1 to 8, wherein, The target embedding sequence is determined based on the embedding of the reference text, the embedding of the start token, the embedding of the reference visual feature and the embedding of the end token by the multi-modal large language model, including: A third key matrix and a third value matrix are constructed according to the reference visual feature based on the multi-modal large language model; A plurality of second learnable vectors are obtained, and a third query matrix is constructed according to the plurality of second learnable vectors, wherein the dimension of the second learnable vector is the same as the dimension of the embedding of the reference text; The embedding of the reference visual feature is obtained based on attention processing of the third query matrix, the third key matrix and the third value matrix; The target embedding sequence is obtained by splicing the embedding of the reference text, the embedding of the start token, the embedding of the reference visual feature and the embedding of the end token.

10. The multi-modal data processing method of any one of claims 1 to 9, wherein, Before the reference text and the reference visual feature with the start token and the end token added at both ends are input into the multi-modal large language model, the multi-modal data processing method further includes: A first sample image, a first sample text, a target image and a target text are obtained, wherein the first sample text is used to describe the content in the first sample image, the target text is used to describe the content in the target image, and the target image is an image associated with the first sample image; extracting a first visual feature of the first sample image, inputting the first sample text and the first visual feature with the start mark and the end mark added at both ends into the multi-modal large language model for reasoning; determining a first loss of the multi-modal large language model for reasoning based on the target text and a second loss of the multi-modal large language model for reasoning based on the target image; training the multi-modal large language model based on the first loss and the second loss.

11. The multi-modal data processing method of claim 10, wherein, The training of the multi-modal large language model based on the first loss and the second loss comprises: concatenating the first loss and the second loss and inputting them into a regression model for prediction to obtain loss weights corresponding to the first loss and the second loss respectively, wherein the regression model is jointly trained with the multi-modal large language model; weighting and summing the first loss and the second loss based on the loss weights to obtain a model loss, and training the multi-modal large language model based on the model loss.

12. The multi-modal data processing method of claim 10, wherein, The first visual feature is extracted by a feature extractor, and before the first visual feature of the first sample image is extracted, the multi-modal data processing method further comprises: obtaining a second sample image and extracting a second visual feature of the second sample image by the feature extractor; obtaining a randomly generated second noise image, taking the second visual feature as a diffusion condition of the diffusion model, and denoising the second noise image by the diffusion model to generate a reconstructed image; determining a third loss based on the difference between the reconstructed image and the second sample image, and jointly training the feature extractor and the diffusion model based on the third loss.

13. The multi-modal data processing method of claim 12, wherein, The joint training of the feature extractor and the diffusion model based on the third loss comprises: freezing the parameters of the diffusion model, extracting a fourth visual feature of the reconstructed image, determining a fifth loss based on the second visual feature and the fourth visual feature, updating the parameters of the feature extractor based on the fifth loss to obtain the trained feature extractor; freezing the parameters of the trained feature extractor, updating the parameters of the diffusion model based on the third loss to obtain the trained diffusion model; fine-tuning the trained feature extractor and the trained diffusion model based on the third loss.

14. The multi-modal data processing method of claim 12, wherein, After the training of the multi-modal large language model based on the first loss and the second loss, the multi-modal data processing method further comprises: obtaining a third sample image, a second sample text and a label image, wherein the second sample text is used to describe the content in the third sample image, and the label image is an image associated with the third sample image; extracting a third visual feature of the third sample image, inputting the second sample text and the third visual feature with the start mark and the end mark added at both ends into the trained multi-modal large language model for reasoning; obtaining a third noise image generated randomly, taking the visual feature output by the trained multi-modal large language model as a diffusion condition of the diffusion model, and performing denoising processing on the third noise image through the diffusion model to generate an associated image; determining a fourth loss according to the associated image and the label image, and training the diffusion model again based on the fourth loss.

15. The multi-modal data processing method of claim 10, wherein, obtaining a first sample image, a first sample text, a target image and a target text, comprising: obtaining a plurality of continuous video frames and caption texts corresponding to each video frame; calling a text description model to perform text generation processing on each video frame to generate description texts corresponding to each video frame; constructing a prompt instruction for prompting content enhancement with reference to the description texts, concatenating the prompt instruction, each caption text and the corresponding description text, and inputting the concatenated texts into a target large language model to perform content enhancement on each caption text based on the target large language model to obtain enhanced texts corresponding to each video frame; determining the last video frame as the target image, the remaining video frames as the first sample images, the enhanced text corresponding to the target image as the target text, and the enhanced texts corresponding to the first sample images as the first sample texts.

16. A multi-modal data processing apparatus, comprising: an obtaining module configured to obtain a reference image and a reference text, wherein the reference text is used to describe the content in the reference image; a first processing module configured to extract a reference visual feature of the reference image, and determine, through a multi-modal large language model, an embedding of the reference text, an embedding of a start token of the reference visual feature, an embedding of the reference visual feature, and an embedding of an end token of the reference visual feature; a second processing module configured to concatenate, based on the multi-modal large language model, the embedding of the reference text, the embedding of the start token, the embedding of the reference visual feature, and the embedding of the end token into a target embedding sequence, perform attention processing based on the embedding of the start token, the embedding of the end token, and the embeddings selected by a sliding window in the target embedding sequence, and output a prediction sequence; a generating module configured to generate a predicted image and a predicted text based on the prediction sequence, wherein the predicted text is used to describe the content in the predicted image.

17. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the multi-modal data processing method of any one of claims 1 to 15 when executing the computer program.

18. A computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program implements the multi-modal data processing method of any one of claims 1 to 15 when executed by a processor.

19. A computer program product, comprising a computer program, and the computer program implements the multi-modal data processing method of any one of claims 1 to 15 when executed by a processor.

Images