A general multimodal fusion method and device based on large language embedding

Abstract

This invention provides a general multimodal fusion method and apparatus based on large language embeddings. The method includes acquiring paired text modal data and at least one non-text modal data; segmenting the text and mapping it to the word embedding space of a pre-trained large language embedding model to obtain a text embedding token sequence; inputting the non-text modal data into a visual Transformer encoder for encoding to obtain a pseudo-language token sequence aligned with the word embedding space and containing modal tokens; concatenating the pseudo-language token sequence with the text embedding token sequence and inserting a separator token to form a unified input sequence, which is then input into the pre-trained large language embedding model to obtain a multimodal fused embedding representation; constructing a cross-modal contrastive learning loss to update the visual Transformer encoder parameters while keeping the parameters of the pre-trained large language embedding model frozen. The technical solution of this invention achieves the alignment and fusion of multimodal embeddings, improving cross-modal retrieval and task generalization capabilities.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a general multimodal fusion method and apparatus based on large language embedding. Background Technology

[0002] With the multimedia development of internet content and enterprise data, non-textual information such as images, audio, and video, along with textual information, constitute massive amounts of heterogeneous data. How to enable machines to understand and integrate this information from different sources and in different forms has become one of the core challenges in the field of artificial intelligence. Multimodal fusion technology is key to promoting advanced applications such as cross-modal retrieval, human-computer interaction, content generation, and autonomous driving.

[0003] Among existing multimodal representation learning schemes, the dual-tower encoding framework based on contrastive learning is widely used: it encodes textual and non-textual modal data separately, achieving cross-modal alignment by narrowing the representation distance of paired samples and widening the representation distance of unpaired samples. This type of scheme has been applied in image and text retrieval scenarios and has been further extended to audio, video, and other modalities to meet more complex retrieval needs.

[0004] However, with the expansion of modality types and application scenarios, existing technologies still have the following shortcomings: First, multimodal alignment often depends on the data representation form and training data distribution of a specific modality. In cross-domain, long-tail semantic, or complex combined query scenarios, the consistency and semantic separability of the representation space are insufficient, which can easily lead to unstable similarity ranking and increased deviation in retrieval results. Second, the statistical characteristics of data from different modalities differ significantly. Existing solutions usually require configuring their own encoding structures and adaptation strategies for different modalities. The model architecture becomes more complex with the expansion of modalities, increasing training and deployment costs and weakening universality. Third, multimodal training highly depends on large-scale, high-quality paired data. However, the acquisition and cleaning costs of high-quality paired data for audio / video and text are high. Problems such as noisy pairing and weakly correlated pairing can reduce the alignment effect of contrastive learning, thereby affecting retrieval accuracy and generalization ability. Fourth, in the process of constructing multimodal fusion representations, existing methods often struggle to simultaneously consider semantic expressiveness, cross-modal consistency, and training stability, which can easily lead to slow training convergence, sensitivity to negative samples, and unstable transfer effects on downstream tasks.

[0005] Therefore, there is an urgent need for a general multimodal fusion method to improve semantic consistency and cross-modal alignment quality, reduce model complexity and data dependence caused by modality expansion, and improve the accuracy and generalization stability of cross-modal retrieval based on semantic similarity. Summary of the Invention

[0006] This invention provides a general multimodal fusion method based on large language embedding. The method constructs a visual Transformer encoder to generate pseudo-language token sequences aligned with the word embedding space of a pre-trained large language embedding model. These pseudo-language token sequences are concatenated with text embedding token sequences and input into the pre-trained large language embedding model to output a multimodal fused embedding representation. Simultaneously, while keeping the parameters of the pre-trained large language embedding model frozen, the parameters of the visual Transformer encoder are updated based on cross-modal contrastive learning loss. This achieves unified representation fusion of paired text modal data and at least one non-text modal data, addressing the technical problems of existing multimodal fusion embedding models in terms of insufficient representation space consistency, model architecture increasing complexity with modal expansion, and strong dependence on training data in terms of multimodal semantic alignment and general expansion.

[0007] A first aspect of the present invention provides a general multimodal fusion method based on large language embedding, the method comprising: Acquire paired text modal data with at least one non-text modal data; The text modal data is segmented to obtain a text token sequence; the text token sequence is mapped to the word embedding space of a pre-trained large language embedding model to obtain a text embedding token sequence; the non-text modal data is encoded by a pre-trained visual Transformer encoder to obtain a pseudo-language token sequence aligned with the word embedding space, and the pseudo-language token sequence contains modal tokens. The pseudo-language token sequence and the text embedding token sequence are concatenated in a preset order, and a separator token is inserted to generate a unified input sequence; the unified input sequence is input into the pre-trained large language embedding model, and the output vector corresponding to the separator token is extracted as a multimodal fusion embedding representation; Based on the multimodal fusion embedding representation, a cross-modal contrastive learning loss function is constructed. While keeping the parameters of the pre-trained large language embedding model unchanged, the parameters of the visual Transformer encoder are updated according to the loss function.

[0008] Using the above scheme, the present invention provides a general multimodal fusion method based on large language embedding. This method obtains a text embedding token sequence by segmenting paired text modal data and mapping it to the word embedding space of a pre-trained large language embedding model, thus achieving a unified alignment benchmark using the language semantic space. It then encodes at least one non-text modal data into a visual Transformer encoder to generate a pseudo-language token sequence aligned with the word embedding space and containing modal tokens, thereby mapping non-text modal semantics into a vector representation that can be measured in the same space as text. Further, it concatenates the pseudo-language token sequence with the text embedding token sequence, inserts a separator token, and inputs it into the pre-trained large language embedding model to obtain a multimodal fusion embedding representation, achieving a fusion expression of cross-modal features under unified sequence modeling. Finally, it constructs a cross-modal contrastive learning loss and updates the visual Transformer encoder parameters while keeping the parameters of the pre-trained large language embedding model frozen, achieving cross-modal alignment optimization with semantic similarity as the target, significantly improving the matching accuracy and generalization stability of the multimodal fusion embedding model in cross-modal similarity retrieval scenarios.

[0009] In some embodiments of the present invention, encoding the non-text modal data using a pre-trained visual Transformer encoder includes: converting the non-text modal data into a two-dimensional matrix representation or a frame sequence representation; dividing the two-dimensional matrix representation or frame sequence representation into blocks and performing embedding mapping to generate a block embedding token sequence; and inputting the block embedding token sequence into the visual Transformer encoder for encoding.

[0010] In some embodiments of the present invention, the two-dimensional matrix represents a time-spectrum diagram including audio; the frame sequence represents a multi-frame image sequence obtained by sampling video frames, and the block tokens corresponding to each frame are superimposed with time position encoding.

[0011] In some embodiments of the present invention, the segmentation includes dividing the two-dimensional matrix representation or the frame sequence representation into multiple blocks; the embedding mapping includes performing a linear projection on each block to generate the block embedding token.

[0012] In some embodiments of the present invention, modal-specific tokens are inserted when generating the block token sequence; the pseudo-language token sequence contains modal tokens obtained by encoding the modal-specific tokens by the visual Transformer encoder.

[0013] In some embodiments of the present invention, the pseudo-language token sequence, in addition to the modal tokens, also includes a preset number of key block feature tokens selected from the non-text modal feature sequence output by the visual Transformer encoder.

[0014] In some embodiments of the present invention, before performing the concatenation, modal type embedding is superimposed on the pseudo-language token sequence, and position codes are assigned to the pseudo-language token sequence and the text embedding token sequence respectively; the concatenation arranges the pseudo-language token sequence, the separator token, and the text token sequence in a preset concatenation order to generate the unified input sequence.

[0015] In some embodiments of the present invention, obtaining paired text modal data and at least one non-text modal data includes: calling a large language model based on a preset prompt template to generate a set of text descriptions for retrieval training; collecting corresponding non-text modal data based on the set of text descriptions; performing quality screening and semantic consistency screening on the collected data to generate paired text-non-text training sample pairs.

[0016] In some embodiments of the present invention, the training sample pairs further include the construction of difficult negative samples. The construction of difficult negative samples includes: generating different editing instructions for the same original sample, and performing generation or editing on the text description and / or non-text modal content based on the editing instructions to obtain multiple samples that are from the same source but have different semantics; constructing positive sample pairs with sample pairs that have consistent semantics among the sample pairs from the same source, and using sample pairs that have inconsistent semantics among the sample pairs from the same source as negative samples, to generate triple training samples consisting of queries, positive samples, and negative samples.

[0017] In some embodiments of the present invention, the training sample pair further includes candidate document set enhancement and diversified query generation, which includes: generating subtopic or subcontent descriptions, expanding the candidate document set based on the subtopic or subcontent descriptions, generating diversified query text based on the candidate document set, and establishing query-candidate document pairs to form a training sample set.

[0018] In some embodiments of the present invention, the cross-modal contrastive learning loss function includes a bidirectional contrastive term, which at least includes: constructing a text-side input sequence based on the text embedding token sequence and the delimiter token, and inputting the text-side input sequence into the pre-trained large language embedding model, extracting the output vector corresponding to the delimiter token as a text embedding vector; constructing a non-text-side input sequence based on the pseudo-language token sequence and the delimiter token, and inputting the non-text-side input sequence into the pre-trained large language embedding model, extracting the output vector corresponding to the delimiter token as a non-text embedding vector; constructing a positive sample pair by combining the text embedding vector and the non-text embedding vector of the same sample, and constructing a negative sample pair based on the embedding vectors corresponding to other samples in the same batch besides the positive sample pair.

[0019] Specifically, the cross-modal contrastive learning loss function includes a bidirectional contrastive loss, which in turn includes a modality-to-text loss function. Text-to-modal loss function ; in, , This represents the embedding vector of the i-th non-text modal data. Let K represent the embedding vector of the j-th text data, where K is the batch size. This refers to temperature hyperparameters. based on Obtain the cross-modal contrastive learning loss function value.

[0020] In some embodiments of the present invention, updating the parameters of the visual Transformer encoder includes updating the low-rank adaptation parameters inserted in the attention projection layer of the visual Transformer encoder.

[0021] Compared with existing technologies, the advantages of this invention are as follows: By converting non-textual modal data into a two-dimensional matrix representation or frame sequence representation and generating a block token sequence, which is then input into a visual Transformer encoder, this invention enables the encodeable processing of heterogeneous modalities such as audio, video, and images under a unified tokenization interface, reducing architectural differences during multimodal expansion; by superimposing video block tokens with temporal position encoding and performing linear projection on the blocks to obtain block embedding tokens, it achieves the joint expression of temporal information and local structural information, enhancing the consistency of cross-modal semantic representation; by inserting modality-specific tokens into the block token sequence to form a pseudo-language token sequence containing modality tokens, and further combining a preset number of key block feature tokens, this invention... This approach achieves collaborative alignment of global semantics and fine-grained cues, enhancing the discriminative power of similarity retrieval. Furthermore, by superimposing modality type embeddings onto pseudo-language token sequences and assigning positional encodings to each, it enables the identifiability and stable fusion of tokens from different modalities in unified sequence modeling. By constructing a bidirectional cross-modal contrastive learning loss that includes in-batch negative samples and updating the visual Transformer encoder while freezing the parameters of the pre-trained large language embedding model (which may only update the low-rank adaptation parameters), it achieves cross-modal alignment optimization with semantic similarity as the goal, reduces the scale of training parameters and training fluctuations, and significantly improves the matching accuracy, robustness, and generalization stability of the multimodal embedding model in cross-modal semantic similarity retrieval scenarios.

[0022] A second aspect of the present invention provides a general multimodal fusion system based on large language embedding, comprising: The data acquisition module is used to acquire paired text modal data and at least one non-text modal data; The text embedding generation module is used to segment the text modal data to obtain a text token sequence, and map the text token sequence to the word embedding space of the pre-trained large language embedding model to obtain a text embedding token sequence; The pseudo-language token generation module is used to encode the non-text modal data using a pre-trained visual Transformer encoder to obtain a pseudo-language token sequence aligned with the word embedding space, wherein the pseudo-language token sequence contains modal tokens. The fusion embedding generation module is used to concatenate the pseudo-language token sequence and the text embedding token sequence in a preset order, and add a separator token to form a unified input sequence; the unified input sequence is input into the pre-trained large language embedding model to obtain a multimodal fusion embedding representation; The training update module is used to construct a cross-modal contrastive learning loss function and update the parameters of the visual Transformer encoder according to the loss function, provided that the parameters of the pre-trained large language embedding model are not updated.

[0023] A third aspect of the present invention provides a general multimodal fusion device based on large language embedding, characterized in that the device includes a computer device, the computer device includes a processor and a memory, the processor stores computer instructions, and when the computer instructions are executed, the device implements the general multimodal fusion method based on large language embedding.

[0024] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the text, or may be learned by practice of the invention. The objects and other advantages of the invention will become apparent from the description and the accompanying drawings.

[0025] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0026] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0027] In the attached diagram: Figure 1 This is a flowchart illustrating a general multimodal fusion method based on large language embedding, provided for an embodiment of the present invention.

[0028] Figure 2This is a schematic diagram of a general multimodal fusion system based on large language embedding, provided as an embodiment of the present invention.

[0029] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0030] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0031] The following detailed description is exemplary and intended to provide further detailed explanation of the invention. Unless otherwise specified, all technical terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention.

[0032] Figure 1 This is a flowchart illustrating a general multimodal fusion method based on large language embedding provided in an embodiment of the present invention.

[0033] Example 1, as Figure 1 As shown, this invention provides a general multimodal fusion method based on large language embedding, the method comprising the following steps: Acquire paired text modal data with at least one non-text modal data; The text modal data is segmented to obtain a text token sequence; the text token sequence is mapped to the word embedding space of a pre-trained large language embedding model to obtain a text embedding token sequence; the non-text modal data is encoded by a pre-trained visual Transformer encoder to obtain a pseudo-language token sequence aligned with the word embedding space, and the pseudo-language token sequence contains modal tokens. In this embodiment, the obtained multimodal data is first preprocessed, including: 1) Text modality: For input text data The pre-trained large language embedding model (such as BGE-M3) uses its built-in word segmenter to segment words and adds special tokens (such as [CLS], [SEP]) to form a text token sequence. .

[0034] (2) Image modality: Input image First, the image is adjusted to a fixed resolution, such as 224×224 pixels. Then, the image is segmented into non-overlapping patches, each 16×16 pixels in size. Each patch is mapped to a vector using a linear projection layer, resulting in a sequence of image patches. ,in A learnable [IMG] token is added before this sequence to aggregate global image information.

[0035] (3) Audio mode: The input audio waveform is first converted into a log-Mel spectrum. Where F is the number of Mel filter banks (e.g., 128) and T is the number of time frames. This spectrogram is treated as a single-channel "image," and is also segmented and linearly projected through a one-dimensional convolutional layer or a ViT patch embedding layer to obtain an audio block sequence. And add the [AUD] token before it.

[0036] (4) Video modality: Video V is considered as a series of frames This embodiment employs a sparse sampling strategy, such as uniform sampling. =8 frames. Each frame independently undergoes the above image processing flow to obtain a corresponding block sequence. Subsequently, learnable frame position codes are added to the spatial block sequence, and the block sequences of all frames are concatenated in the time dimension to form a video block sequence. Furthermore, a [VID] token can be added before it, and a lightweight temporal attention layer can be inserted into ViT to model inter-frame dependencies.

[0037] Image patch sequence obtained after preprocessing Audio block sequence or video block sequence The data is fed into a multi-layer visual Transformer encoder. This ViT encoder typically consists of L layers (e.g., L=24), each layer containing a multi-head self-attention mechanism and a feedforward neural network, and applying residual connections and layer normalization. After the deep nonlinear transformation by the ViT encoder, the final hidden state of the special tokens ([IMG], [AUD], [VID]) at the beginning of the sequence is considered as a "semantic summary" of the entire non-textual modality. The core innovation of this embodiment is that we consider this "semantic summary" vector and several key block feature vectors following it (e.g., selected through attention weights) together as a "pseudo-language token sequence". This sequence is aligned in dimension with the word embedding space of the language model.

[0038] The pseudo-language token sequence and the text embedding token sequence are concatenated in a preset order, and a separator token is inserted to generate a unified input sequence; the unified input sequence is input into the pre-trained large language embedding model, and the output vector corresponding to the separator token is extracted as a multimodal fusion embedding representation; In this embodiment, a pre-trained large language embedding model is used. (e.g., BGE-M3) performs feature embedding fusion, with its parameters frozen during the initial training phase. The fusion process is as follows: Single-modal coding: For plain text queries, use directly. right Encode to obtain text embedding (Usually, the normalized hidden state of the [CLS] token is taken.)

[0039] Cross-modal coding (key step): For non-textual modalities, the pseudo-language token sequence output by ViT is used. Enter directly into In the middle. Due to Originally designed for a language, in order for it to "understand" these pseudo-tokens, we need to... It is added to a learnable "modal type embedding" and given an independent positional encoding. Similar to processing ordinary text sequences, self-attention computation and context encoding are performed on this sequence. Finally, the hidden state of the first pseudo-token (corresponding to [CLS] or [IMG], etc.) in the sequence is output as the fusion embedding of this non-text modality. , , .

[0040] Multimodal joint coding: For combined inputs such as "image + text", we will use image pseudo-token sequences and text token sequence Concatenate them into a longer sequence: [CLS] + + [SEP] + + [SEP]. Input this sequence The model's internal attention mechanism naturally establishes a correlation between image pseudo-tokens and text tokens, achieving deep feature-level fusion. The final fused embedding is taken from the representation of the final layer [CLS] token.

[0041] Based on the multimodal fusion embedding representation, a cross-modal contrastive learning loss function is constructed. While keeping the parameters of the pre-trained large language embedding model unchanged, the parameters of the visual Transformer encoder are updated according to the loss function.

[0042] In this embodiment, the specific implementation method is as follows: Given a batch of N matching pairs ,in It can be an embedding of images, audio, video, or a combination of modalities. , It is the corresponding text embedding .

[0043] The loss is calculated using a symmetric cross-entropy contrastive loss function, as follows:

[0044]

[0045]

[0046] in, Cosine similarity; This is a temperature hyperparameter, usually set to a decimal (e.g., 0.02 to 0.07), used to adjust the smoothness of the probability distribution.

[0047] During model training, the above optimization methods can improve training stability, efficiency, and effectiveness, specifically: First, regarding the optimization strategy, the technical solution of this invention employs gradient pruning to prevent gradient explosion and selects the AdamW optimizer, enhancing the model's generalization ability by setting an appropriate weight decay (e.g., 0.05). The learning rate is scheduled using a cosine annealing strategy with a warm-up phase, allowing the learning rate to decrease smoothly and facilitating model convergence.

[0048] Secondly, to accelerate training and reduce memory usage, mixed precision training (such as FP16 or BF16) is adopted, which significantly improves computation and storage efficiency while ensuring that the accuracy is basically unaffected.

[0049] Finally, in the fine-tuning stage, the parameter-efficient LoRA (Low-Rank Adaptation) method is introduced. This method inserts only a small number of trainable low-rank adapter parameters into the attention projection layers of the visual Transformer encoder and the large language model, while keeping the original model parameters frozen. This approach can effectively adapt to the model's capabilities with an extremely low number of training parameters.

[0050] Using the above scheme, the present invention provides a general multimodal fusion method based on large language embedding. This method obtains a text embedding token sequence by segmenting paired text modal data and mapping it to the word embedding space of a pre-trained large language embedding model, thus achieving a unified alignment benchmark using the language semantic space. It then encodes at least one non-text modal data into a visual Transformer encoder to generate a pseudo-language token sequence aligned with the word embedding space and containing modal tokens, thereby mapping non-text modal semantics into a vector representation that can be measured in the same space as text. Further, it concatenates the pseudo-language token sequence with the text embedding token sequence, inserts a separator token, and inputs it into the pre-trained large language embedding model to obtain a multimodal fusion embedding representation, achieving a fusion expression of cross-modal features under unified sequence modeling. Finally, it constructs a cross-modal contrastive learning loss and updates the visual Transformer encoder parameters while keeping the parameters of the pre-trained large language embedding model frozen, achieving cross-modal alignment optimization with semantic similarity as the target, significantly improving the matching accuracy and generalization stability of the multimodal fusion embedding model in cross-modal similarity retrieval scenarios.

[0051] In some embodiments of the present invention, encoding the non-text modal data using a pre-trained visual Transformer encoder includes: converting the non-text modal data into a two-dimensional matrix representation or a frame sequence representation; dividing the two-dimensional matrix representation or frame sequence representation into blocks and performing embedding mapping to generate a block embedding token sequence; and inputting the block embedding token sequence into the visual Transformer encoder for encoding.

[0052] In this embodiment, the image in the image modality data is segmented into non-overlapping blocks. Each block is mapped to a vector through a linear projection layer to obtain an image block sequence, and a [IMG] token is added before the sequence. The audio waveform of the audio modality data is converted into a spectrogram, which is treated as a single-channel "image." Similarly, it is segmented and linearly projected through a one-dimensional convolutional layer or the Patch Embedding layer of ViT to obtain an audio block sequence, and a [AUD] token is added before the sequence. The video modality data is processed by extracting video frames, and all video frames are stitched together in the time dimension to form a video block sequence, and a [VID] token is added before the sequence. The resulting image block sequence is then processed. Audio block sequence or video block sequence It is fed into a multi-layer visual Transformer encoder for encoding.

[0053] In some embodiments of the present invention, the two-dimensional matrix represents a time-spectrum diagram including audio; the frame sequence represents a multi-frame image sequence obtained by sampling video frames, and the block tokens corresponding to each frame are superimposed with time position encoding.

[0054] In this embodiment, the two-dimensional matrix representation, taking audio as an example, can be represented by a log-Mel spectrogram. A time-frequency transformation is performed on the input audio waveform to obtain a spectrum matrix with the number of Mel filter bank bands F and the number of time frames T as dimensions. This spectrum matrix is ​​then used as the input for a single-channel image in the subsequent block segmentation and embedding mapping process to generate an audio block sequence.

[0055] The frame sequence representation, taking video as an example, can be represented by a multi-frame image sequence with sparse sampling. The video is regarded as a frame sequence and frame sampling is performed (e.g., 8 frames are uniformly extracted). For each frame image, the same resolution adjustment, block division and linear projection processing as the image modality are performed to obtain the corresponding block sequence. Subsequently, in order to distinguish different frames, learnable frame position codes (temporal position codes) are added to the spatial block sequence, and the frame block sequences are spliced ​​in the time dimension to form a video block sequence, thereby realizing the explicit encoding of video temporal information.

[0056] In some embodiments of the present invention, the segmentation includes dividing the two-dimensional matrix representation or the frame sequence representation into multiple blocks; the embedding mapping includes performing a linear projection on each block to generate the block embedding token.

[0057] In this embodiment, taking the image modality as an example, the input image can first be adjusted to a fixed resolution (e.g., 224×224 pixels), and then divided into multiple non-overlapping image patches (e.g., patch size 16×16 pixels). Each image patch is embedded and mapped through a linear projection layer, mapping the patch pixel features into vector representations, thereby forming an image patch sequence (i.e., a patch embedding token sequence). The above-mentioned patching and linear projection can also be applied to audio spectrum matrices or video frame sequences: the two-dimensional matrix or frame image is regarded as an image input, and a patch embedding token sequence is generated using the same patch embedding method as the image modality, and then fed into the visual Transformer encoder for depth encoding.

[0058] In some embodiments of the present invention, modal-specific tokens are inserted when generating the block token sequence; the pseudo-language token sequence contains modal tokens obtained by encoding the modal-specific tokens by the visual Transformer encoder.

[0059] In this embodiment, modality-specific tokens can be inserted when generating the block embedding token sequence to aggregate global modality information. The block embedding token sequence containing the aforementioned modality-specific tokens is input into a visual Transformer encoder. After encoding by multiple layers of self-attention and feedforward networks, the final hidden state of the modality-specific token at the beginning of the sequence can be used as the "semantic summary" vector of the non-textual modality. In the claims system, the "semantic summary" vector is the modality token in the pseudo-language token sequence (or constitutes part of the modality token).

[0060] In some embodiments of the present invention, the pseudo-language token sequence, in addition to the modal tokens, also includes a preset number of key block feature tokens selected from the non-text modal feature sequence output by the visual Transformer encoder.

[0061] In this embodiment, to obtain paired text and non-text training sample pairs, an automatic construction method based on augmentation and expansion of existing datasets can be adopted. Descriptive text related to modality content is extracted from multiple public multimodal datasets, and part-of-speech tagging and keyword frequency statistics are performed on the descriptive text to filter high-frequency core words. The core words are combined with preset prompt templates and input into a large language model to generate a diverse and high-quality multimodal text query description library. Based on this text query description library, corresponding non-text modality data such as images, audio, or video are collected from Internet open platforms, and multi-level filtering based on semantic matching degree and quality indicators (such as semantic matching degree filtering, user evaluation filtering, duration limit, tag cleaning, etc.) is performed on the collected data to form multimodal data pairs with high signal-to-noise ratio.

[0062] Optionally, after forming multimodal data pairs, text generation models can be used to further enhance and optimize the text query descriptions to improve the fluency, accuracy, and richness of the text descriptions.

[0063] In some embodiments of the present invention, before performing the concatenation, modal type embedding is superimposed on the pseudo-language token sequence, and position codes are assigned to the pseudo-language token sequence and the text embedding token sequence respectively; the concatenation arranges the pseudo-language token sequence, the separator token, and the text token sequence in a preset concatenation order to generate the unified input sequence.

[0064] In this embodiment, to construct difficult negative samples and form triple training samples, data can be actively synthesized based on cross-modal generation technology: given source multimodal content and its original description, a large language model is used to generate diverse editing instructions that can change semantic content, and a target content description corresponding to the editing instructions is generated; then, a cross-modal generation model (such as text-to-image, text-to-audio, etc. generation model) is used to generate the edited target multimodal content based on the target content description.

[0065] Furthermore, for multiple target contents originating from the same source content but generated by different editing instructions, they can be mutually designated as difficult negative samples. For example, using "source content / query + a certain editing instruction" as the query-side information, the target content generated corresponding to that editing instruction is designated as a positive sample, and the target content generated by other editing instructions under the same seed source is designated as a difficult negative sample, thereby constructing triplet training data containing multimodal queries, editing instructions, and target contents.

[0066] In some embodiments of the present invention, obtaining paired text modal data and at least one non-text modal data includes: calling a large language model based on a preset prompt template to generate a set of text descriptions for retrieval training; collecting corresponding non-text modal data based on the set of text descriptions; performing quality screening and semantic consistency screening on the collected data to generate paired text and non-text training sample pairs.

[0067] Specifically, in this embodiment, to obtain paired samples for contrastive learning training, descriptive text can be extracted from multiple public multimodal datasets. Keyword mining and frequency analysis are performed using natural language processing tools to select core words highly relevant to the modal content. Statistical analysis is then performed on text descriptions from public datasets or historical samples to extract high-frequency semantic keywords, and prompt templates are constructed accordingly. These prompt templates are input into a large-scale language model to generate a large-scale, diverse, and high-quality multimodal text query description database. Subsequently, based on the description database, raw audio, video, or image data is acquired from open media platforms. Multi-level quality and semantic consistency screening is performed on the collected data, including semantic matching degree screening, user review filtering, duration limits, and tag cleaning. Vague, repetitive, low-resolution, and mismatched samples are removed, resulting in paired text and non-text training sample pairs, forming massive, high signal-to-noise ratio multimodal data. This method can further utilize existing generative large-scale models to enhance text descriptions, improve semantic integrity and expression quality, and achieve automatic construction of training sample pairs.

[0068] In some embodiments of the present invention, the training sample pairs further include the construction of difficult negative samples. The construction of difficult negative samples includes: generating different editing instructions based on the same original sample; performing generation or editing on text descriptions and / or non-text modal content based on the editing instructions to obtain multiple samples that are from the same source but have different semantics; constructing positive sample pairs with sample pairs that have consistent semantics among the sample pairs from the same source; and using sample pairs that have inconsistent semantics among the sample pairs from the same source as negative samples to generate triple training samples consisting of queries, positive samples, and negative samples.

[0069] In this embodiment, to construct difficult negative samples, different editing instructions can be generated for the same original sample. Based on the editing instructions, cross-modal generation or editing is performed on the text description and / or non-text modal content to obtain multiple samples that are from the same source but have different semantics. A pair of samples from the same source with consistent semantics is taken as a positive sample pair, and samples from the same source with different semantics are taken as negative samples, constructing triple training data including queries, positive samples, and negative samples. Since negative samples and positive samples are from the same source and have similar appearance / semantics, this type of negative sample can be used as difficult negative samples for contrastive learning training. In some embodiments of the present invention, the training sample pair further includes candidate document set enhancement and diversified query generation. The candidate document set enhancement and diversified query generation includes: generating subtopic or subcontent descriptions, expanding the candidate document set based on the subtopic or subcontent descriptions; generating diversified query text based on the candidate document set, establishing query-candidate document pairs, and generating a training sample set.

[0070] In this embodiment, to enhance the coverage of retrieval training, subtopics or sub-content descriptions can be generated for the target domain content, and the candidate document set can be expanded based on the subtopics / sub-content descriptions. Furthermore, diverse query texts can be generated based on the candidate document set, and query-candidate document pairs can be established to obtain a retrieval sample set for training. The candidate document set can include candidates that are strongly related, weakly related, and irrelevant to the query, so as to train the model's ability to distinguish similarity ranking.

[0071] Specifically, this embodiment generates training data on demand based on cross-modal generation technology. This method may include two sub-processes: First, a hybrid multimodal query dataset is constructed. Based on given multimodal content and its description, a series of editing instructions that can change its semantics and corresponding target descriptions are generated using a large language model. Then, the target content is synthesized through a generative model. Content derived from the same content but edited differently is set as difficult negative samples to construct challenging triplet data. Then, a hybrid multimodal candidate document dataset is constructed. For a given multimodal document, a large language model is used to automatically generate its related subtopic descriptions, which are then combined with the original document to form enhanced candidate documents. Diverse queries targeting these documents are automatically generated, forming complete retrieval training data pairs. By generating data on demand, the discriminative learning is improved, effectively enhancing the flexibility and scalability of data construction.

[0072] To better illustrate the process of generating cross-modal training data in this embodiment, the following details how to automatically construct a high-quality hybrid multimodal query dataset, including the following steps: 1) Seed data preparation: Select a small subset of high-quality samples as seeds from an existing multimodal dataset containing rich descriptions (such as the MS-COCO image description dataset or the AudioCaps audio description dataset). . It is source multimodal content (such as images). This is its detailed textual description.

[0073] 2) Large language model-driven instruction and description generation includes: Prompt configuration: Design a structured prompt template to input into the large language model. For example:

[0074] Generation and parsing: Describing each seed sample Enter the prompt word template and call the large language model API. Parse the returned JSON to obtain a set of editing instructions. and corresponding new content description .

[0075] 3) Cross-modal generative model synthesis content: Image generation: Description for each new content Use a text-to-image generation model (such as Stable DiffusionXL) to generate the corresponding target image. To improve quality, negative prompts can be used, and generation parameters (such as guiding scale and number of sampling steps) can be tuned.

[0076] Audio generation: For audio-related descriptions, a text-to-audio generation model (such as AudioLDM 2) is used to generate the corresponding target audio segments. .

[0077] Quality Filtering: The generated content may vary in quality. This embodiment uses a pre-trained cross-modal matching model (e.g., CLIP for text-to-image matching, CLAP for audio-to-text matching) to filter the generated content. and its description Perform similarity scoring. Only retain high-quality samples with a similarity score higher than a set threshold (e.g., 0.25).

[0078] 4) Difficult negative sample mining and triple construction: This step is crucial for improving the model's discriminative power, especially for samples originating from the same seed sample. All generated target samples and its instructions The training samples are constructed according to the following rules: Based on source content And one of the editing commands When a combination of two elements is used as a query, at the embedding level, it can be represented as the sum or concatenation of the embeddings of the two elements.

[0079] The generated content corresponding to the above instructions As a positive example.

[0080] Content generated by other editing commands under the same seed As a difficult negative example. Because difficult negative examples share some underlying elements with the source content, but do not match the semantic target of the current query, they are difficult to distinguish.

[0081] Through the above four steps, a form of is obtained. The training tuples. This method automatically constructs massive amounts of training data rich in semantic nuances.

[0082] In some embodiments of the present invention, the cross-modal contrastive learning loss function includes a bidirectional contrastive term, which at least includes: constructing a text-side input sequence based on the text embedding token sequence and the delimiter token, and inputting the text-side input sequence into the pre-trained large language embedding model, extracting the output vector corresponding to the delimiter token as a text embedding vector; constructing a non-text-side input sequence based on the pseudo-language token sequence and the delimiter token, and inputting the non-text-side input sequence into the pre-trained large language embedding model, extracting the output vector corresponding to the delimiter token as a non-text embedding vector; constructing a positive sample pair by combining the text embedding vector and the non-text embedding vector of the same sample, and constructing a negative sample pair based on the embedding vectors corresponding to other samples in the same batch other than the positive sample pair.

[0083] Optionally, the cross-modal contrastive learning loss function includes a bidirectional contrastive loss, which includes a modality-to-text loss function. Text-to-modal loss function ; in, , This represents the embedding vector of the i-th non-text modal data. Let K represent the embedding vector of the j-th text data, where K is the batch size. This refers to temperature hyperparameters. based on Obtain the cross-modal contrastive learning loss function value.

[0084] In some embodiments of the present invention, updating the parameters of the visual Transformer encoder includes updating the low-rank adaptation parameters inserted in the attention projection layer of the visual Transformer encoder. During training, low-rank adaptation techniques are used to fine-tune the parameters of some attention layers in the visual Transformer encoder, thereby improving training efficiency and preventing catastrophic forgetting, which can accelerate training and improve model adaptability.

[0085] Compared with the prior art, the beneficial effects of the present invention are as follows: In terms of semantic fusion depth and fundamental robustness, this method achieves deep fusion at the feature level by mapping non-textual modal data to the input space of a large language model. This strategy fully leverages the rich semantic knowledge and reasoning capabilities inherent in pre-trained language models, providing a solid semantic foundation for multimodal understanding.

[0086] In terms of model versatility, thanks to the unified architecture design, this method can flexibly handle inputs in single modal or any combination thereof, such as text, images, audio, and video, and effectively support complex task scenarios such as zero-shot, few-shot, and cross-modal combined retrieval.

[0087] Regarding the efficiency of training data construction, in response to the challenge of the scarcity of high-quality multimodal data, the automated data construction scheme proposed in this method can significantly reduce the human and time costs of dataset construction, providing large-scale, high-quality data support for model training.

[0088] In terms of empirical performance, after verification by multiple standard benchmark tests (including audio-text retrieval, video-text retrieval and cross-modal classification tasks), the performance of this method is significantly better than the existing mainstream technologies, demonstrating its excellent fusion ability and robustness.

[0089] Figure 2 This is a flowchart illustrating a general multimodal fusion system based on large language embedding, provided in an embodiment of the present invention.

[0090] Example 2, as Figure 2 As shown, the present invention also provides a general multimodal fusion system based on large language embedding, including: a data acquisition module S11, a text embedding generation module S12, a pseudo-language token generation module S13, a fusion embedding generation module S14, and a training update module S15.

[0091] The data acquisition module is used to acquire paired text modal data and at least one non-text modal data; The text embedding generation module is used to segment the text modal data to obtain a text token sequence, and map the text token sequence to the word embedding space of the pre-trained large language embedding model to obtain a text embedding token sequence; The pseudo-language token generation module is used to encode the non-text modal data using a pre-trained visual Transformer encoder to obtain a pseudo-language token sequence aligned with the word embedding space, wherein the pseudo-language token sequence contains modal tokens. The fusion embedding generation module is used to concatenate the pseudo-language token sequence and the text embedding token sequence in a preset order, and add a separator token to form a unified input sequence; the unified input sequence is input into the pre-trained large language embedding model to obtain a multimodal fusion embedding representation; The training update module is used to construct a cross-modal contrastive learning loss function and update the parameters of the visual Transformer encoder according to the loss function, provided that the parameters of the pre-trained large language embedding model are not updated.

[0092] Example 3: This example describes how to deploy the UNITE model trained by the present invention into a system that can provide services.

[0093] 1) Service Architecture: Model Service Layer: Loads trained models using efficient inference service frameworks (such as Triton Inference Server, TensorFlow Serving, or the simplified FastAPI + PyTorch). This service provides two core endpoints: / encode and / search.

[0094] Vector Database Layer: Uses a specialized vector database (such as Milvus, Pinecone, Weaviate, or FAISS index) to store the embedded representation h_candidate of all candidate multimodal content and its metadata.

[0095] Application Interface Layer: Provides RESTful APIs or gRPC interfaces for front-end applications or clients to call.

[0096] 2) Online reasoning process: Encoding the query: The client submits a query (e.g., a user-uploaded image and the text "Find videos with a similar style to this scene that contain upbeat music"). The request is sent to the / encode endpoint of the model service layer, and the UNITE model of this invention encodes the query into a uniform embedding vector h_query.

[0097] Vector retrieval: The system sends h_query to the vector database layer. The vector database performs an approximate nearest neighbor search, calculates the cosine similarity between h_query and all candidate embeddings h_candidate, and returns the Top-K most similar candidate IDs.

[0098] Result return: Based on the returned candidate ID, the system retrieves the corresponding actual multimedia file address (URL) and related information from the metadata storage, packages it, and returns it to the client via API.

[0099] 3) Performance optimization: Model Quantization: Dynamic Quantization or Static Quantization is performed on the deployed UNITE model of this invention to convert FP32 parameters into INT8, which greatly reduces the model size and improves inference speed, while the accuracy loss is minimal.

[0100] Graph optimization and compilation: Use PyTorch's TorchScript or ONNX Runtime to optimize, merge, and compile the model computation graph to further improve inference efficiency.

[0101] Caching mechanism: Cache popular or repeated queries and their encoded results to reduce redundant calculations.

[0102] Batch processing: Support batch requests at the / encode endpoint, which encodes multiple queries at once, improving throughput.

[0103] Example 4: In order to verify the effectiveness of the present invention, a comparative experiment was designed below to verify the superiority of the present invention.

[0104] 1) Constructing a benchmark dataset: Image-text: Flickr30K, MS-COCO.

[0105] Audio-Text: Clotho, AudioCaps.

[0106] Video-Text: MSR-VTT, DiDeMo.

[0107] Cross-modal classification: ImageNet (images), ESC-50 (audio), Kinetics-400 (videos).

[0108] Emerging Assessment: Using the hybrid multimodal query test set built in Example 1.

[0109] 2) Constructing a baseline model: The model of this invention is compared with the following strong baseline models: Specialized models: CLIP (text and image), Wav2CLIP (audio-image), ImageBind (multimodal binding).

[0110] The latest universal model: LanguageBind (multimodal alignment anchored by language).

[0111] Traditional methods: based on handcrafted features (such as SIFT, MFCC) and early fusion methods.

[0112] 3) Evaluation indicators are: Retrieval task: Recall@1, Recall@5, Recall@10 (R@K), Mean Reciprocal Rank (MRR).

[0113] Classification task: Top-1 accuracy.

[0114] 4) Core experimental results (example): Zero-shot cross-modal retrieval: On Clotho audio-text retrieval, UNITE achieved an R@1 of 14.7%, a 2.5 percentage point improvement over LanguageBind's 12.2%. On MSR-VTT video-text retrieval, UNITE achieved an R@1 of 41.7%, outperforming InternVideo (40.7%), which requires more training data.

[0115] Combined modal retrieval: On the self-built "image + text -> audio" test set, UNITE achieved an R@1 of 32.5%, while ImageBind only achieved 21.8%, and the C-MCR model achieved 28.1%, demonstrating UNITE's significant advantage in understanding complex compound queries.

[0116] Efficiency Comparison: Thanks to LoRA and model freezing strategies, UNITE achieves state-of-the-art performance while requiring only ~5% of the number of parameters for training compared to the full fine-tuning approach, resulting in a training speed improvement of approximately 3 times.

[0117] 5) Ablation experiment analysis: Ablation of large language embedding models: Replacing it with a randomly initialized Transformer resulted in a performance drop of over 50%, demonstrating that pre-trained language knowledge is the cornerstone of effective fusion.

[0118] Ablation of ViT Segmenter: An attempt was made to replace ViT with simple global average pooling to convert the image into a single vector and then project it onto the language space. The performance dropped significantly, indicating that the fine-grained, structured pseudo-token sequences provided by ViT are crucial.

[0119] Ablation contrastive loss: By replacing the mean squared error loss with direct regression of text embeddings, the model failed to converge at all, verifying the effectiveness of contrastive learning in multimodal alignment.

[0120] Ablation of difficult negative samples: In dataset construction, random negative samples are used instead of difficult negative samples from the same source. The model's performance on fine-grained retrieval tasks decreases by about 15%.

[0121] In summary, the detailed descriptions of the above specific embodiments fully demonstrate the technological advancement, feasibility of implementation, and superior performance of the general multimodal fusion method based on large language embedding provided by this invention. Those skilled in the art can adjust and implement the method based on the above disclosure and in conjunction with specific application scenarios.

[0122] Example 3: The present invention also provides a general multimodal fusion device based on large language embedding. The device includes a computer device, which includes a processor and a memory. The processor stores computer instructions. When the computer instructions are executed, the device implements the general multimodal fusion method based on large language embedding.

[0123] Example 4, as Figure 3 As shown, the present invention also provides an electronic device 100 for implementing a general multimodal fusion method based on large language embedding.

[0124] The electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on at least one processor 102, and at least one communication bus 104.

[0125] The memory 101 can be used to store the computer program 103. The processor 102 implements the steps of the general multimodal fusion method based on large language embedding described in the first aspect of the present invention by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101.

[0126] The memory 101 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the electronic device 100 (such as audio data), etc. In addition, the memory 101 may include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0127] At least one processor 102 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 102 may be a microprocessor or any conventional processor. Processor 102 is the control center of electronic device 100, connecting various parts of electronic device 100 via various interfaces and lines.

[0128] The memory 101 in the electronic device 100 stores multiple instructions to implement a general multimodal fusion method based on large language embedding, and the processor 102 can execute multiple instructions to achieve: Acquire paired text modal data with at least one non-text modal data; The text modal data is segmented to obtain a text token sequence; the text token sequence is mapped to the word embedding space of a pre-trained large language embedding model to obtain a text embedding token sequence; the non-text modal data is encoded by a pre-trained visual Transformer encoder to obtain a pseudo-language token sequence aligned with the word embedding space, and the pseudo-language token sequence contains modal tokens. The pseudo-language token sequence and the text embedding token sequence are concatenated in a preset order, and a separator token is inserted to generate a unified input sequence; the unified input sequence is input into the pre-trained large language embedding model, and the output vector corresponding to the separator token is extracted as a multimodal fusion embedding representation; Based on the multimodal fusion embedding representation, a cross-modal contrastive learning loss function is constructed. While keeping the parameters of the pre-trained large language embedding model unchanged, the parameters of the visual Transformer encoder are updated according to the loss function.

[0129] Example 5: If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, and read-only memory (ROM).

[0130] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0132] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0133] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0134] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A general multimodal fusion method based on large language embedding, characterized in that, The method includes: Acquire paired text modal data with at least one non-text modal data; The text modal data is segmented to obtain a text token sequence; the text token sequence is mapped to the word embedding space of a pre-trained large language embedding model to obtain a text embedding token sequence; the non-text modal data is encoded by a pre-trained visual Transformer encoder to obtain a pseudo-language token sequence aligned with the word embedding space, and the pseudo-language token sequence contains modal tokens. The pseudo-language token sequence and the text embedding token sequence are concatenated in a preset order, and a separator token is inserted to generate a unified input sequence; the unified input sequence is input into the pre-trained large language embedding model, and the output vector corresponding to the separator token is extracted as a multimodal fusion embedding representation; Based on the multimodal fusion embedding representation, a cross-modal contrastive learning loss function is constructed. While keeping the parameters of the pre-trained large language embedding model unchanged, the parameters of the visual Transformer encoder are updated according to the loss function.

2. The general multimodal fusion method based on large language embedding as described in claim 1, characterized in that, The process of encoding the non-text modal data using a pre-trained visual Transformer encoder includes: The non-text modal data is converted into a two-dimensional matrix representation or a frame sequence representation; the two-dimensional matrix representation or frame sequence representation is divided into blocks and an embedding mapping is performed to generate a block embedding token sequence; the block embedding token sequence is input into the visual Transformer encoder for encoding.

3. The general multimodal fusion method based on large language embedding according to claim 2, characterized in that, The two-dimensional matrix represents a time-spectrum diagram including audio; the frame sequence represents a multi-frame image sequence obtained by sampling video frames, and the block tokens corresponding to each frame are superimposed with time position encoding.

4. The general multimodal fusion method based on large language embedding according to claim 2, characterized in that, The segmentation includes dividing the two-dimensional matrix representation or the frame sequence representation into multiple blocks; the embedding mapping includes performing a linear projection on each block to generate the block embedding token.

5. The general multimodal fusion method based on large language embedding according to claim 2, characterized in that, Modal special tokens are inserted when generating the block token sequence; the pseudo-language token sequence contains modal tokens obtained by encoding the modal special tokens by the visual Transformer encoder.

6. The general multimodal fusion method based on large language embedding according to claim 1, characterized in that, The acquisition of paired text modal data and at least one non-text modal data includes: Based on the preset prompt template, a large language model is invoked to generate a set of text descriptions for retrieval training; Collect corresponding non-text modal data based on the aforementioned text description set; The collected data is subjected to quality screening and semantic consistency screening to generate paired text-non-text training sample pairs.

7. The general multimodal fusion method based on large language embedding according to claim 6, characterized in that, The training sample pairs also include the construction of difficult negative samples, which includes: Generate different editing instructions based on the same original sample, and perform generation or editing on text descriptions and / or non-text modal content based on the editing instructions to obtain multiple samples that are from the same source but have different semantics; Positive sample pairs are constructed using semantically consistent sample pairs from the same source sample, and negative samples are generated using semantically inconsistent sample pairs from the same source sample, resulting in triplet training samples consisting of queries, positive samples, and negative samples.

8. The general multimodal fusion method based on large language embedding according to claim 1, characterized in that, The cross-modal contrastive learning loss function includes bidirectional contrastive loss, which includes a modality-to-text loss function. and text-to-modal loss function ; in, This represents the embedding vector of the i-th non-text modal data. Let K represent the embedding vector of the j-th text data, where K is the batch size. This refers to temperature hyperparameters. based on Obtain the cross-modal contrastive learning loss function value.

9. The general multimodal fusion method based on large language embedding according to claim 6, characterized in that, The training sample pairs also include candidate document set enhancement and diversified query generation, which include: Generate subtopic or subcontent descriptions, and expand the candidate document set based on the subtopic or subcontent descriptions; Based on the candidate document set, generate diverse query texts, establish query-candidate document pairs, and generate a training sample set.

10. A general multimodal fusion device based on large language embedding, characterized in that, The apparatus includes a computer device, which includes a processor and a memory. The processor stores computer instructions, and when the computer instructions are executed, the apparatus implements the general multimodal fusion method based on large language embedding as described in any one of claims 1 to 9.

Images