Method and device for semantic alignment of multi-modal data based on cross-modal attention mechanism

Abstract

The application relates to the field of multimodal semantic alignment, and provides a multimodal data semantic alignment method and device based on a cross-modal attention mechanism. The method comprises the following steps: acquiring multimodal data and corresponding category labels, and mapping the multimodal data into multimodal embedding vectors through a pre-trained multimodal encoder; generating text descriptions corresponding to the multimodal data according to a multimodal large language model, and screening the text descriptions in combination with the category labels to construct a multimodal knowledge base; based on the multimodal knowledge base, a multi-embedding center is constructed; through a cross-modal attention mechanism, the multimodal embedding vectors and the text descriptions are interacted to generate semantic-enhanced multimodal embedding vectors; and the semantic-enhanced multimodal embedding vectors and the multi-embedding center are compared and learned to realize semantic alignment of the multimodal data. In this way, the accuracy of semantic representation is enhanced, and the alignment effect of the multimodal data in a unified semantic space is remarkable.

Description

Technical Field

[0001] This application relates to the field of multimodal semantic alignment technology, and in particular to a multimodal data semantic alignment method and apparatus based on a cross-modal attention mechanism. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, joint modeling and semantic alignment of multimodal data (such as images, videos, audio, point clouds, thermal imaging, etc.) has become a research hotspot. Multimodal learning aims to construct a unified representation space through cross-modal semantic association, thereby supporting tasks such as cross-modal retrieval, zero-shot identification, and multimodal reasoning. However, due to the inherent differences in feature distribution, semantic granularity, and spatiotemporal dimensions of different modal data, traditional methods face the following challenges:

[0003] The semantic gap problem: Existing methods (such as CLIP and ImageBind) map modalities such as images and text to a shared space through contrastive learning, but they rely on a single modality (such as text or image) as the semantic center, resulting in semantic bias when aligning across modalities. For example, ImageBind aligns other modalities with images as the center, but when processing non-visual data (such as audio and event streams), it is difficult to capture complex spatiotemporal relationships due to the lack of fine-grained semantic guidance.

[0004] Noise interference and representation imbalance: While methods based on Large Language Models (LLMs) to generate multimodal descriptions (such as UniBind) can expand semantic coverage, the generated text descriptions often contain noise (such as background details) that is irrelevant to the target category. Directly using these descriptions for contrastive learning can introduce semantic shifts. Furthermore, relying solely on embedding centers generated by LLMs can easily overlook the complementarity of multimodal data, leading to an imbalance in the representation space.

[0005] Limitations of attention mechanisms: Existing cross-modal attention methods (such as BLIP-2 and InstructBLIP) mostly use fixed templates or one-way interactions, lacking the ability to dynamically adjust multimodal semantic weights. For example, in video-text alignment tasks, traditional attention mechanisms may overemphasize local frames while ignoring temporal correlations, resulting in insufficient semantic enhancement.

[0006] Therefore, there is an urgent need for a unified framework that can integrate multimodal semantic knowledge, construct a robust embedding center, and achieve accurate semantic alignment through a dynamic attention mechanism. Summary of the Invention

[0007] This application provides a method and apparatus for semantic alignment of multimodal data based on a cross-modal attention mechanism, which aims to enhance multimodal embedding by generating MLLMs and fusing input features, and then achieve representation alignment through contrastive learning.

[0008] According to a first aspect of this application, a multimodal data semantic alignment method based on a cross-modal attention mechanism is provided, comprising:

[0009] Obtain multimodal data and its corresponding category labels, and map the multimodal data into multimodal embedding vectors using a pre-trained multimodal encoder;

[0010] The multimodal large language model generates text descriptions corresponding to the multimodal data, and the text descriptions are filtered in combination with the category labels to construct a multimodal knowledge base;

[0011] Based on the aforementioned multimodal knowledge base, a multi-element embedding center is constructed, comprising a basic prompt word embedding center, a multimodal knowledge base embedding center, and a large language model embedding center;

[0012] By using a cross-modal attention mechanism, the multimodal embedding vector is interacted with the text description to generate a semantically enhanced multimodal embedding vector;

[0013] The semantically enhanced multimodal embedding vectors are compared and learned with the multi-element embedding centers to achieve semantic alignment of multimodal data;

[0014] Based on the aforementioned multimodal knowledge base, a multi-element embedding center is constructed, comprising a basic prompt word embedding center, a multimodal knowledge base embedding center, and a large language model embedding center, including:

[0015] The top N text descriptions with the highest cosine similarity to the category label are selected from the multimodal knowledge base. The mean of the top N text descriptions is aggregated by a text encoder to generate the multimodal knowledge base embedding center.

[0016] The preset static prompt template is expanded into a multimodal dynamic prompt word set by a dynamic template generator, and mapped into an embedding vector by a text encoder to generate the basic prompt word embedding center;

[0017] Based on the large language model, an abstract semantic description related to the category label is generated, and the abstract semantic description is filtered. The filtered abstract semantic description is then mapped into an embedding vector through a text encoder to generate the large language model embedding center.

[0018] The multimodal knowledge base embedding center is integrated with the basic prompt word embedding center and the large language model embedding center to form a multi-element embedding center.

[0019] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein the step of generating text descriptions corresponding to the multimodal data based on a multimodal large language model, and filtering the text descriptions in conjunction with the category labels to construct a multimodal knowledge base includes:

[0020] Generate a text description corresponding to the multimodal data based on the multimodal large language model;

[0021] The cosine similarity between the generated text description and the corresponding category label is calculated using a pre-trained text encoder.

[0022] A knowledge base for the corresponding category is constructed based on text descriptions with a cosine similarity higher than a preset threshold.

[0023] In addition to the aspects described above and any possible implementations, a further implementation is provided in which the multimodal large language model includes the InstructBLIP model and the LLaMA-adapter model.

[0024] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein the step of interacting the multimodal embedding vector with the text description through a cross-modal attention mechanism to generate a semantically enhanced multimodal embedding vector includes:

[0025] The multimodal embedding vector is used as the key vector and value vector, and the text description is used as the query vector;

[0026] The similarity between the query vector and the key vector is calculated using a multi-head attention mechanism to generate an attention weight matrix;

[0027] The attention weight matrix is ​​modified, and the modified attention weight matrix is ​​weighted and summed with the value vector to generate a semantically enhanced multimodal embedding vector.

[0028] In addition to the aspects and any possible implementations described above, a further implementation is provided in which the modification of the attention weight matrix includes:

[0029] Set the values ​​of items in the attention weight matrix that are below a preset threshold to zero;

[0030] The attention weights are dynamically adjusted based on the semantic similarity between category labels and text descriptions in the multimodal data knowledge base.

[0031] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein the step of comparing and learning the semantically enhanced multimodal embedding vector with the multivariate embedding center to achieve semantic alignment of the multimodal data includes:

[0032] The multi-element embedding centers are dynamically fused according to a dynamic weighted fusion strategy to generate a fused category representation.

[0033] The multimodal embedding vectors are semantically aligned and optimized simultaneously with the category features and each independent embedding center using a contrastive learning framework. Each independent embedding center includes a basic prompt word embedding center, a multimodal knowledge base embedding center, and a large language model embedding center.

[0034] According to a second aspect of this application, a multimodal data semantic alignment device based on a cross-modal attention mechanism is provided, comprising:

[0035] A multimodal embedding vector generation unit is used to acquire multimodal data and its corresponding category labels, and to map the multimodal data into multimodal embedding vectors through a pre-trained multimodal encoder.

[0036] A multimodal knowledge base construction unit is used to generate text descriptions corresponding to the multimodal data based on a multimodal large language model, and to filter the text descriptions in conjunction with the category tags to construct a multimodal knowledge base;

[0037] The multi-element embedding center construction unit is used to construct a multi-element embedding center based on the multimodal knowledge base, comprising a basic prompt word embedding center, a multimodal knowledge base embedding center, and a large language model embedding center, specifically including:

[0038] The top N text descriptions with the highest cosine similarity to the category label are selected from the multimodal knowledge base. The mean of the top N text descriptions is aggregated by a text encoder to generate the multimodal knowledge base embedding center.

[0039] The preset static prompt template is expanded into a multimodal dynamic prompt word set by a dynamic template generator, and mapped into an embedding vector by a text encoder to generate the basic prompt word embedding center;

[0040] Based on the large language model, an abstract semantic description related to the category label is generated, and the abstract semantic description is filtered. The filtered abstract semantic description is then mapped into an embedding vector through a text encoder to generate the large language model embedding center.

[0041] The multimodal knowledge base embedding center is integrated with the basic prompt word embedding center and the large language model embedding center to form a multi-element embedding center;

[0042] The semantic enhancement unit is used to interact the multimodal embedding vector with the text description through a cross-modal attention mechanism to generate a semantically enhanced multimodal embedding vector.

[0043] The contrastive learning unit is used to compare and learn the semantically enhanced multimodal embedding vector with the multivariate embedding center to achieve semantic alignment of multimodal data.

[0044] According to a third aspect of this application, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0045] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0046] Compared with the prior art, this application achieves the following beneficial effects:

[0047] (1) This application achieves deep interaction between multimodal embedding vectors and text descriptions through a cross-modal attention mechanism, dynamically adjusting attention weights to focus on key semantic information, effectively solving the problem of information fragmentation between modalities in traditional methods. Combined with high-quality text descriptions selected from the knowledge base, the accuracy of semantic representation is enhanced, and the alignment effect of multimodal data in a unified semantic space is significant.

[0048] (2) In constructing the multi-embedding center, this application innovatively integrates three knowledge sources: basic prompt words, knowledge base filtered text, and abstract semantics of a large language model. By combining prior knowledge with real-time generated features through a dynamic weighting strategy, it not only retains the basic concepts of the domain but also incorporates dynamic semantics of the context, significantly improving the model's adaptability to noisy data and long-tail categories, especially showing stronger generalization performance in low-resource scenarios.

[0049] (3) This application uses multimodal large language models such as InstructBLIP to generate descriptions and combines a cosine similarity threshold screening mechanism to construct a high-quality multimodal knowledge base. Compared with traditional manual annotation methods, this scheme improves the efficiency of knowledge base construction while ensuring semantic relevance, and effectively suppresses redundant information interference in the generated text through attention weight correction technology, thereby reducing computational complexity.

[0050] It should be understood that the description in the Summary Section is not intended to limit the key or essential features of the embodiments of this application, nor is it intended to restrict the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0051] The above and other features, advantages, and aspects of the embodiments of this application will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of the application. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0052] Figure 1A flowchart of a multimodal data semantic alignment method based on a cross-modal attention mechanism according to an embodiment of this application is shown;

[0053] Figure 2 The overall framework diagram of MLLMs-MR according to an embodiment of this application is shown;

[0054] Figure 3 A flowchart illustrating the knowledge base construction process of an embodiment of this application is shown;

[0055] Figure 4 A schematic diagram of the ternary embedding center positioning according to an embodiment of this application is shown;

[0056] Figure 5 A block diagram of a multimodal data semantic alignment device based on a cross-modal attention mechanism according to an embodiment of this application is shown;

[0057] Figure 6 A block diagram of an exemplary electronic device capable of implementing embodiments of this application is shown. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0060] The following description, in conjunction with the accompanying drawings, illustrates a specific implementation of a multimodal data semantic alignment based on a cross-modal attention mechanism provided in this application.

[0061] Example 1

[0062] like Figure 1 The diagram shows a flowchart of the multimodal data semantic alignment based on a cross-modal attention mechanism according to this application. The method 100 includes:

[0063] S110, acquire multimodal data and its corresponding category labels, and map the multimodal data into multimodal embedding vectors using a pre-trained multimodal encoder.

[0064] In some embodiments, the multimodal data mentioned above includes at least one of image, video, audio, point cloud, thermal imaging, event, and text data, and the data generally comes from public datasets or custom acquisition devices (such as cameras, LiDAR, etc.).

[0065] Accordingly, multimodal encoders include image encoders, text encoders, and audio encoders, which are used to map multimodal data into embedding vectors, specifically:

[0066] First, the multimodal data is preprocessed. For example, image, video, and thermal imaging data are standardized and spatially aligned to ensure that the input resolution is consistent with the requirements of the multimodal large language model. The audio data is converted into Mel spectrograms and time-sliced ​​to generate fixed-length spectral segments to ensure the consistency of the input dimension.

[0067] Secondly, feature encoding is performed on the preprocessed multimodal data, and the encoded features are standardized (e.g., L2 normalization) to eliminate scale differences between modalities. Through linear layers or adapters, the features of different modalities are mapped to an embedding space of a unified dimension.

[0068] S120: Generate text descriptions corresponding to the multimodal data based on the multimodal large language model, and filter the text descriptions in conjunction with the category labels to construct a multimodal knowledge base.

[0069] In some embodiments, such as Figure 3 As shown, constructing a multimodal knowledge base includes the following steps:

[0070] A prompt word template is created ("Generate a description of this {modal type} that is related to the {Category}."). Based on this prompt word template, a text description corresponding to the multimodal data is generated using a multimodal large language model. The multimodal large language model includes the InstructBLIP model and the LLaMA-adapter model, and its architecture is as follows: Figure 2 As shown, the text descriptions corresponding to multimodal data are generated using the following formula:

[0071]

[0072] in, It is a description of the generated multimodal data. It is a text encoder. It is the extracted text embedding.

[0073] The cosine similarity between the generated text description and the corresponding category label is then calculated using a pre-trained text encoder. Specifically, the generated text description (e.g., “A white car turns left at an intersection”) and the corresponding category label (e.g., “car”) are input into a pre-trained text encoder (e.g., CLIP’s text encoder or BERT) to generate embedding vectors for the text description and category labels, respectively.

[0074] Further, the cosine similarity between the embedding vector of the text description and the embedding vector of the category label is calculated. :

[0075] ;

[0076] in, , These represent the embedding vectors for the text description and the category label, respectively.

[0077] Finally, text descriptions with a cosine similarity higher than a preset threshold are filtered out, and a knowledge base for the corresponding category is constructed. For example, a similarity threshold (such as 0.7) is set, and the text descriptions with a similarity higher than the threshold are filtered out and stored in the knowledge base according to the category.

[0078] S130, Based on the multimodal knowledge base, construct a multi-element embedding center that includes a basic prompt word embedding center, a multimodal knowledge base embedding center, and a large language model embedding center.

[0079] The basic prompt word embedding center is constructed through a dynamic template generator. The dynamic template generator combines multimodal contextual information (such as the visual features of the input image or the semantic attributes of the target category) to expand the preset static prompt template into a multimodal dynamic prompt word set, and maps it into an embedding vector through a text encoder.

[0080] The multimodal knowledge base embedding center can select the top N text descriptions with the highest cosine similarity to the category label from the multimodal knowledge base, and generate the data by means aggregation of the top N text descriptions through a text encoder;

[0081] The large language model embedding center can generate abstract semantic descriptions related to the category labels through the large language model, filter the abstract semantic descriptions, and map the filtered abstract semantic descriptions into embedding vectors through a text encoder;

[0082] Since the feature dimensions and numerical distributions of these three embedding centers are naturally different, directly merging them would lead to the problem of "numerical scale differences masking semantic associations". Therefore, before fusing the multimodal knowledge base embedding center, the basic prompt word embedding center and the large language model embedding center, it is necessary to adopt an appropriate normalization strategy to eliminate the differences between the embedding centers, such as using Min-Max normalization or Z-Score normalization.

[0083] After normalizing the multimodal knowledge base embedding center, basic prompt word embedding center, and large language model embedding center, they are then merged into a new localization center, namely the multi-embedding center, using a feature aggregation strategy. Specifically, as follows... Figure 4 As shown.

[0084] Among them, the above feature aggregation strategy can adopt a dynamic weighted fusion strategy, which determines the weight ratio of each embedding center through manual setting or model training, or it can introduce an attention mechanism to analyze the semantic correlation between the current task and each embedding center.

[0085] S140, through a cross-modal attention mechanism, the multimodal embedding vector is interacted with the text description to generate a semantically enhanced multimodal embedding vector.

[0086] In some embodiments, generating semantically enhanced multimodal embedding vectors includes the following steps:

[0087] The multimodal embedding vector is used as the key vector (K) and the value vector (V), and the text description is used as the query vector (Q).

[0088] The similarity between the query vector (Q) and the key vector (K) is calculated using a multi-head attention mechanism to generate an attention weight matrix.

[0089] The attention weight matrix is ​​modified, and the modified attention weight matrix is ​​weighted and summed with the value vector (V) to generate a semantically enhanced multimodal embedding vector. Specifically, the semantically enhanced multimodal embedding vector is generated using the following formula:

[0090]

[0091]

[0092] Specifically, a multi-head attention mechanism is used to calculate the similarity between the query vector (Q) and the key vector (K), generating an attention weight matrix, including:

[0093] The query vector (Q), key vector (K), and value vector (V) are projected onto multiple heads (h heads) through linear transformations. Assume the original dimensions are... Then the dimension of each head is:

[0094] ;

[0095] For each head, calculate the dot product of the transposes of Q and K to obtain the similarity score matrix. The formula is:

[0096] ;

[0097] To avoid gradient vanishing due to increased dimension dk, the fractional matrix is ​​scaled, and the softmax function is applied to the scaled fractional matrix to normalize it along the last dimension (key sequence direction), generating the attention weight matrix:

[0098] The correction of the attention weight matrix includes:

[0099] Items in the attention weight matrix with values ​​below a preset threshold are set to zero, and the attention weights are dynamically adjusted based on the semantic similarity between category labels and text descriptions in the multimodal data knowledge base.

[0100] S150, compare and learn the semantically enhanced multimodal embedding vector with the multivariate embedding center to achieve semantic alignment of multimodal data.

[0101] In some embodiments, semantic alignment of multimodal data includes the following steps:

[0102] The multi-element embedding centers are dynamically fused according to a dynamic weighted fusion strategy to generate a fused category representation.

[0103] The multimodal embedding vectors are semantically aligned and optimized simultaneously with the category features and each independent embedding center using a contrastive learning framework. Each independent embedding center includes a basic prompt word embedding center, a multimodal knowledge base embedding center, and a large language model embedding center.

[0104] The contrastive learning framework employs the contrastive loss function (InfoNCE) to semantically align and optimize the multimodal embedding vectors with the category features and each independent embedding center, thereby achieving a unified representation of seven modalities: image, video, audio, point cloud, thermal imaging, event, and text.

[0105] According to the above embodiments of this application, the following technical effects are achieved:

[0106] (1) This application achieves deep interaction between multimodal embedding vectors and text descriptions through a cross-modal attention mechanism, dynamically adjusting attention weights to focus on key semantic information, effectively solving the problem of information fragmentation between modalities in traditional methods. Combined with high-quality text descriptions selected from the knowledge base, the accuracy of semantic representation is enhanced, and the alignment effect of multimodal data in a unified semantic space is significantly better than that of single-modal or simple fusion methods.

[0107] (2) In constructing the multi-embedding center, this application innovatively integrates three knowledge sources: basic prompt words, knowledge base filtered text, and abstract semantics of a large language model. By combining prior knowledge with real-time generated features through a dynamic weighting strategy, it not only retains the basic concepts of the domain but also incorporates dynamic semantics of the context, significantly improving the model's adaptability to noisy data and long-tail categories, especially showing stronger generalization performance in low-resource scenarios.

[0108] (3) This application uses multimodal large language models such as InstructBLIP to generate descriptions and combines a cosine similarity threshold screening mechanism to construct a high-quality multimodal knowledge base. Compared with traditional manual annotation methods, this scheme improves the efficiency of knowledge base construction while ensuring semantic relevance, and effectively suppresses redundant information interference in the generated text through attention weight correction technology, thereby reducing computational complexity.

[0109] (4) The multimodal large language model (MLLMs-MR) of this application for multimodal retrieval demonstrates strong multimodal zero-shot recognition performance compared to the baseline UniBind, for example, an accuracy improvement of 6.42% on the video dataset MSR-VTT. Excellent results were also achieved in multimodal fine-tuning, for example, an accuracy improvement of 8.19% on the audio dataset ESC 5-folds.

[0110] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0111] The above is an introduction to the method embodiments. The following describes the solution described in this application through device embodiments.

[0112] Figure 5 A block diagram of a cross-modal alignment device 500 for fusing multi-semantic embedding centers according to an embodiment of this application is shown. Figure 5 As shown, the device 500 includes:

[0113] The multimodal embedding vector generation unit 510 is used to acquire multimodal data and its corresponding category labels, and to map the multimodal data into multimodal embedding vectors through a pre-trained multimodal encoder.

[0114] The multimodal knowledge base construction unit 520 is used to generate text descriptions corresponding to the multimodal data based on the multimodal large language model, and to filter the text descriptions in combination with the category tags to construct a multimodal knowledge base;

[0115] The multi-element embedding center construction unit 530 is used to construct a multi-element embedding center based on the multimodal knowledge base, including a basic prompt word embedding center, a multimodal knowledge base embedding center, and a large language model embedding center;

[0116] The semantic enhancement unit 540 is used to interact the multimodal embedding vector with the text description through a cross-modal attention mechanism to generate a semantically enhanced multimodal embedding vector.

[0117] The contrastive learning unit 550 is used to compare and learn the semantically enhanced multimodal embedding vector with the multivariate embedding center to achieve semantic alignment of multimodal data.

[0118] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0119] The acquisition, storage, and application of user personal information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0120] Figure 6 A schematic block diagram of an electronic device 600 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0121] Electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in ROM 602 or a computer program loaded into RAM 603 from storage unit 608. RAM 603 may also store various programs and data required for the operation of electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. I / O interface 605 is also connected to bus 604.

[0122] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0123] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as method 100. For example, in some embodiments, method 100 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via ROM 402 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform method 100 by any other suitable means (e.g., by means of firmware).

[0124] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0125] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0126] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0127] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, or tactile input).

[0128] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0129] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0130] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0131] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A multimodal data semantic alignment method based on a cross-modal attention mechanism, characterized in that, include: Obtain multimodal data and its corresponding category labels, and map the multimodal data into multimodal embedding vectors using a pre-trained multimodal encoder; The multimodal large language model generates text descriptions corresponding to the multimodal data, and the text descriptions are filtered in combination with the category labels to construct a multimodal knowledge base; Based on the aforementioned multimodal knowledge base, a multi-element embedding center is constructed, comprising a basic prompt word embedding center, a multimodal knowledge base embedding center, and a large language model embedding center; By using a cross-modal attention mechanism, the multimodal embedding vector is interacted with the text description to generate a semantically enhanced multimodal embedding vector; The semantically enhanced multimodal embedding vectors are compared and learned with the multi-element embedding centers to achieve semantic alignment of multimodal data; The construction of a multi-element embedding center based on the multimodal knowledge base, comprising a basic prompt word embedding center, a multimodal knowledge base embedding center, and a large language model embedding center, includes: The top N text descriptions with the highest cosine similarity to the category label are selected from the multimodal knowledge base. The mean of the top N text descriptions is aggregated by a text encoder to generate the multimodal knowledge base embedding center. The preset static prompt template is expanded into a multimodal dynamic prompt word set by a dynamic template generator, and mapped into an embedding vector by a text encoder to generate the basic prompt word embedding center; Based on the large language model, an abstract semantic description related to the category label is generated, and the abstract semantic description is filtered. The filtered abstract semantic description is then mapped into an embedding vector through a text encoder to generate the large language model embedding center. The multimodal knowledge base embedding center is integrated with the basic prompt word embedding center and the large language model embedding center to form a multi-element embedding center.

2. The method according to claim 1, characterized in that, The step of generating text descriptions corresponding to the multimodal data based on a multimodal large language model, and filtering the text descriptions in conjunction with the category labels to construct a multimodal knowledge base includes: Generate a text description corresponding to the multimodal data based on the multimodal large language model; The cosine similarity between the generated text description and the corresponding category label is calculated using a pre-trained text encoder. A knowledge base for the corresponding category is constructed based on text descriptions with a cosine similarity higher than a preset threshold.

3. The method according to claim 2, characterized in that, The multimodal large language model includes the InstructBLIP model and the LLaMA-adapter model.

4. The method according to claim 1, characterized in that, The step of interacting the multimodal embedding vector with the text description through a cross-modal attention mechanism to generate a semantically enhanced multimodal embedding vector includes: The multimodal embedding vector is used as the key vector and value vector, and the text description is used as the query vector; The similarity between the query vector and the key vector is calculated using a multi-head attention mechanism to generate an attention weight matrix; The attention weight matrix is ​​modified, and the modified attention weight matrix is ​​weighted and summed with the value vector to generate a semantically enhanced multimodal embedding vector.

5. The method according to claim 4, characterized in that, The modification of the attention weight matrix includes: Set the values ​​of items in the attention weight matrix that are below a preset threshold to zero; The attention weights are dynamically adjusted based on the semantic similarity between category labels and text descriptions in the multimodal data knowledge base.

6. The method according to claim 1, characterized in that, The step of comparing and learning the semantically enhanced multimodal embedding vector with the multivariate embedding center to achieve semantic alignment of multimodal data includes: The multi-element embedding centers are dynamically fused according to a dynamic weighted fusion strategy to generate a fused category representation. The multimodal embedding vectors are semantically aligned and optimized simultaneously with the fused category representation and each independent embedding center using a contrastive learning framework. Each independent embedding center includes a basic cue word embedding center, a multimodal knowledge base embedding center, and a large language model embedding center.

7. A multimodal data semantic alignment device based on a cross-modal attention mechanism, characterized in that, include: A multimodal embedding vector generation unit is used to acquire multimodal data and its corresponding category labels, and to map the multimodal data into multimodal embedding vectors through a pre-trained multimodal encoder. A multimodal knowledge base construction unit is used to generate text descriptions corresponding to the multimodal data based on a multimodal large language model, and to filter the text descriptions in conjunction with the category tags to construct a multimodal knowledge base; The multi-element embedding center construction unit is used to construct a multi-element embedding center based on the multimodal knowledge base, comprising a basic prompt word embedding center, a multimodal knowledge base embedding center, and a large language model embedding center; specifically including: The top N text descriptions with the highest cosine similarity to the category label are selected from the multimodal knowledge base. The mean of the top N text descriptions is aggregated by a text encoder to generate the multimodal knowledge base embedding center. The preset static prompt template is expanded into a multimodal dynamic prompt word set by a dynamic template generator, and mapped into an embedding vector by a text encoder to generate the basic prompt word embedding center; Based on the large language model, an abstract semantic description related to the category label is generated, and the abstract semantic description is filtered. The filtered abstract semantic description is then mapped into an embedding vector through a text encoder to generate the large language model embedding center. The multimodal knowledge base embedding center is integrated with the basic prompt word embedding center and the large language model embedding center to form a multi-element embedding center; The semantic enhancement unit is used to interact the multimodal embedding vector with the text description through a cross-modal attention mechanism to generate a semantically enhanced multimodal embedding vector. The contrastive learning unit is used to compare and learn the semantically enhanced multimodal embedding vector with the multivariate embedding center to achieve semantic alignment of multimodal data.

8. An electronic device, characterized in that, The electronic device includes: At least one processor, and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the method of any one of claims 1-6.

Images