A cross-modal alignment method and device fusing multi-element semantic embedding centers

Abstract

The application relates to the technical field of cross-modal semantic alignment, and provides a cross-modal alignment method and device fusing multi-element semantic embedding centers. The method comprises the following steps: mapping inputted multi-modal data into embedding vectors through a pre-trained multi-modal encoder; generating multi-element embedding centers containing respective independent embedding centers based on a target category; dynamically fusing the multi-element embedding centers according to a dynamic weighting fusion strategy to generate fused category representations; simultaneously performing semantic alignment optimization on the embedding vectors of the multi-modal data, the category features and the respective independent embedding centers through a contrast learning framework; calculating the similarity of the embedding vectors of the multi-modal data, the category features and the respective independent embedding centers; and selecting the category with the highest similarity as a recognition result output. In this way, the application makes full use of the complementary information of the multi-modal data, and realizes more accurate cross-modal alignment.

Description

A cross-modal alignment method and apparatus integrating multi-semantic embedding centers Technical Field

[0001] This application relates to the field of cross-modal semantic alignment technology, and in particular to a cross-modal alignment method and apparatus that integrates multi-semantic embedding centers. Background Technology

[0002] In the field of multimodal contrastive learning, embedding center localization is a core step, aiming to construct highly representative embedding vectors for each semantic category to achieve accurate cross-modal alignment and semantic understanding. Traditional methods such as CLIP directly use the text embedding of the category name as the center, but due to the limited semantic expression, it is difficult to fully capture intra-class diversity. While ImageBind enhances semantic information by leveraging the central position of the image, its over-reliance on the image modality leads to an imbalance in intermodal representation. Subsequent methods include Ovis, which uses a visual embedding table to structurally align visual and textual data and generates probabilistic markers to locate the center; C-MCR, which uses an intermediate modality to connect different pre-trained contrastive spaces to achieve semantic alignment; and UniBind, which uses a large language model to generate multi-descriptive text and selects highly similar text to construct the embedding center, thus improving semantic richness. However, most existing technologies are limited to relying on category text or text modality generated by a large language model to construct the center, failing to fully explore the complementary advantages between multimodal data, and still have significant room for optimization in cross-modal information fusion and center representation capabilities. Summary of the Invention

[0003] This application provides a cross-modal alignment method and apparatus that integrates multiple semantic embedding centers. It aims to construct a multiple semantic embedding center that integrates a basic prompt word embedding center, a multimodal knowledge base embedding center, and a large language model embedding center. This breaks through the limitations of traditional methods that rely on a single modality to construct the center. The three complement each other to cover intra-class diversity and significantly enhance the semantic expressive power of the center.

[0004] According to a first aspect of this application, a cross-modal alignment method for fusing multi-semantic embedding centers is provided, comprising:

[0005] The input multimodal data is mapped into an embedding vector using a pre-trained multimodal encoder;

[0006] Based on the target category, a multi-element embedding center containing each independent embedding center is dynamically generated, wherein each independent embedding center includes a basic prompt word embedding center, a multimodal knowledge base embedding center, and a large language model embedding center;

[0007] The multi-element embedding centers are dynamically fused according to a dynamic weighted fusion strategy to generate a fused category representation.

[0008] The embedding vectors of the multimodal data are semantically aligned and optimized simultaneously with the category features and each independent embedding center using a contrastive learning framework.

[0009] Calculate the similarity between the embedding vector of the multimodal data and the category features and each independent embedding center, and select the category with the highest similarity as the recognition result output.

[0010] In addition to the aspects described above and any possible implementation, a further implementation is provided in which the basic cue word embedding center is constructed by a dynamic template generator, which expands a preset static cue template into a multimodal dynamic cue word set and maps it into an embedding vector through a text encoder.

[0011] In addition to the aspects described above and any possible implementations, a further implementation is provided in which the multimodal knowledge base embedding center is constructed through the following steps:

[0012] Extract graphic and textual descriptions related to the target category from a multimodal knowledge base;

[0013] Valid image and text descriptions are filtered from the image and text descriptions through a dual-channel filtering mechanism, wherein the dual-channel filtering mechanism includes semantic relevance calculation based on cosine similarity and visual-text consistency detection.

[0014] The selected valid image and text descriptions are aggregated using the mean to generate embedding centers.

[0015] In addition to the aspects described above and any possible implementations, a further implementation is provided in which the large language model embedding center is constructed through the following steps:

[0016] Utilize a large language model to perform chained reasoning on the target category to generate a multi-level semantic description;

[0017] The multi-level semantic description is structured and represented by a graph attention network to generate an embedding vector with topological relationships.

[0018] As described above and in any possible implementation, a further implementation is provided, wherein the step of dynamically fusing the multi-embedding centers according to a dynamic weighted fusion strategy to generate a fused category representation includes:

[0019] The embedding vectors of the multimodal data are mapped to the feature space of each independent embedding center;

[0020] The cosine similarity between the mapped embedding vector and each independent embedding center vector is calculated to generate a modal affinity score that reflects the degree of association between the multimodal data and each independent embedding center.

[0021] A differentiable weight allocation matrix is ​​constructed based on the modal affinity score, wherein the element values ​​of the differentiable weight allocation matrix correspond to the fusion weights of each independent embedding center;

[0022] Based on the differentiable weight allocation matrix, the features of the ternary embedding center are soft-selected and combined by the gated fusion unit to generate a preliminary fusion representation.

[0023] The basic prompt words are embedded in the central features and then linearly transformed before being added to the preliminary fusion representation to generate the fused category representation.

[0024] In addition to the aspects and any possible implementations described above, a further implementation is provided in which the embedding vectors of the multimodal data are simultaneously semantically aligned and optimized with the category features and each independent embedding center using a contrastive learning framework, including the following parallel alignment paths:

[0025] Main alignment path: Constrain the semantic distance between the embedding vectors of multimodal data and the fused category representations by using an improved semantically aware InfoNCE loss function;

[0026] Auxiliary alignment path: A multi-teacher distillation framework is adopted to ensure that the embedding vectors of multimodal data are consistent with each independent embedding center.

[0027] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein calculating the similarity between the embedding vector of the multimodal data and the category features and each independent embedding center, and selecting the category with the highest similarity as the recognition result output, includes:

[0028] Calculate the Jensen-Shannon divergence value for the predicted class of each independent embedding center;

[0029] If the Jensen-Shannon divergence is lower than a preset threshold, it is determined that semantic consensus has been reached, and the category with the highest similarity in the average probability distribution is output as the recognition result.

[0030] If the Jensen-Shannon divergence is higher than or equal to a preset threshold, semantic divergence is determined, and local and global attention weights are generated using a multi-scale similarity calculator.

[0031] By combining the Jensen-Shannon divergence with the local and global attention weights, the similarity between the embedding vector of the multimodal data and the category features and each independent embedding center is calculated, and the category with the highest similarity is output as the recognition result.

[0032] According to a second aspect of this application, a cross-modal alignment apparatus for fusing multiple semantic embedding centers is provided, comprising:

[0033] The multimodal data mapping unit is used to map the input multimodal data into an embedding vector using a pre-trained multimodal encoder;

[0034] The multi-element embedding center generation unit is used to dynamically generate multi-element embedding centers containing various independent embedding centers based on the target category. The various independent embedding centers include basic prompt word embedding centers, multimodal knowledge base embedding centers, and large language model embedding centers.

[0035] The dynamic fusion unit is used to dynamically fuse the multi-element embedding centers according to a dynamic weighted fusion strategy to generate a fused category representation.

[0036] The semantic alignment optimization unit is used to perform semantic alignment optimization on the embedding vector of the multimodal data with the category features and each independent embedding center simultaneously through a contrastive learning framework.

[0037] The recognition result output unit is used to calculate the similarity between the embedding vector of the multimodal data and the category features and each independent embedding center, and select the category with the highest similarity as the recognition result output.

[0038] According to a third aspect of this application, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0039] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0040] Compared with the prior art, this application achieves the following beneficial effects:

[0041] (1) This application breaks through the limitations of traditional methods that rely on a single modality to construct the center by integrating the basic prompt word embedding center, the multimodal knowledge base embedding center, and the large language model embedding center. The basic prompt words dynamically generate a multimodal prompt word set, the multimodal knowledge base combines text and image information, and the large language model provides multi-level semantic descriptions. The three complement each other to cover intra-class diversity and significantly enhance the semantic expression ability of the center.

[0042] (2) This application adaptively allocates fusion weights based on the modality affinity scores between multimodal data and each independent embedding center to avoid the modality imbalance problem. At the same time, by mapping the data embedding vectors to different center feature spaces for fusion, the complementary information of multimodal data is fully utilized to achieve more accurate cross-modal alignment.

[0043] (3) This application adopts parallel optimization of the main alignment path and the auxiliary alignment path. The main path constrains the semantic distance between the data and the fused category representation through the improved loss function, while the auxiliary path uses multi-teacher distillation to keep the data consistent with each independent center. This dual guarantee improves the accuracy and stability of semantic alignment.

[0044] (4) This application judges semantic consensus or disagreement based on Jensen-Shannon divergence. When there is disagreement, it combines multi-scale similarity calculation and comprehensively considers local and global information to adjust the similarity calculation, avoids misjudgment by a single standard, and makes the category recognition result more reliable, especially suitable for complex semantic scenarios.

[0045] It should be understood that the description in the Summary Section is not intended to limit the key or essential features of the embodiments of this application, nor is it intended to restrict the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0046] The above and other features, advantages, and aspects of the embodiments of this application will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of the application. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0047] Figure 1 shows a flowchart of the cross-modal alignment method for fusing multi-semantic embedding centers according to an embodiment of this application;

[0048] Figure 2 shows a schematic diagram of the ternary embedding center positioning according to an embodiment of this application;

[0049] Figure 3 shows a block diagram of a cross-modal alignment device for fusing multi-semantic embedding centers according to an embodiment of this application;

[0050] Figure 4 shows a block diagram of an exemplary electronic device capable of implementing embodiments of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0053] The following description, in conjunction with the accompanying drawings, illustrates a specific implementation of the cross-modal alignment method for integrating multi-semantic embedding centers provided in this application.

[0054] Example 1

[0055] Figure 1 shows a flowchart of a cross-modal alignment method for fusing multi-semantic embedding centers according to this application. The method 100 includes:

[0056] S110 maps the input multimodal data into embedding vectors using a pre-trained multimodal encoder.

[0057] In some embodiments, the multimodal data mentioned above includes at least one of image, video, audio, point cloud, thermal imaging, event, and text data, and the data generally comes from public datasets or custom acquisition devices (such as cameras, LiDAR, etc.).

[0058] Accordingly, multimodal encoders include image encoders, text encoders, and audio encoders, which are used to map multimodal data into embedding vectors, specifically:

[0059] First, the multimodal data is preprocessed. For example, image, video, and thermal imaging data are standardized and spatially aligned to ensure that the input resolution is consistent with the requirements of the multimodal large language model. The audio data is converted into Mel spectrograms and time-sliced ​​to generate fixed-length spectral segments to ensure the consistency of the input dimension.

[0060] Secondly, feature encoding is performed on the preprocessed multimodal data, and the encoded features are standardized (e.g., L2 normalization) to eliminate scale differences between modalities. Through linear layers or adapters, the features of different modalities are mapped to an embedding space of a unified dimension.

[0061] S120, Based on the target category, dynamically generate a multi-element embedding center containing each independent embedding center, wherein each independent embedding center includes a basic prompt word embedding center, a multimodal knowledge base embedding center, and a large language model embedding center.

[0062] In some embodiments, the basic cue word embedding center is constructed through a dynamic template generator. The dynamic template generator uses a cross-attention mechanism to combine multimodal context information with the global features of the input multimodal data to interact with the static template, thereby expanding the preset static cue template into a multimodal dynamic cue word set, and mapping it to an embedding vector through a pre-trained text encoder. Specifically, this includes the following steps:

[0063] (a) Extract global features of the input image (such as the output of the CLIP visual encoder), and generate dynamic descriptive words related to the image content by interacting with the static template through a cross-attention mechanism;

[0064] (b) Retrieve image and text descriptions related to the target category from multimodal knowledge bases (such as Conceptual Captions), extract high-frequency keywords (such as "furry" and "large dog") to supplement the multimodal dynamic prompt word set, and use models such as GPT-3 to generate diverse semantic descriptions;

[0065] (c) Filter low-quality prompts by semantic relevance score and retain entries with scores higher than the threshold (e.g., 0.7);

[0066] (d) Input the filtered set of dynamic prompt words into a pre-trained text encoder (such as CLIP text encoder or BERT) to generate embedding vectors, and then further normalize the generated embedding vectors.

[0067] In some embodiments, the above-mentioned multimodal knowledge base embedding center is constructed through the following steps:

[0068] Extract image and text descriptions related to the target category from a multimodal knowledge base; for example, use a pre-trained multimodal retrieval model (such as CLIP or ALIGN) to map the target category name to a text embedding and retrieve the most relevant image and text pairs from the knowledge base.

[0069] Valid image-text descriptions are selected from the image-text descriptions using a dual-channel filtering mechanism. The dual-channel filtering mechanism includes semantic relevance calculation based on cosine similarity and visual-text consistency detection. Weights are assigned based on the comprehensive score of the image-text pair (semantic relevance + visual consistency), with higher weights indicating a greater contribution to the embedding center.

[0070] The selected valid image and text descriptions are aggregated using the mean to generate embedding centers.

[0071] The semantic relevance calculation based on cosine similarity includes: using the cosine similarity algorithm to calculate the cosine similarity between the text embedding in the image and text description and the text embedding of the target category, and retaining text with a screening threshold greater than a preset threshold (such as 0.75);

[0072] Visual-text consistency detection includes: using a pre-trained multimodal retrieval model to calculate the matching score between image embedding and text embedding, determining image-text pairs with a matching score ≥ 0.8 as "consistent", and retaining valid image-text pairs that simultaneously satisfy "semantic relevance ≥ 0.75 and matching score ≥ 0.8".

[0073] In some embodiments, the above-mentioned large language model embedding center is constructed through the following steps:

[0074] Utilize a large language model to perform chained reasoning on the target category to generate a multi-level semantic description;

[0075] The multi-level semantic description is structured and represented by a graph attention network to generate an embedding vector with topological relationships.

[0076] For example, chained reasoning can be set to three levels, with the appearance description, functional description, and comparative description generated through the first, second, and third levels of reasoning, respectively.

[0077] S130, The multi-element embedding centers are dynamically fused according to the dynamic weighted fusion strategy to generate the fused category representation.

[0078] In some embodiments, the step of dynamically fusing the multi-embedded centers according to a dynamic weighted fusion strategy to generate a fused category representation includes the following steps:

[0079] Step S131: Map the embedding vectors of the multimodal data to the feature space of each independent embedding center.

[0080] Specifically, different modal data (text, images, audio, etc.) can be mapped to their respective independent feature spaces through independent neural networks (such as fully connected layers, convolutional layers, etc.), with each space corresponding to the core feature (embedding center) of a modality.

[0081] Step S132: Calculate the cosine similarity between the mapped embedding vector and each independent embedding center vector, and generate a modal affinity score that reflects the degree of association between the multimodal data and each independent embedding center.

[0082] Step S133: Construct a differentiable weight allocation matrix based on the modal affinity score, wherein the element values ​​of the differentiable weight allocation matrix correspond to the fusion weights of each independent embedding center;

[0083] Specifically, the affinity scores of each modality are input into a differentiable function (such as Softmax) to generate a normalized fusion weight vector, which is then expanded into a weight matrix. Each element in the matrix corresponds to a fusion weight of an embedding center, which is used to weight and aggregate multimodal features.

[0084] Step S134: Based on the differentiable weight allocation matrix, the features of the ternary embedding center are soft-selected and combined by the gated fusion unit to generate a preliminary fusion representation.

[0085] Specifically, a differentiable weight allocation matrix is ​​used to generate dynamic gating weights (such as Sigmoid or Softmax) to weight and filter the features of the text, image, and audio embedding centers. The filtering results of each modality are then added or concatenated to generate a preliminary fusion representation.

[0086] Step S135: After linear transformation, the basic prompt word embedded central feature is added to the preliminary fusion representation to generate the fused category representation.

[0087] S140, The embedding vector of the multimodal data is semantically aligned and optimized with the category features and each independent embedding center simultaneously through a contrastive learning framework.

[0088] In some embodiments, the embedding vectors of the multimodal data are simultaneously semantically aligned and optimized with the category features and each independent embedding center, including the following parallel alignment paths:

[0089] Main alignment path: Constrain the semantic distance between the embedding vectors of multimodal data and the fused category representations by using an improved semantically aware InfoNCE loss function;

[0090] Auxiliary alignment path: A multi-teacher distillation framework is adopted to ensure that the embedding vectors of multimodal data are consistent with each independent embedding center.

[0091] S150, calculate the similarity between the embedding vector of the multimodal data and the category features and each independent embedding center, and select the category with the highest similarity as the recognition result output.

[0092] In some embodiments, calculating the similarity between the embedding vector of the multimodal data and the category features and each independent embedding center, and selecting the category with the highest similarity as the recognition result output, includes:

[0093] Calculate the Jensen-Shannon divergence value for the predicted class of each independent embedding center;

[0094] If the Jensen-Shannon divergence is lower than a preset threshold, it is determined that semantic consensus has been reached, and the category with the highest similarity in the average probability distribution is output as the recognition result.

[0095] If the Jensen-Shannon divergence is higher than or equal to a preset threshold, semantic divergence is determined, and local and global attention weights are generated using a multi-scale similarity calculator.

[0096] By combining the Jensen-Shannon divergence value with the local and global attention weights, the similarity between the embedding vector of the multimodal data and the category features and each independent embedding center is calculated, and the category with the highest similarity is output as the recognition result.

[0097] According to the above embodiments of this application, the following technical effects are achieved:

[0098] (1) This application breaks through the limitations of traditional methods that rely on a single modality to construct the center by integrating the basic prompt word embedding center, the multimodal knowledge base embedding center, and the large language model embedding center. The basic prompt words dynamically generate a multimodal prompt word set, the multimodal knowledge base combines text and image information, and the large language model provides multi-level semantic descriptions. The three complement each other to cover intra-class diversity and significantly enhance the semantic expression ability of the center.

[0099] (2) This application adaptively allocates fusion weights based on the modality affinity scores between multimodal data and each independent embedding center to avoid the modality imbalance problem. At the same time, by mapping the data embedding vectors to different center feature spaces for fusion, the complementary information of multimodal data is fully utilized to achieve more accurate cross-modal alignment.

[0100] (3) This application adopts parallel optimization of the main alignment path and the auxiliary alignment path. The main path constrains the semantic distance between the data and the fused category representation through the improved loss function, while the auxiliary path uses multi-teacher distillation to keep the data consistent with each independent center. This dual guarantee improves the accuracy and stability of semantic alignment.

[0101] (4) This application judges semantic consensus or disagreement based on Jensen-Shannon divergence. When there is disagreement, it combines multi-scale similarity calculation and comprehensively considers local and global information to adjust the similarity calculation, avoids misjudgment by a single standard, and makes the category recognition result more reliable, especially suitable for complex semantic scenarios.

[0102] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0103] The above is an introduction to the method embodiments. The following describes the solution described in this application through device embodiments.

[0104] Figure 3 shows a block diagram of a cross-modal alignment device 300 for fusing multi-semantic embedding centers according to an embodiment of this application. As shown in Figure 3, the device 300 includes:

[0105] The multimodal data mapping unit 310 is used to map the input multimodal data into an embedding vector using a pre-trained multimodal encoder;

[0106] The multi-embedded center generation unit 320 is used to dynamically generate multi-embedded centers containing each independent embedded center based on the target category, wherein each independent embedded center includes a basic prompt word embedded center, a multimodal knowledge base embedded center, and a large language model embedded center.

[0107] The dynamic fusion unit 330 is used to dynamically fuse the multi-element embedding centers according to a dynamic weighted fusion strategy to generate a fused category representation.

[0108] The semantic alignment optimization unit 340 is used to perform semantic alignment optimization on the embedding vector of the multimodal data with the category features and each independent embedding center simultaneously through a contrastive learning framework.

[0109] The recognition result output unit 350 is used to calculate the similarity between the embedding vector of the multimodal data and the category features and each independent embedding center, and select the category with the highest similarity as the recognition result output.

[0110] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0111] The acquisition, storage, and application of user personal information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0112] According to embodiments of this application, this application also provides an electronic device, a readable storage medium, and a computer program product.

[0113] Figure 4 illustrates a schematic block diagram of an electronic device 400 that can be used to implement embodiments of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0114] Electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in ROM 402 or a computer program loaded into RAM 403 from storage unit 408. RAM 403 may also store various programs and data required for the operation of electronic device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via bus 404. I / O interface 405 is also connected to bus 404.

[0115] Multiple components in electronic device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of displays, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0116] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as method 100. For example, in some embodiments, method 100 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to perform method 100 by any other suitable means (e.g., by means of firmware).

[0117] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0118] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0119] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0120] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, or tactile input).

[0121] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0122] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0123] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0124] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A cross-modal alignment method integrating multi-semantic embedding centers, characterized in that, include: The input multimodal data is mapped into an embedding vector using a pre-trained multimodal encoder; Based on the target category, a multivariate embedding center is dynamically generated, comprising each independent embedding center, including a basic prompt word embedding center, a multimodal knowledge base embedding center, and a large language model embedding center. The multivariate embedding centers are then dynamically fused according to a dynamic weighted fusion strategy to generate a fused category representation. This includes: mapping the embedding vectors of the multimodal data to the feature spaces of each independent embedding center; calculating the cosine similarity between the mapped embedding vectors and the vectors of each independent embedding center to generate a modal affinity score indicating the degree of association between the multimodal data and each independent embedding center; and constructing a differentiable weight allocation matrix based on the modal affinity score. The elements of the differentiable weight allocation matrix correspond to the fusion weights of each independent embedding center. Based on the differentiable weight allocation matrix, the features of the ternary embedding centers are soft-selected and combined by a gated fusion unit to generate a preliminary fusion representation. The basic prompt word embedding center features are linearly transformed and added to the preliminary fusion representation to generate a fused category representation. The embedding vectors of the multimodal data are semantically aligned and optimized with the category features and each independent embedding center using a contrastive learning framework. The similarity between the embedding vectors of the multimodal data and the category features and each independent embedding center is calculated, and the category with the highest similarity is selected as the recognition result output.

2. The method according to claim 1, characterized in that, The basic prompt word embedding center is constructed through a dynamic template generator, which expands the preset static prompt template into a multimodal dynamic prompt word set and maps it into an embedding vector through a text encoder.

3. The method according to claim 1, characterized in that, The multimodal knowledge base embedding center is constructed through the following steps: extracting image and text descriptions related to the target category from the multimodal knowledge base; filtering effective image and text descriptions from the image and text descriptions through a dual-channel filtering mechanism, wherein the dual-channel filtering mechanism includes semantic relevance calculation based on cosine similarity and visual-text consistency detection; and generating the embedding center by aggregating the filtered effective image and text descriptions through mean.

4. The method according to claim 1, characterized in that, The large language model embedding center is constructed through the following steps: using the large language model to perform chained reasoning on the target category to generate a multi-level semantic description; and using a graph attention network to perform structured representation learning on the multi-level semantic description to generate an embedding vector with topological relationships.

5. The method according to claim 1, characterized in that, The step of using a contrastive learning framework to perform semantic alignment optimization on the embedding vectors of the multimodal data with the category features and each independent embedding center includes the following parallel alignment paths: main alignment path: constraining the semantic distance between the embedding vectors of the multimodal data and the fused category representation through an improved semantic-aware InfoNCE loss function; auxiliary alignment path: using a multi-teacher distillation framework to ensure that the embedding vectors of the multimodal data are consistent with each independent embedding center.

6. The method according to claim 1, characterized in that, The step of calculating the similarity between the embedding vector of the multimodal data and the category features and each independent embedding center, and selecting the category with the highest similarity as the recognition result output, includes: calculating the Jensen-Shannon divergence value of the predicted category of each independent embedding center; if the Jensen-Shannon divergence is lower than a preset threshold, it is determined that semantic consensus has been reached, and the category with the highest similarity in the average probability distribution is output as the recognition result; if the Jensen-Shannon divergence is higher than or equal to the preset threshold, it is determined that semantic divergence has occurred, and local and global attention weights are generated through a multi-scale similarity calculator; combining the Jensen-Shannon divergence and the local and global attention weights, the similarity between the embedding vector of the multimodal data and the category features and each independent embedding center is calculated, and the category with the highest similarity is output as the recognition result.

7. A cross-modal alignment device integrating a multi-semantic embedding center, characterized in that, include: A multimodal data mapping unit is used to map input multimodal data into embedding vectors using a pre-trained multimodal encoder; a multi-embedding center generation unit is used to dynamically generate multi-embedding centers containing independent embedding centers based on the target category, wherein each independent embedding center includes a basic prompt word embedding center, a multimodal knowledge base embedding center, and a large language model embedding center; a dynamic fusion unit is used to dynamically fuse the multi-embedding centers according to a dynamic weighted fusion strategy to generate a fused category representation, including: mapping the embedding vectors of the multimodal data to the feature spaces of each independent embedding center; calculating the cosine similarity between the mapped embedding vectors and the vectors of each independent embedding center, and generating a modal affinity score indicating the degree of association between the multimodal data and each independent embedding center; A differentiable weight allocation matrix is ​​constructed based on the modality affinity score, wherein the element values ​​of the differentiable weight allocation matrix correspond to the fusion weights of each independent embedding center. According to the differentiable weight allocation matrix, the features of the ternary embedding centers are soft-selected and combined through a gated fusion unit to generate a preliminary fusion representation. The basic prompt word embedding center features are linearly transformed and added to the preliminary fusion representation to generate a fused category representation. A semantic alignment optimization unit is used to perform semantic alignment optimization on the embedding vectors of the multimodal data with the category features and each independent embedding center simultaneously through a contrastive learning framework. A recognition result output unit is used to calculate the similarity between the embedding vectors of the multimodal data and the category features and each independent embedding center, and select the category with the highest similarity as the recognition result output.

8. An electronic device, characterized in that, The electronic device includes: at least one processor, and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the method of any one of claims 1-6.

Images