Knowledge enhancement method and device fusing heterogeneous multi-modal retrieval and cross-modal alignment

Abstract

The invention provides a knowledge enhancement method and device fusing heterogeneous multi-modal retrieval and cross-modal alignment. The method comprises the following steps: constructing a knowledge base containing different modal data; obtaining query features, and obtaining M data features from the knowledge base to form an initial data feature set, wherein the cosine similarities of the M data features and the query features are from high to low; reordering the initial data feature set based on the semantic similarity score to obtain a final data feature set; contextual cues are generated based on the final data feature set, with which knowledge enhancement is performed on the user query. According to the scheme, the initial feature set is formed through the cosine similarity, and then reordering is performed by using the pre-trained multi-modal semantic similarity score calculation model, so that globally similar but locally unmatched noise segments are filtered through fine-grained semantic matching, and the semantic correlation of a retrieval result is remarkably improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a knowledge enhancement method and apparatus that integrates heterogeneous multimodal retrieval and cross-modal alignment. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have demonstrated outstanding capabilities in the field of natural language processing. Retrieval-Augmented Generation (RAG) technology, as a mainstream method to improve the factual accuracy and expand the knowledge coverage of large models, is widely used in scenarios such as intelligent question answering and content generation. The core logic of traditional RAG technology is to retrieve text fragments related to the user's query from an external text knowledge base and input them as contextual information into the large language model, thereby enabling the model to generate more reliable and relevant answers. This has achieved good application results in pure text knowledge interaction scenarios.

[0003] However, information in the real world exhibits significant heterogeneous and multimodal characteristics. Multimodal information carriers such as web pages with images and text, video content with accompanying narration, and multimedia courseware with audio explanations have become the main forms of knowledge. Traditional RAG technology, which relies solely on the text modality, has obvious limitations in information utilization and will lose a large amount of potential useful knowledge in multimodal carriers. This results in a significant reduction in the richness and accuracy of the content generated by the model. In particular, in cross-modal knowledge reasoning and question answering tasks involving visual features and auditory information, the performance of traditional RAG technology is even more difficult to meet the needs of practical applications.

[0004] While multimodal retrieval technology has been applied in single fields such as image search and video retrieval, it has not yet been deeply integrated with the RAG generation process, lacking a unified retrieval framework for heterogeneous multimodal data that mixes text, images, audio, and video. Furthermore, existing methods lack effective cross-modal alignment mechanisms; the feature representations of different modalities are independent, making precise matching at the semantic level impossible. This hinders cross-modal knowledge retrieval and information retrieval, preventing the effective utilization of large-scale models and creating a technological barrier to knowledge utilization.

[0005] Some studies have attempted to compensate for the aforementioned shortcomings by directly inputting multimodal information into large multimodal models such as GPT-4V. However, these methods face significant technical bottlenecks: firstly, non-textual modal information needs to be encoded into model-specific tokens, placing extremely high demands on hardware computing resources and increasing the cost of technology implementation; secondly, the model's context window has fixed limitations, resulting in a limited size of the available knowledge base, which cannot support the retrieval and use of large-scale heterogeneous multimodal knowledge. In summary, existing technologies have not yet solved the problem of efficient utilization of heterogeneous multimodal information by large models. There is an urgent need for a large model knowledge enhancement generation method that can achieve unified retrieval of heterogeneous multimodal data, cross-modal semantic alignment, and deep integration with the RAG generation process, in order to improve the ability of large models to understand and generate complex multimodal information from the real world. Summary of the Invention

[0006] This application provides a knowledge enhancement method and apparatus that integrates heterogeneous multimodal retrieval and cross-modal alignment. First, an initial feature set is formed by calculating cosine similarity. Then, a pre-trained multimodal semantic similarity score calculation model is used to reorder the initial feature set. This allows for the filtering of globally similar but locally mismatched noise segments through fine-grained semantic matching, thereby significantly improving the semantic relevance of the retrieval results.

[0007] In a first aspect, embodiments of this application provide a data import method, the method comprising:

[0008] Construct a knowledge base containing data of different modalities, extract features from the data of each modality in the knowledge base, and map the extracted data features to the embedding space; Based on user queries, query features are obtained, and M data features in the embedding space with cosine similarity to the query features from high to low are obtained to form an initial data feature set; The semantic similarity score between each data feature in the initial data feature set and the query feature is calculated using a pre-trained multimodal semantic similarity score calculation model. The initial data feature set is then reordered from high to low based on the semantic similarity scores to obtain the final data feature set. The multimodal semantic similarity score calculation model uses encoders corresponding to different modalities to extract features from the data features of different modalities in the initial data feature set. In the final data feature set, K data features with semantic similarity from high to low with the query features are obtained as semantic similarity features. All semantic similarity features are fused to obtain contextual hints, and the user query is enhanced with the contextual hints.

[0009] Secondly, embodiments of this application provide a knowledge enhancement device that integrates heterogeneous multimodal retrieval and cross-modal alignment, comprising: The construction module is used to build a knowledge base containing data of different modalities, extract features from the data of each modality in the knowledge base and map the extracted data features to the embedding space; The similarity calculation module obtains query features based on user queries and acquires M data features in the embedding space with cosine similarity to the query features from high to low to form an initial data feature set; The reordering module uses a pre-trained multimodal semantic similarity score calculation model to calculate the semantic similarity score between each data feature in the initial data feature set and the query feature. Based on the semantic similarity score, the initial data feature set is reordered from high to low to obtain the final data feature set. The multimodal semantic similarity calculation model uses encoders corresponding to different modalities to extract features from the data features of different modalities in the initial data feature set. The knowledge enhancement module is used to obtain K data features with high to low semantic similarity to the query features from the final data feature set as semantic similarity features, fuse all semantic similarity features to obtain contextual hints, and use the contextual hints to enhance the user query.

[0010] Thirdly, embodiments of this application provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform a knowledge enhancement method that integrates heterogeneous multimodal retrieval and cross-modal alignment.

[0011] Fourthly, embodiments of this application provide a readable storage medium storing a computer program, which, when executed by a processor, implements a knowledge enhancement method that integrates heterogeneous multimodal retrieval and cross-modal alignment.

[0012] The main contributions and innovations of this invention are as follows: This application embodiment extracts multimodal features through corresponding encoding modules and maps them to the same embedding space. Simultaneously, it jointly trains the encoding modules and constructs a cross-modal contrastive loss function to achieve cross-modal alignment, breaking the limitations of traditional RAG single-modal knowledge utilization, unifying the feature representation space of multimodal data, and achieving accurate semantic matching of different modal data. This application embodiment performs modality recognition on user queries. Single-modal queries directly extract features using the corresponding encoder, while multimodal queries are segmented, sub-features are extracted, and then fused to obtain query features, accurately adapting to single / multimodal user query scenarios and ensuring the targeting and accuracy of query feature extraction. This application embodiment first uses the remaining... The initial feature set is composed of string similarity, and then the pre-trained multimodal semantic similarity score calculation model is used to re-rank them. This allows for the filtering of globally similar but locally mismatched noise segments through fine-grained semantic matching, significantly improving the semantic relevance of the retrieval results. The embodiments of this application construct a multimodal semantic similarity score calculation model based on the Transformer architecture, integrating sequence encoding, cross-modal interaction, and semantic scoring units. Cross-modal interaction fusion is achieved through self-attention + cross-attention, effectively capturing the fine-grained local alignment relationship between the query and candidate features, solving the problem of local information loss in global feature matching, and achieving more accurate cross-modal semantic matching.

[0013] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0014] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a knowledge enhancement method that integrates heterogeneous multimodal retrieval and cross-modal alignment according to an embodiment of this application; Figure 2 This is a structural block diagram of a knowledge enhancement device that integrates heterogeneous multimodal retrieval and cross-modal alignment according to an embodiment of this application; Figure 3 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0015] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0016] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0017] Example 1 This application provides a knowledge enhancement method that integrates heterogeneous multimodal retrieval and cross-modal alignment. First, an initial feature set is constructed using cosine similarity. Then, a pre-trained multimodal semantic similarity score calculation model is used to re-rank the features. This allows for fine-grained semantic matching to filter out globally similar but locally mismatched noise segments, significantly improving the semantic relevance of the retrieval results. Specifically, refer to... Figure 1 The method includes: Construct a knowledge base containing data of different modalities, extract features from the data of each modality in the knowledge base, and map the extracted data features to the embedding space; Based on user queries, query features are obtained, and M data features in the embedding space with cosine similarity to the query features from high to low are obtained to form an initial data feature set; The semantic similarity score between each data feature in the initial data feature set and the query feature is calculated using a pre-trained multimodal semantic similarity score calculation model. The initial data feature set is then reordered from high to low based on the semantic similarity scores to obtain the final data feature set. The multimodal semantic similarity score calculation model uses encoders corresponding to different modalities to extract features from the data features of different modalities in the initial data feature set. In the final data feature set, K data features with semantic similarity from high to low with the query features are obtained as semantic similarity features. All semantic similarity features are fused to obtain contextual hints, and the user query is enhanced with the contextual hints.

[0018] In the current embodiment, data of multiple modalities, including text, images, audio, and video, are obtained from a public dataset and stored in a knowledge base. Multiple data units are constructed in the knowledge base, and each data unit stores a single modal data type or stores a multimodal data type.

[0019] Specifically, before saving data of multiple modalities to the knowledge base, the data is preprocessed by cleaning, slicing, and labeling to ensure data quality.

[0020] Specifically, each data unit in the knowledge base can be used to store single-modal data types, such as plain text or plain images, or it can be used to store multimodal data types, such as videos with subtitles or articles with pictures and text. In the knowledge base, each data unit is stored as an independent entry, and information such as the modality type, original content, and metadata stored in each data unit is recorded.

[0021] For example, a knowledge base is built to store information about various tourist attractions and to answer users' questions about them. Therefore, data is first collected from channels such as tourism websites, video platforms, and social media, including multimodal data such as text introducing the attractions, photos of the attractions, audio recordings of tour guides, and promotional videos of the attractions. After cleaning and removing irrelevant noise, this multimodal data is stored in the knowledge base as independent data units.

[0022] In the current embodiment, multiple pre-trained encoding modules corresponding to different modalities are used to extract features from the data of the corresponding modalities in the knowledge base and map them to the embedding space. Each encoding module includes an encoder and a linear projection layer. The encoder is used to extract the feature representation of the data of the corresponding modality, and the linear projection layer is used to map the extracted feature representation to the embedding space.

[0023] Specifically, for text data, encoders such as BERT and RoBERTa are used for feature extraction; for image data, encoders such as ViT and ResNet are used for feature extraction; for audio data, encoders such as Wav2Vec 2.0 and HuBERT are used for feature extraction; and for video data, encoders such as VideoBERT and TimeSformer are used for feature extraction.

[0024] Specifically, the linear projection layer in each encoding module maps the feature representations of different modalities to the same embedding space by unifying the dimensions of the extracted feature representations. In this scheme, the linear projection layer projects the data features to 512 dimensions.

[0025] Furthermore, multiple coding modules are jointly trained. During the joint training process, a cosine similarity matrix is ​​constructed for the data features of any two different modalities in the embedding space. Based on the cosine similarity matrix, a cross-modal contrastive loss function is constructed by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. The weighted sum of the cross-modal contrastive loss functions corresponding to the feature representations of all cross-modalities is used as the total loss function. The parameters of each coding module are jointly adjusted with the goal of minimizing the total loss function to complete the joint training.

[0026] Specifically, the multiple encoding modules obtained through joint training can ensure that similar data features are close together in the embedding space, while dissimilar data features are far apart, thereby completing the data alignment of different modalities.

[0027] Specifically, batch training is used to jointly train multiple coding modules. Positive sample pairs during training are pre-labeled similar data features, while the remaining sample pairs are negative sample pairs.

[0028] For example, the cross-modal contrastive loss function for image and text modalities is expressed as follows:

[0029] in, Let be the contrast loss function for image and text modalities. , For image modal data features, , For text modal data features, For cosine similarity calculation, Let i be the total number of image-text pairs, and j be the index values. The temperature coefficient is 0.07 in this scheme.

[0030] Similarly, we construct cross-modal contrastive loss functions for other cross-modal feature representations, such as video-text and audio-text, to obtain the total loss function:

[0031] in, For the total loss function, Let be the contrast loss function for image and text modalities. Let be the contrastive loss function for audio and text modalities. Let be the contrastive loss function for video and text modalities. , , These are the corresponding weighting coefficients.

[0032] Specifically, this scheme achieves cross-modal alignment of data features output by encoding modules of different modalities within the same semantic space through continuous optimization of the total loss function.

[0033] In the current embodiment, a data index is constructed for each piece of data in the knowledge base and saved to the index library. Data features within the embedding space are obtained based on the data index and cosine similarity is calculated with the query features. The feature index contains the spatial location of the corresponding data features in the embedding space.

[0034] Specifically, the index is constructed based on FAISS. In addition to the spatial location of the corresponding data feature in the embedding space, the feature index also includes the modality type, data ID, storage path, etc. of each data feature.

[0035] In some other embodiments, IVF, HNSW, etc. are used to build the index database in order to balance retrieval speed and accuracy.

[0036] Specifically, by constructing an index to indicate the spatial location of each data feature within the embedding space, the acquisition of the corresponding data feature is accelerated, thereby speeding up the calculation of the cosine similarity between the query feature and each data feature.

[0037] In the current embodiment, after obtaining the user query, modality recognition is performed on the user query. If the user query is unimodal, the corresponding encoder is used to extract features from the user query to obtain query features. If the user query is multimodal, the user query is modally segmented to obtain multiple user subqueries of different modalities. The corresponding encoders of different modalities are used to extract features from the user subqueries to obtain multiple user sub-features. Then, all user sub-features are fused to obtain query features.

[0038] Specifically, all user sub-features are fused using average pooling or attention fusion.

[0039] Specifically, the encoder used for feature extraction of user queries is a pre-trained encoder structure.

[0040] For example, if the user enters a query such as "What are the famous palaces in the Forbidden City? Please attach a picture", then a text encoder is used to extract features from the user query to obtain user features; if the user enters a query such as "Where is this picture?" and attaches an image of the Forbidden City, a text encoder is used to extract features from the text, an image encoder is used to extract features from the image, and the text features and image features are fused together as query features.

[0041] In the current embodiment, the cosine similarity between the query feature and each data feature in the embedding space is calculated in a traversal manner, expressed by the formula:

[0042] in, To query features, For the i-th data feature in the embedding space, This is for calculating cosine similarity.

[0043] Then, an initial data feature set composed of multiple data features with cosine similarity from high to low is returned as the preliminary search results.

[0044] For example, the search returns 100 search results that are similar to the query features by calculating cosine similarity. These search results include data features of multiple modalities such as text, images, audio, and video, such as introductory texts about the Forbidden City, pictures of the Hall of Supreme Harmony, audio clips of tour guide explanations, and promotional video clips.

[0045] In this scheme, the initial data feature set obtained through cosine similarity calculation depends on the distance between data features in the embedding space. Although cross-modal alignment ensures the comparability of features from different modalities in the semantic space, the similarity calculation of global features is essentially a coarse-grained matching, which struggles to capture the fine-grained local alignment relationship between the query and candidate fragments. For example, when a user query is "a white cat sleeping on the sofa," the image may contain multiple objects such as the cat and the sofa. Global features may mismatch irrelevant images due to background similarity, while fine-grained interaction information, such as the cat's posture, color, and relative position to the sofa, is lost due to average pooling operations. Furthermore, semantic interactions between different modalities require cross-modal attention mechanisms to achieve accurate alignment.

[0046] Therefore, in order to improve the accuracy of cross-modal matching, this solution reorders the initial data feature set and user query based on semantic similarity, in order to select data features that are completely matched with the user query in terms of both local and global features for knowledge enhancement.

[0047] Specifically, a pre-trained multimodal semantic similarity score calculation model is constructed based on the Transformer architecture. The multimodal semantic similarity score calculation model includes a sequence encoding unit, a cross-modal interaction unit, and a semantic relevance scoring unit. Query features and features to be matched are input into the semantic matching model. The sequence encoding unit encodes the query features and features to be matched into query feature sequences and features to be matched sequences, respectively. The cross-modal interaction unit performs cross-modal interaction and fusion of the query feature sequences and features to be matched sequences based on self-attention and cross-attention to obtain a cross-modal fused feature sequence. The semantic relevance scoring unit maps the cross-modal fused feature sequence into a semantic similarity score through a multilayer perceptron. The features to be matched are any features in the initial data feature set.

[0048] In this scheme, the sequence encoding unit includes sequence encoders corresponding to multiple modalities. Based on the query features and the modality of the features to be matched, the corresponding sequence encoder is selected to perform sequence encoding to obtain the query feature sequence and the feature sequence to be matched. For example, the BERT model is used to perform sequence encoding on the text modal features to obtain a text sequence encoding result containing a series of word vectors; ViT is used to decompose the image modal features into multiple fixed-size image blocks, and after linear projection, the image sequence encoding result is obtained; VideoBERT is used to extract video modal features and perform spatiotemporal feature extraction to obtain the video sequence encoding result; Wav2Vec 2.0 is used to extract the acoustic feature sequence of the audio modal features to obtain the audio sequence encoding result.

[0049] The cross-modal interaction unit consists of multiple Transformers. Each Transformer layer contains self-attention and cross-attention. Through self-attention and cross-attention, the text token can focus on the local regions in the visual token that are semantically related to it. Thus, the cross-modal fusion feature sequence is obtained through the interaction of multiple Transformers. In this scheme, in order to ensure the lightweight deployment of the multimodal semantic similarity score calculation model, the number of Transformer layers is limited to 6.

[0050] In the semantic relevance scoring unit, the cross-modal fused feature sequence is averaged to obtain a global fused vector, and the multilayer perceptron maps the global fused vector to a semantic similarity score.

[0051] The multimodal semantic similarity score calculation model is pre-trained on a publicly available multimodal dataset and employs an objective function that combines contrastive learning and matching loss. It utilizes batch inference and model quantization techniques to achieve a throughput of hundreds of candidate segments per second on a single GPU, meeting the needs of practical applications. The contrastive loss is used to bring matching sample pairs closer together and push away mismatched sample pairs. The matching loss is a binary cross-entropy loss. The query and the segment are concatenated and input into the model to predict whether they match, thereby encouraging the model to learn fine-grained semantic alignment.

[0052] In the current embodiment, reordering the initial data feature set can effectively filter out noise segments that are globally similar but locally mismatched, significantly improving the semantic relevance of the retrieval results. Experiments show that on multiple multimodal question-answering datasets, after introducing reordering, the Recall@K index is improved by an average of 5% to 8%, and the user satisfaction of the generated answers is improved by more than 12%.

[0053] In the current embodiment, if the knowledge enhancement result is used for the first large language model, the token representation of each semantically similar feature is concatenated to obtain a contextual prompt; if the knowledge enhancement result is used for the second large language model, each semantically similar feature is converted into a text token representation, and the text token representation of each semantically similar feature is concatenated to obtain a contextual prompt. The first large language model supports multimodal input, while the second large language model only supports text input.

[0054] For example, when the knowledge augmentation results are used in the second language model, BLIP-2 is used to convert the semantic similarity features of the image modality into corresponding text tokens, Whisper is used to convert the semantic similarity features of the audio modality into corresponding text tokens, keyframes of the semantic similarity features of the video modality are extracted, and BLIP-2 is used to convert the keyframe images into corresponding text tokens.

[0055] In other embodiments, user feedback on the generated results can be recorded each time, and the training strategy of the multimodal semantic similarity score calculation model can be optimized using the feedback results, such as adjusting the weights of the re-ranking model through reinforcement learning, or updating the parameters of the feature extractor through online learning.

[0056] For example, user actions such as clicking and liking answers are recorded. If a user likes an image, the system can increase the weight of that image segment for fine-tuning of the subsequent retrieval re-ranking model.

[0057] Example 2 Based on the same concept, referencing Figure 2 This application also proposes a knowledge enhancement device that integrates heterogeneous multimodal retrieval and cross-modal alignment, comprising: The construction module is used to build a knowledge base containing data of different modalities, extract features from the data of each modality in the knowledge base and map the extracted data features to the embedding space; The similarity calculation module obtains query features based on user queries and acquires M data features in the embedding space with cosine similarity to the query features from high to low to form an initial data feature set; The reordering module uses a pre-trained multimodal semantic similarity score calculation model to calculate the semantic similarity score between each data feature in the initial data feature set and the query feature. Based on the semantic similarity score, the initial data feature set is reordered from high to low to obtain the final data feature set. The multimodal semantic similarity calculation model uses encoders corresponding to different modalities to extract features from the data features of different modalities in the initial data feature set. The knowledge enhancement module is used to obtain K data features with high to low semantic similarity to the query features from the final data feature set as semantic similarity features, fuse all semantic similarity features to obtain contextual hints, and use the contextual hints to enhance the user query.

[0058] Example 3 This embodiment also provides an electronic device, see reference. Figure 3 It includes a memory 404 and a processor 402, the memory 404 storing a computer program and the processor 402 being configured to run the computer program to perform the steps in any of the above method embodiments.

[0059] Specifically, the processor 402 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0060] Memory 404 may include a mass storage device for data or instructions. For example, and not limitingly, memory 404 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 404 may include removable or non-removable (or fixed) media. Where appropriate, memory 404 may be internal or external to a data processing device. In a particular embodiment, memory 404 is non-volatile memory. In a particular embodiment, memory 404 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0061] The memory 404 can be used to store or cache various data files that need to be processed and / or communicated, as well as possible computer program instructions executed by the processor 402.

[0062] The processor 402 reads and executes computer program instructions stored in the memory 404 to implement any of the knowledge enhancement methods in the above embodiments that integrate heterogeneous multimodal retrieval and cross-modal alignment.

[0063] Optionally, the electronic device may further include a transmission device 406 and an input / output device 408, wherein the transmission device 406 is connected to the processor 402, and the input / output device 408 is connected to the processor 402.

[0064] Transmission device 406 can be used to receive or send data via a network. Specific examples of the network described above may include wired or wireless networks provided by the communication provider of the electronic device. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, transmission device 406 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0065] Input / output device 408 is used to input or output information. In this embodiment, the input information may be data features of different modalities, user queries, etc., and the output information may be contextual prompts, knowledge enhancement results, etc.

[0066] Optionally, in this embodiment, the processor 402 can be configured to perform the following steps via a computer program: Construct a knowledge base containing data of different modalities, extract features from the data of each modality in the knowledge base, and map the extracted data features to the embedding space; Based on user queries, query features are obtained, and M data features in the embedding space with cosine similarity to the query features from high to low are obtained to form an initial data feature set; The semantic similarity score between each data feature in the initial data feature set and the query feature is calculated using a pre-trained multimodal semantic similarity score calculation model. The initial data feature set is then reordered from high to low based on the semantic similarity scores to obtain the final data feature set. The multimodal semantic similarity score calculation model uses encoders corresponding to different modalities to extract features from the data features of different modalities in the initial data feature set. In the final data feature set, K data features with semantic similarity from high to low with the query features are obtained as semantic similarity features. All semantic similarity features are fused to obtain contextual hints, and the user query is enhanced with the contextual hints.

[0067] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0068] Generally, various embodiments can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects of the invention can be implemented in hardware, while others can be implemented by firmware or software executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. Although various aspects of the invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that, by way of non-limiting example, these blocks, apparatuses, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0069] Embodiments of the present invention can be implemented by computer software, which may be executable by a data processor of a mobile device, such as a processor entity, or by hardware, or by a combination of software and hardware. Computer software or programs (also referred to as program products) including software routines, applets, and / or macros can be stored in any device-readable data storage medium, and they include program instructions for performing specific tasks. The computer program product may include one or more computer-executable components configured to perform the embodiments when the program is run. The one or more computer-executable components may be at least one piece of software code or a portion thereof. Additionally, it should be noted in this respect that, as Figure 3 Any box in the logical flow can represent a program step, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. Software can be stored on physical media such as memory chips or blocks of storage implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as DVDs and their data variants, CDs, etc. The physical medium is a non-transient medium.

[0070] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0071] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A knowledge enhancement method integrating heterogeneous multimodal retrieval and cross-modal alignment, characterized in that, Includes the following steps: Construct a knowledge base containing data of different modalities, extract features from the data of each modality in the knowledge base, and map the extracted data features to the embedding space; Based on user queries, query features are obtained, and M data features in the embedding space with cosine similarity to the query features from high to low are obtained to form an initial data feature set; The semantic similarity score between each data feature in the initial data feature set and the query feature is calculated using a pre-trained multimodal semantic similarity score calculation model. The initial data feature set is then reordered from high to low based on the semantic similarity scores to obtain the final data feature set. The multimodal semantic similarity score calculation model uses encoders corresponding to different modalities to extract features from the data features of different modalities in the initial data feature set. In the final data feature set, K data features with semantic similarity from high to low with the query features are obtained as semantic similarity features. All semantic similarity features are fused to obtain contextual hints, and the user query is enhanced with the contextual hints.

2. The knowledge enhancement method integrating heterogeneous multimodal retrieval and cross-modal alignment according to claim 1, characterized in that, Multiple pre-trained encoding modules corresponding to different modalities are used to extract features from the data of the corresponding modalities in the knowledge base and map them to the embedding space. Each encoding module includes an encoder and a linear projection layer. The encoder is used to extract the feature representation of the data of the corresponding modality, and the linear projection layer is used to map the extracted feature representation to the embedding space.

3. The knowledge enhancement method integrating heterogeneous multimodal retrieval and cross-modal alignment according to claim 2, characterized in that, Multiple encoding modules are jointly trained. During the joint training process, a cosine similarity matrix is ​​constructed in the embedding space using the feature representations of any two different modalities. Based on the cosine similarity matrix, a cross-modal contrastive loss function is constructed by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. The total loss function is obtained by weighting the cross-modal contrastive loss functions corresponding to all cross-modal feature representations. The parameters of each encoding module are jointly adjusted with the goal of minimizing the total loss function to complete the joint training.

4. The knowledge enhancement method integrating heterogeneous multimodal retrieval and cross-modal alignment according to claim 1, characterized in that, A data index is constructed for each piece of data in the knowledge base and saved to the index library. Data features within the embedding space are obtained based on the data index and cosine similarity is calculated with the query features. The feature index contains the spatial location of the corresponding data feature in the embedding space.

5. The knowledge enhancement method integrating heterogeneous multimodal retrieval and cross-modal alignment according to claim 1, characterized in that, After obtaining the user query, modality recognition is performed on the user query. If the user query is unimodal, the corresponding modality encoder is used to extract the query features. If the user query is multimodal, the user query is modally segmented to obtain multiple user subqueries of different modalities. The encoders corresponding to different modalities are used to extract features from the user subqueries to obtain multiple user sub-features. Then, all user sub-features are fused to obtain the query features.

6. The knowledge enhancement method integrating heterogeneous multimodal retrieval and cross-modal alignment according to claim 1, characterized in that, A pre-trained multimodal semantic similarity score calculation model is constructed based on the Transformer architecture. The multimodal semantic similarity score calculation model includes a sequence encoding unit, a cross-modal interaction unit, and a semantic relevance scoring unit. Query features and features to be matched are input into the semantic matching model. The sequence encoding unit encodes the query features and features to be matched into query feature sequences and features to be matched sequences, respectively. The cross-modal interaction unit performs cross-modal interaction and fusion of the query feature sequences and features to be matched sequences based on self-attention and cross-attention to obtain a cross-modal fused feature sequence. The semantic relevance scoring unit maps the cross-modal fused feature sequence into a semantic similarity score through a multilayer perceptron. The features to be matched are any features in the initial data feature set.

7. The knowledge enhancement method integrating heterogeneous multimodal retrieval and cross-modal alignment according to claim 1, characterized in that, If the knowledge augmentation results are used in the first language model, the token representations of each semantically similar feature are concatenated to obtain contextual hints. If the knowledge augmentation results are used in the second language model, each semantic similarity feature is converted into a text token representation, and the text token representations of each semantic similarity feature are concatenated to obtain contextual hints. The first language model supports multimodal input, while the second language model only supports text input.

8. A knowledge enhancement device integrating heterogeneous multimodal retrieval and cross-modal alignment, characterized in that, include: The construction module is used to build a knowledge base containing data of different modalities, extract features from the data of each modality in the knowledge base and map the extracted data features to the embedding space; The similarity calculation module obtains query features based on user queries and acquires M data features in the embedding space with cosine similarity to the query features from high to low to form an initial data feature set; The reordering module uses a pre-trained multimodal semantic similarity score calculation model to calculate the semantic similarity score between each data feature in the initial data feature set and the query feature. Based on the semantic similarity score, the initial data feature set is reordered from high to low to obtain the final data feature set. The multimodal semantic similarity calculation model uses encoders corresponding to different modalities to extract features from the data features of different modalities in the initial data feature set. The knowledge enhancement module is used to obtain K data features with high to low semantic similarity to the query features from the final data feature set as semantic similarity features, fuse all semantic similarity features to obtain contextual hints, and use the contextual hints to enhance the user query.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to execute a knowledge enhancement method according to any one of claims 1-7 that integrates heterogeneous multimodal retrieval and cross-modal alignment.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements a knowledge enhancement method as described in any one of claims 1-7, which integrates heterogeneous multimodal retrieval and cross-modal alignment.

Images