Cross-modal retrieval method for semantic and vector fusion in data space

Abstract

The invention provides a cross-modal retrieval method for semantic and vector fusion in a data space, which belongs to the field of cross-modal information retrieval, and comprises the following steps: firstly, collecting and preprocessing multi-modal data; generating modal embedding and storing by utilizing the pre-training model; a shared semantic space is constructed, cross-modal vector alignment is optimized through comparative learning, and a modal mapping network is designed to enhance the embedding projection effect; storing the aligned embedding by using a Milvus database, and constructing an HNSW index; user text or image query is processed, text query analyzes limiting conditions to generate enhanced embedding, and image query extracts characters through OCR and fuses the characters with image features to generate embedding; in a database, through condition screening and semantic similarity calculation, a Top-K candidate item is retrieved; performing multi-modal correlation sorting on the candidate results and returning the results; according to the method, the shared semantic space is constructed, the alignment effect of different modal embedding is optimized, efficient storage and index management of multi-modal embedding are carried out, and real-time retrieval of large-scale cross-modal data is achieved.

Description

Technical Field

[0001] The present invention belongs to the field of cross-modal information retrieval, and more specifically, relates to a cross-modal retrieval method for the fusion of semantics and vectors in a data space. Background Art

[0002] With the rapid development of the big data era, user data files exhibit the characteristics of multi-source heterogeneity and rich semantics. On the one hand, users need to manage multi-modal data, including text, images, videos, and audio, etc. However, existing technologies are difficult to support cross-modal retrieval, resulting in low retrieval efficiency. On the other hand, due to the diversity of semantic expressions, the same semantics may be presented in different forms, and traditional keyword matching methods are difficult to deeply understand complex semantics, leading to low matching accuracy in retrieval. Therefore, heterogeneous multi-modal information processing technology is increasingly becoming a research and application hotspot. To build an efficient operating environment, the industry has put forward higher requirements for heterogeneous multi-modal information retrieval technology, especially while improving retrieval efficiency, more attention is paid to the improvement of accuracy. The current research focus is to deeply innovate and optimize these retrieval technologies to cope with the increasingly complex challenges of information processing, reduce redundant work, and improve the overall system efficiency.

[0003] With the gradual in-depth study of multi-modal data processing technology, researchers are increasingly concerned about how to map heterogeneous data into a common data space for processing. The data space refers to a method of embedding different modalities of data (such as text, images, videos, etc.) into a unified semantic space, so that these essentially different data can be compared and operated in the same coordinate system. In cross-modal retrieval, the construction and optimization of the data space play a crucial role. By constructing an efficient and consistent data space, different modalities of data can form meaningful representations in this space, enabling cross-modal retrieval to more accurately calculate the semantic similarity and matching degree between different modalities.

[0004] When dealing with the challenges of heterogeneous multi-modal information retrieval, cross-modal information retrieval technology allows users to retrieve relevant information in another modality with the same or similar semantics through semantic alignment between different modalities by inputting data in one modality (such as text, image, etc.). For example, users can retrieve relevant pictures by inputting a text description, or find matching text descriptions or video content by uploading a picture. The core goal of cross-modal retrieval is to retrieve information in another modality corresponding to it at the semantic level under the query conditions of a given modality through the calculation of cross-modal semantic similarity. Specifically, cross-modal retrieval includes multiple subtasks, such as image-text cross-modal retrieval, video-text cross-modal retrieval, etc., and its classification criteria are usually based on the types of input and output modalities. Due to the heterogeneity of multi-modal data in features, different modalities of data need to adopt specialized feature representation methods, which makes the calculation of cross-modal semantic similarity a technical problem. Therefore, the key to cross-modal retrieval lies in how to effectively represent the data features of different modalities and achieve accurate matching of features between different modalities.

[0005] Feature representation is a fundamental step in cross-modal retrieval. Its goal is to map heterogeneous modal data into a shared semantic space to achieve semantic alignment between different modalities. For modality-specific feature representations, the text modality typically uses pre-trained models (such as BERT and GPT) to capture contextual semantics through large-scale corpus training and generate high-dimensional embedding vectors. In the image modality, deep convolutional neural networks (such as ResNet and VGG) are widely used to extract global and local features. Multi-modal pre-trained models (such as CLIP) optimize the semantic alignment ability between images and text in the shared semantic space through contrastive learning. Feature representation in the audio modality relies on the combination of spectral analysis and deep learning. Traditional methods such as MFCC and Chroma features extract the time-frequency information of audio, while deep models such as WaveNet and OpenL3 significantly improve the quality of audio semantic embedding through end-to-end feature learning. Time series modeling based on Transformer and LSTM further optimizes the semantic capture of long-term audio. The feature representation in the video modality is more complex because it involves both spatial and temporal information simultaneously. 3D-CNN and VideoTransformer models capture the spatio-temporal features of videos by combining convolutional networks with attention mechanisms. I3D and SlowFast networks optimize the expression of dynamic characteristics through multi-scale modeling. Combining multi-modal pre-trained frameworks (such as MERLOT Reserve and VideoCLIP), the video modality can achieve precise alignment with other modalities. The construction of the shared semantic space is the core challenge in cross-modal feature representation. Researchers have proposed various methods: projection based on linear transformation (such as the research of Frome et al.) and non-linear mapping based on deep adversarial learning (such as the method of Goodfellow et al.). However, modality heterogeneity, semantic gap, and data sparsity are still the main problems in feature representation. To address these issues, self-supervised learning (such as the method of Chen) further improves the robustness and generalization ability of feature representation by mining semantic features in unlabeled data. Although significant progress has been made in the feature representation technology of cross-modal retrieval, its application still faces many challenges. Modality heterogeneity leads to differences in feature distribution, representation form, and semantic structure among different modalities, making unified modeling complex. At the same time, the semantic gap between modalities makes it difficult to capture cross-modal semantic consistency. Data sparsity and noise problems further increase the challenges of model generalization and robustness in real-world scenarios.

[0006] After completing the feature representation of multimodal data, the next step is to match the features of different modalities. By calculating the semantic similarity between modalities, the similarity of cross-modal data is ranked to achieve cross-modal retrieval. Feature matching methods are generally divided into two types: global feature matching and local feature matching, each with its application scenarios and implementation strategies. The goal of global feature matching methods is to embed the entire image and text into a shared semantic space and calculate the cross-modal similarity through simple similarity metrics. Early global feature matching methods mainly relied on global feature representation networks with a two-tower structure, such as the DeViSE model, which used CNN and Skip-Gram techniques to project images and text into a unified embedding space to achieve global feature matching. This method has certain limitations and is difficult to capture fine-grained information in images and text, resulting in insufficient feature expression ability. To address this issue, subsequent research focused on improving the network architecture and enhancing the robustness of feature matching through more complex pre-training techniques. In recent years, significant progress has been made in the application of large-scale pre-training strategies in global feature matching. With the help of fine-grained fine-tuning techniques, these methods have achieved breakthroughs in model performance and cross-modal retrieval accuracy. Global feature matching methods still face many challenges in practical applications. For example, the similarity distribution bias of embedding vectors in retrieval tasks may lead to intra-class differences and inter-class feature ambiguity, thus limiting the generalization ability of the model. To solve these problems, researchers have proposed methods to suppress intra-class differences and inter-class distribution biases, thereby enhancing the model's representation ability for cross-modal data. Addressing the semantic differences and attention distribution differences between modalities is one of the key directions for future research. By constructing a more powerful semantic alignment mechanism, the performance of global feature matching can be further improved. Local feature matching methods are an important strategy for solving the fine-grained alignment problem in cross-modal retrieval. This method matches the local regions of images with words or phrases in text, making up for the deficiencies of global feature matching methods in capturing local details and semantic associations. Researchers have proposed various methods to achieve local feature matching. Early methods focused on selecting relevant local feature regions to reduce computational complexity and improve matching efficiency. For example, researchers such as Liu achieved more accurate matching by selectively aligning local semantics. Some studies have proposed local matching methods based on hierarchical relationship alignment networks, which can improve the accuracy of cross-modal retrieval through semantic alignment strategies from local to global or global to local. Further research uses graph structure representations of semantic relationships between modalities to capture the interactions between image regions and text vocabulary, while strengthening the alignment consistency of fine-grained semantic information in visual and language modalities. However, the challenges faced by local feature matching methods in practical applications include high computational costs and the impact of intra-class semantic differences on model performance. To solve these problems, researchers have tried to enhance the generalization ability of the model by suppressing intra-class differences and optimizing the model's attention distribution.Although local feature matching methods have significantly improved the fine-grained alignment ability of cross-modal retrieval, in practical scenarios, their computational resource requirements are relatively high, especially their applications on large-scale cross-modal data may be limited. Therefore, the development of cross-modal retrieval will focus on enhancing the semantic consistency and generalization ability of feature representations and seeking a balance between global and local feature matching. By introducing a fusion design of semantic parsing, query embedding, and modality alignment mechanisms, the development of cross-modal file information retrieval technology will be further promoted. Summary of the Invention

[0007] For the above reasons, the present invention proposes a cross-modal retrieval method that fuses semantics and vectors in the data space, aiming to solve the problems in existing cross-modal file information retrieval technologies, such as insufficient handling of the heterogeneity of multimodal data, limited semantic understanding ability, and difficulty in balancing retrieval efficiency and accuracy. By integrating semantic parsing and query embedding technologies, this method can uniformly index multimodal data, generate vector representations in a shared semantic space, and design an intelligent file search method, thereby achieving efficient and accurate cross-modal retrieval.

[0008] Cross-modal retrieval method that fuses semantics and vectors in the data space: The method specifically includes the following steps:

[0009] Step 1: Collect multimodal data and perform data preprocessing;

[0010] Step 2: Generate the embedding representation of each modality through a pre-trained model and save it in a vector database;

[0011] Step 3: Map the vectors of different modalities to a shared semantic space to ensure that they can be queried and compared in a unified manner;

[0012] Step 4: Process the user query and generate an embedding representation. For text queries and image queries, combine natural language processing technology and OCR technology to enhance the semantic vectorization effect;

[0013] Step 5: Perform semantic similarity calculation and candidate retrieval;

[0014] Step 6: Display cross-modal relevance ranking and multimodal retrieval results.

[0015] Furthermore, in Step 1, it includes

[0016] Step 1.1: Select appropriate data sources and obtain text, image, video, or audio data through crawler tools, data collection, or open platforms;

[0017] Step 1.2: Perform text cleaning, word segmentation, deduplication, and standardization processing on the text data for subsequent model training;

[0018] Step 1.3: For image data, uniformly adjust the image size according to the model requirements, perform image pixel normalization processing, and perform data augmentation operations;

[0019] Step 1.4: For video data, decompose the video into frames and select important key frames for processing;

[0020] Step 1.5: For audio data, remove noise from the audio signal, clean the background noise in the audio data through a filtering denoising algorithm, and improve the quality of the speech signal.

[0021] Furthermore, in Step 2, it includes

[0022] Step 2.1: For text data, input the preprocessed text data into the Sentence-BERT model, output a 768-dimensional text embedding vector, generate a semantic representation of the text, and save it in the vector database;

[0023] Step 2.2: For image data, input the preprocessed image data into the CLIP model, and use the ViT-Transformer architecture of CLIP to convert the image into a 512-dimensional semantic vector to represent the semantic features of the image, and save it in the vector database;

[0024] Step 2.3: For video data, input the video data into the VideoCLIP model, generate a video embedding by extracting the key frames of the video and using temporal information, and save the obtained 512-dimensional semantic vector in the vector database;

[0025] Step 2.4: For audio data, input the preprocessed audio data into the OpenL3 model, and use its deep neural network to convert the audio signal into a 512-dimensional semantic embedding representation.

[0026] Furthermore, in Step 3, it includes

[0027] Step 3.1: Define the shared semantic space and its construction method to provide a unified alignment benchmark for the embeddings of all modalities; select the general cross-modal embedding CLIP model, and use the contrastive learning strategy to train the shared space based on the aligned semantic labels and feature data;

[0028] For the text modality, directly use the Sentence-BERT embedding as the input to the shared space;

[0029] For the image, video, and audio modalities, use their respective modality embeddings as the input;

[0030] Step 3.2: On the basis of the shared semantic space, design a modality mapping network to project the embeddings of different modalities into the shared semantic space;

[0031] Step 3.3: Store the aligned multi-modal embeddings in the Milvus database and build an indexing mechanism.

[0032] Furthermore, in Step 3.2, the specific steps of designing the modal mapping network to project embeddings of different modalities into a shared semantic space are as follows:

[0033] Step 3.2.1: Design the text modal mapping network: Use a multi-layer perceptron (MLP) or direct projection.

[0034] Step 3.2.2: Design the image modal mapping network: Add a small Transformer model to capture image context features.

[0035] Step 3.2.3: Design the video modal mapping network: Combine temporal LSTM or Transformer to enhance the alignment of temporal features.

[0036] Step 3.2.4: Design the audio modal mapping network: Use a convolutional layer + LSTM layer to extract time and frequency features.

[0037] Step 3.2.5: Gradually debug the network structure parameters to ensure the projection quality of each modality in the shared semantic space.

[0038] Furthermore, in Step 3.3, the specific steps of building the indexing mechanism are as follows:

[0039] Step 3.3.1: Create a vector collection (Collection) in Milvus to store the aligned multi-modal embeddings.

[0040] Step 3.3.2: Formulate each aligned embedding, normalize all vectors to ensure the retrieval performance of the embedding vectors, and store them in the vector database.

[0041] Step 3.3.3: Build an HNSW index for the stored vector fields.

[0042] Step 3.3.4: Build a dynamic update mechanism to achieve incremental data insertion and index reconstruction.

[0043] Furthermore, in Step 4,

[0044] Step 4.1: When the user inputs a text query, use natural language processing technology to parse the key limiting conditions in the query statement and then perform embedding representation, and jointly filter to achieve retrieval enhancement.

[0045] Step 4.2: When the user inputs an image query, use OCR technology to extract the text content from the input image and then perform embedding representation. Combine the extracted text content with the image features input by the user to enhance the semantic vectorization effect.

[0046] Further, in Step 5,

[0047] Step 5.1: Pre-screen candidate data in the Milvus database according to the parsed limiting conditions, and use SQL-like filtering syntax to match the conditions of the stored data;

[0048] Step 5.2: Calculate the similarity between the query vector and the candidate embedding vectors based on the shared semantic space;

[0049] Step 5.3: Use the HNSW index provided by Milvus to retrieve the Top-K candidate results that are most similar to the query semantics.

[0050] Further, in Step 6,

[0051] Step 6.1: For the retrieved candidates, fuse the weight scores of multi-modal features and calculate the comprehensive relevance of each candidate file;

[0052] Step 6.2: Introduce context-based semantic relevance scoring to re-rank the candidate results; use the BERT model to recalculate the semantic relationship between the query and the candidates:

[0053] Step 6.3: For the retrieved candidate files, according to the user input modality, preferentially display the results of the corresponding modality, and at the same time attach information of other modalities in the results.

[0054] Cross-modal Retrieval System for Semantic and Vector Fusion in Data Space

[0055] The retrieval system includes a preprocessing module, an embedding representation module, a mapping module, a semantic enhancement module, a similarity calculation module, and a retrieval output module;

[0056] The preprocessing module collects multi-modal data and performs data preprocessing;

[0057] The embedding representation module generates the embedding representation of each modality through a pre-trained model and saves it in the vector database;

[0058] The mapping module maps the vectors of different modalities to a shared semantic space to ensure that they can be queried and compared in a unified way;

[0059] The semantic enhancement module processes user queries and generates embedding representations, enhancing the semantic vectorization effect by combining natural language processing technology and OCR technology for text queries and image queries respectively;

[0060] The similarity calculation module performs semantic similarity calculation and candidate retrieval;

[0061] The retrieval output module displays cross-modal relevance ranking and multi-modal retrieval results.

[0062] An electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the above method are implemented.

[0063] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps of the above method are implemented.

[0064] Advantages of the present invention

[0065] Compared with the prior art, the present invention realizes unified indexing and semantic alignment of multi-modal data, solves the problem that it is difficult to share the semantic space between heterogeneous modalities, and at the same time combines advanced semantic parsing, embedding generation and efficient database retrieval technologies, significantly improving the accuracy and efficiency of cross-modal information retrieval. Through dynamic condition screening and enhanced query embedding methods, the present invention can better meet complex query requirements and achieve in-depth semantic understanding and precise matching. Description of the drawings

[0066] Figure 1 It is a flowchart of a method for unified indexing and semantic alignment of multi-modal data;

[0067] Figure 2 It is a framework diagram of a unified indexing and semantic alignment of multi-modal data;

[0068] Figure 3 It is a flowchart of a cross-modal retrieval method for semantic and vector fusion in the data space;

[0069] Figure 4 It is a framework diagram of a cross-modal retrieval method for semantic and vector fusion in the data space. Detailed implementation manners

[0070] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0071] Such asFigures 1 to 2 As shown in the figure, the present invention proposes a cross-modal retrieval method for the fusion of semantics and vectors in the data space. The cross-modal file retrieval method of the present invention is divided into two parts: a multi-modal embedding unified index and semantic alignment method, and a cross-modal file information retrieval method that fuses semantic parsing and query embedding.

[0072] Among them, the multi-modal embedding unified index and semantic alignment method is as shown in steps 1-3;

[0073] Step 1: Collect multi-modal data and perform data preprocessing, which specifically includes the following steps:

[0074] Step 1.1: Select a suitable data source (such as a public dataset or a proprietary dataset), obtain text, image, video or audio data through a crawler tool, data collection or an open platform, and ensure that the data source is legal and representative.

[0075] Step 1.2: Perform text cleaning, word segmentation, deduplication and standardization processing on the text data for subsequent model training.

[0076] Step 1.3: For image data, uniformly adjust the image size, normalize the image pixels according to the model requirements, and perform data augmentation operations.

[0077] Use OpenCV to uniformly adjust the image size according to the model requirements to ensure that the image size meets the input requirements of CLIP. The original image size is H orig ×W orig , and the adjusted image size is H new ×W new = 224. The change in the image size is achieved through the following formula:

[0078] ResizedImage = Resize(Image,(H new ,W new ))

[0079] Use PyTorch to perform data augmentation operations, including random cropping, rotation, horizontal flipping, color jittering, etc., to improve the robustness of the model. Use AutoAugment to improve the generalization ability of CLIP in image-text retrieval.

[0080] Step 1.4: For video data, decompose the video into frames and select important key frames for processing.

[0081] Use OpenCV to decompose the video into frames. Assuming the video length is T seconds and N frames are extracted per second, then the total number of frames in the video is T×N. The frame sequence of the video can be expressed by the following formula, where I t represents the t-th frame image:

[0082] V = {I1, I2, ..., I T×N}

[0083] Then, important key frames are selected through K-means clustering for processing. By calculating the feature vector f(I t ), and then performing K-means clustering on the feature vectors. Assume the video contains K clusters, and the set of key frames after clustering is:

[0084] κ = {K1, K2, ..., K K}

[0085] K i is the key frame in cluster i, and the central frame of each cluster is selected as the representative frame.

[0086] Step 1.5: For audio data, noise removal is performed on the audio signal, and the background noise in the audio data is cleared through a filtering denoising algorithm to improve the quality of the speech signal.

[0087] For audio data, WavLM performs noise removal on the audio signal, clears the background noise in the audio data through a filtering denoising algorithm, and uses WaveGlow to improve the quality of the speech signal. First, the mel spectrogram of the speech signal is generated, and a smoother and clearer audio signal is generated through a conditional generative adversarial network (cGAN) and invertible layers (invertible layers). Finally, the generated spectral signal is converted back to the time-domain audio signal through inverse transformation to obtain a high-quality speech signal. The process is as follows:

[0088]

[0089] Step 2: After completing Step 1, the embedding representations of each modality are generated through a pre-trained model and saved in the vector database. Specifically, it includes the following steps:

[0090] Step 2.1: For text data, the preprocessed text data is input into the Sentence-BERT model, and a 768-dimensional text embedding vector is output to generate the semantic representation of the text and save it in the vector database.

[0091] Step 2.1.1: Assume the text input is Text i Convert it into a token sequence through a tokenizer.

[0092] Step 2.1.2: Text i is encoded through the Sentence-BERT model to obtain the context embedding representation of each token

[0093] Step 2.1.3: Pool all the token embedding vectors to obtain the sentence-level embedding E of the text sentence :

[0094] E sentence = pool(h i )

[0095] Step 2.1.4: Save the text embedding vector in the vector database milvus for subsequent semantic alignment and similarity calculation.

[0096] Step 2.2: For image data, input the preprocessed image data into the CLIP model, and use the ViT-Transformer architecture of CLIP to convert the image into a 512-dimensional semantic vector to represent the semantic features of the image, and save it in the vector database.

[0097] Step 2.2.1: First, divide the preprocessed image I norm into N small blocks, each block with a size of P×P. Each small block p i is embedded into a high-dimensional space to obtain the 768-dimensional embedding vector E of the i-th image block i :

[0098] E i = LinearProjection(p i )

[0099] Step 2.2.2: The embedding vectors of each image block are encoded through the self-attention mechanism to generate the global context information of the image. The self-attention calculation formula in the ViT model is as follows:

[0100] Q i = W Q E i , K j = W K E j , V j = W V E j

[0101] Among them, W Q , W K , W V are the projection matrices of the query, key, and value respectively. Q i , K j , V j are the calculated query, key, and value vectors. The final self-attention output is the weighted sum of all blocks:

[0102]

[0103] Step 2.2.3: Feed the self-attention output of each image patch into a multi-layer Transformer Encoder for further processing. Output the vector z image , which contains the global context information of the image.

[0104] Step 2.2.4: The CLIP model aggregates the image patch vectors output by the Transformer Encoder to generate the global semantic representation of the image, and outputs a 512-dimensional image embedding vector E image representing the semantic features of the image.

[0105] E image = CLIP Vision (I norm )

[0106] Step 2.2.5: Store the calculated image embedding vector E image (512-dimensional) in the vector database milvus for subsequent retrieval.

[0107] Step 2.3: For video data, input the video data into the VideoCLIP model, generate a video embedding by extracting the key frames of the video and using the temporal information, and save the resulting 512-dimensional semantic vector in the vector database.

[0108] Step 2.3.1: First, extract features from the key frames. Assume the input key frame is K i After passing through the visual encoder, obtain the image feature vector That is:

[0109]

[0110] Step 2.3.2: To capture the temporal relationship, the VideoCLIP model uses a long short-term memory network (LSTM) to extract the temporal features in the video. The LSTM combines the temporal information of all frames into a final temporal representation, as shown in the following formula:

[0111]

[0112] Step 2.3.3: After temporal modeling, the resulting video feature vector needs to go through an aggregation process to integrate the spatio-temporal information of the entire video into a fixed-length vector. VideoCLIP performs an aggregation operation on the temporal features, taking the last hidden state h T of the LSTM output as the final embedding representation.

[0113] Ε video = Pool(h T )

[0114] Pool(·) represents the pooling operation, i.e., max pooling.

[0115] Step 2.3.4: The obtained 512-dimensional vector Ε video represents the global semantic feature of the video and stores it in the vector database milvus.

[0116] Step 2.4: For audio data, input the preprocessed audio data into the OpenL3 model, and use its deep neural network to convert the audio signal into a 512-dimensional semantic embedding representation.

[0117] Step 2.4.1: After completing the preprocessing of the audio signal, use the Mel-Spectrogram and Mel Frequency Cepstral Coefficients (MFCC) as the feature representation of the audio. First, calculate the Mel-Spectrogram M(t,f). The calculation formula is:

[0118] M(t,f) = |F -1 {F(x(t)) · h f}| 2

[0119] where F(x(t)) is the short-time Fourier transform of the audio signal, h f is the Mel filter bank, and F -1 is the inverse Fourier transform.

[0120] Step 2.4.2: MFCC is a further processing of the Mel-Spectrogram, representing the frequency characteristics of the audio. First, perform a logarithmic transformation on the Mel-Spectrogram, and then perform a discrete cosine transformation on the result to compress the dimension. The calculation of MFCC can be obtained through the following formula:

[0121]

[0122] where n is the dimension of MFCC and F is the total number of frequencies.

[0123] Step 2.4.3: After preprocessing and feature extraction, the Mel-Spectrogram and MFCC features of the audio signal will be input into the OpenL3 model for further embedding generation. Assume the input feature is X input :

[0124] X input = {X Mel (t,f)|t = 1,2,...,T}

[0125] The input features are respectively processed through the convolutional layer and the LSTM layer to process the temporal features. Finally, OpenL3 outputs a 512-dimensional audio embedding E audio :

[0126] E audio=FC(LSTM(Conv(X input )))

[0127] Step 3: After completing step 2, it is necessary to map the vectors of different modalities into a shared semantic space to ensure that they can be queried and compared in a unified way. The specific process is as follows:

[0128] Step 3.1: Define a shared semantic space and its construction method to provide a unified and aligned benchmark for embeddings of all modalities. Select a universal cross-modal embedding CLIP model and use a contrastive learning strategy to train the shared space based on aligned semantic labels and feature data.

[0129] For text modalities, Sentence-BERT embedding is used directly as the input of the shared space; for image, video, and audio modalities, their respective modality embeddings are used as input.

[0130] We use contrastive learning to bring semantically similar multimodal embeddings closer together. The specific optimization objective function is as follows:

[0131]

[0132] Among them, z i ,z j Denotes semantically related vectors of different modalities, sim(·,·) denotes cosine similarity, and τ is a temperature coefficient used to adjust the smoothness of the distribution. For the text modality, Sentence-BERT embedding is used directly as the input of the shared space; for the image, video, and audio modalities, the embeddings of their respective modalities are used as input.

[0133] Step 3.2: Based on the shared semantic space, a modality mapping network is designed to project the embeddings of different modalities into the shared semantic space, further optimizing their alignment while preserving the characteristic details of each modality. The specific steps are as follows:

[0134] Step 3.2.1: Text Modality Mapping Network: Use an enhanced version of the multi-layer perceptron (MLP) or direct projection, and add nonlinear activation functions such as GELU and normalization layers to improve training stability.

[0135] Step 3.2.2: Image Modality Mapping Network: Add a small Transformer model to capture image context. Specifically, DeiT-Tiny, a small version of the ViT architecture, is used to capture image context. The preprocessed image embedding vector is used as a sequence input, and a positional encoding P is added to enhance the model's understanding of spatial distribution. The formula is as follows:

[0136] h image= TransformerEncoder(e image + p)

[0137] Step 3.2.3: Video modality mapping network: Combine temporal LSTM or Transformer to enhance the alignment of temporal features. The embedding of each frame extracts local features through Transformer, and uses the LSTM layer to perform temporal modeling on the frame sequence. The formula is as follows:

[0138] h video = LSTM(TransformerEncoder(e frame ))

[0139] Step 3.2.4: Audio modality mapping network: Convolutional layer + LSTM layer extract time and frequency features. The formula is as follows:

[0140] h audio = Bi-LSTM(Conv2D(e audio ))

[0141] Step 3.2.5: Gradually debug the network structure parameters (such as the number of LSTM units, the number of Transformer heads, etc.), find the model configuration with the best performance, and ensure the projection quality of each modality in the shared semantic space. The embeddings of all modalities are mapped to the shared semantic space and unified into a 512-dimensional vector representation:

[0142]

[0143] Step 3.3: Store the aligned multi-modal embeddings in the Milvus database and build an efficient indexing mechanism to support real-time retrieval of large-scale multi-modal data. The specific steps are as follows:

[0144] Step 3.3.1: Use Milvus to create a vector collection to store the aligned multi-modal embeddings, and support modality tagging and additional metadata storage. The fields are designed as shown in Table 1, and the collection is defined as follows:

[0145] Collection = {(id, embedding, modality, metadata)}

[0146]

[0147] Table 1 Vector database collection data structure design Step 3.3.2: Formulate each aligned embedding as follows:

[0148]

[0149] Normalize all vectors to ensure the retrieval performance of the embedded vectors and store them in the vector database.

[0150] Step 3.3.3: Build an HNSW index for the stored vector fields to improve the retrieval efficiency.

[0151] Step 3.3.4: Build a dynamic update mechanism to achieve incremental data insertion and index reconstruction.

[0152] After completing the unified indexing and semantic alignment of multimodal data, it is necessary to design a cross-modal retrieval method for semantic and vector fusion in the data space. The method process is as Figure 3 and Figure 4 shown below. The specific steps are as follows:

[0153] Step 4: Process the user query and generate an embedded representation. Combine natural language processing technology and OCR technology for text queries and image queries respectively to enhance the semantic vectorization effect.

[0154] Step 4.1: When the user inputs a text query, use natural language processing technology to parse the key limiting conditions in the query statement and then perform an embedded representation, and perform joint screening to achieve retrieval enhancement.

[0155] Step 4.1.1: Collect and clean the data, annotate the modality type (text / image, etc.), time information ("2024"), keywords ("ocean", etc.), combine them to generate a fine-tuning statement, and then fine-tune the large model ChatGLM4 as a key condition extraction model so that it can extract key entity information.

[0156] Step 4.1.2: Input the user's text query into the key condition extraction model to obtain the condition screening result. Use the RAKE algorithm to sort the importance of the extracted keywords, and retain the high-weight keywords as the main semantic information. It is expressed as:

[0157] Condition = {Type, Time, Topic}

[0158] When the input is: "Retrieve ocean-related images after 2024", the output is:

[0159] Condition = {Type: image, Time: 2024+, Topic: Ocean}

[0160] Step 4.1.3: Input the parsed query text into the Sentence-BERT model to generate a query embedding, so that the query embedding and other multimodal information are in a shared semantic space:

[0161] q text= Sentence - BERT query text )

[0162] Step 4.1.4: Combine the key conditions (Condition) with the embedding generated by fusing the semantic vectors through a multi - layer perceptron, and after forming a condition - enhanced query embedding representation, perform normalization. The feature fusion is as follows:

[0163] q enhanced = MLP([q text ; Condition])

[0164] Step 4.2: When the user inputs an image query, use OCR technology to extract the text content from the input image and then perform an embedding representation. Combine the extracted text content with the image features input by the user to enhance the semantic vectorization effect.

[0165] Step 4.2.1: For the input query image, use the CLIP model to extract the image features and generate the image semantic embedding q image .

[0166] Step 4.2.2: Use the optical character recognition technology GoogleVisionAPI to extract the text content from the image, and input the extracted OCR text into the Sentence - BERT model to generate the semantic embedding q OCR .

[0167] Step 4.2.3: Use the attention mechanism for the multi - modal feature fusion network to fuse and normalize the image features and the OCR text embedding. The feature fusion is as follows:

[0168]

[0169] Step 5: Semantic similarity calculation and candidate retrieval. Combine the efficient retrieval mechanism of the Milvus database to generate candidate results of cross - modal file information that are most relevant to the user's query.

[0170] Step 5.1: Pre - screen candidate data in the Milvus database according to the parsed limiting conditions (modal type, time range, etc.), and use SQL - like filtering syntax to match the conditions of the stored data.

[0171] Step 5.2: Based on the shared semantic space, calculate the cosine similarity between the query vector and the candidate embedding vectors, as follows:

[0172]

[0173] Step 5.3: Use the HNSW index provided by Milvus to retrieve the Top - K candidate results that are most similar to the query semantics.

[0174] Input query vector q final and candidate set D filtered Using the hierarchical graph structure of the HNSW index, find the Top-K vectors v that are closest to q final as follows: top-K

[0175]

[0176] Sort the candidate results in descending order of similarity and return the Top-K file information of the final ranking.

[0177] Step 6: Cross-modal relevance ranking and multi-modal retrieval result display.

[0178] Step 6.1: For the retrieved candidates, fuse the weight scores of multi-modal features and calculate the comprehensive relevance of each candidate file. α, β, γ, δ are weight parameters of different modalities, and the formula is as follows:

[0179] R = α·sim text + β·sim image + γ·sim video + δ·sim audio

[0180] Step 6.2: Introduce context-based semantic relevance scoring to re-rank the candidate results. Use the BERT model to recalculate the semantic relationship between the query and the candidates:

[0181] R final = BERT(query, candidate)

[0182] Step 6.3: For the retrieved candidate files, according to the user input modality, preferentially display the results of the corresponding modality, and at the same time attach information of other modalities (such as text descriptions, thumbnails, etc.) in the results.

[0183] A retrieval system for a cross-modal retrieval method of semantic and vector fusion in a data space

[0184] The retrieval system includes a preprocessing module, an embedding representation module, a mapping module, a semantic enhancement module, a similarity calculation module, and a retrieval output module;

[0185] The preprocessing module collects multi-modal data and performs data preprocessing;

[0186] The embedding representation module generates the embedding representation of each modality through a pre-trained model and saves it in a vector database;

[0187] ​The mapping module maps vectors of different modalities into a shared semantic space, ensuring that they can be queried and compared in a unified manner;

[0188] The semantic enhancement module processes user queries and generates embedded representations. By combining natural language processing techniques and OCR techniques for text queries and image queries respectively, the semantic vectorization effect is enhanced;

[0189] The similarity calculation module performs semantic similarity calculation and candidate retrieval;

[0190] The retrieval output module displays cross-modal relevance ranking and multi-modal retrieval results.

[0191] An electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the above method are implemented.

[0192] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps of the above method are implemented.

[0193] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory for the method described in the present invention is intended to include, but not be limited to, these and any other suitable types of memory.

[0194] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means such as coaxial cable, fiber optic, digital subscriber line (DSL), or wireless means such as infrared, wireless, microwave, etc. The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server, data center, etc. that includes one or more integrated available media. The available media can be magnetic media such as floppy disks, hard disks, magnetic tapes, optical media such as high-density digital video discs (DVDs), or semiconductor media such as solid-state discs (SSDs), etc.

[0195] In the implementation process, the steps of the above method can be completed by the integrated logic circuit in the hardware of the processor or the instructions in the form of software. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by the hardware processor or completed by a combination of the hardware and software modules in the processor. The software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method. To avoid repetition, it will not be described in detail here.

[0196] It should be noted that the processor in the embodiments of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the above method embodiments can be completed by the integrated logic circuit in the hardware of the processor or instructions in the form of software. The above-mentioned processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by a combination of the hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.

[0197] The above has introduced in detail the cross-modal retrieval method for semantic and vector fusion in the data space proposed by the present invention, and elaborated on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A cross-modal retrieval method for semantic and vector fusion in a data space, characterized in that: The method specifically includes the following steps: Step 1: Collect multi-modal data and perform data preprocessing; Step 2: Generate embedded representations for each modality through a pre-trained model and save them in a vector database; Step 3: Map vectors of different modalities to a shared semantic space to ensure that they can be queried and compared in a unified manner; Step 4: Process user queries and generate embedded representations. For text queries and image queries, combine natural language processing techniques and OCR techniques to enhance the semantic vectorization effect; Step 5: Perform semantic similarity calculation and candidate retrieval; Step 6: Display cross-modal relevance ranking and multi-modal retrieval results.

2. The method according to claim 1, wherein: In Step 1, it includes Step 1.1: Select a suitable data source and obtain text, image, video or audio data through a crawler tool, data collection or an open platform; Step 1.2: Perform text cleaning, word segmentation, deduplication and standardization processing on the text data for subsequent model training; Step 1.3: For image data, uniformly adjust the image size, normalize the image pixels according to the model requirements, and perform data augmentation operations; Step 1.4: For video data, decompose the video into frames and select important key frames for processing; Step 1.5: For audio data, remove noise from the audio signal, and clean the background noise in the audio data through a filter denoising algorithm to improve the quality of the speech signal.

3. The method according to claim 2, wherein: In Step 2, it includes Step 2.1: For text data, input the preprocessed text data into the Sentence-BERT model, output a 768-dimensional text embedding vector, generate the semantic representation of the text, and save it in the vector database; Step 2.2: For image data, input the preprocessed image data into the CLIP model, and use the ViT-Transformer architecture of CLIP to convert the image into a 512-dimensional semantic vector to represent the semantic features of the image, and save it in the vector database; Step 2.3: For video data, input the video data into the VideoCLIP model, extract the key frames of the video and use the temporal information to generate a video embedding, and obtain a 512-dimensional semantic vector and save it in the vector database; Step 2.4: For audio data, input the preprocessed audio data into the OpenL3 model, and use its deep neural network to convert the audio signal into a 512-dimensional semantic embedding representation.

4. The method according to claim 3, wherein: In Step 3, it includes Step 3.1: Define a shared semantic space and its construction method to provide a unified alignment benchmark for the embeddings of all modalities; select a general cross-modal embedding CLIP model, and use a contrastive learning strategy to train the shared space based on the aligned semantic labels and feature data; For the text modality, directly use the Sentence-BERT embedding as the input to the shared space; For the image, video, and audio modalities, use their respective modality embeddings as the input; Step 3.2: Based on the shared semantic space, design a modality mapping network to project embeddings of different modalities into the shared semantic space; Step 3.3: Store the aligned multi-modal embeddings in the Milvus database and build an indexing mechanism.

5. The method according to claim 4, wherein: In Step 3.2, the specific steps of designing a modality mapping network to project embeddings of different modalities into the shared semantic space are as follows: Step 3.2.1: Design a text modality mapping network: use a multi-layer perceptron (MLP) or direct projection; Step 3.2.2: Design an image modality mapping network: add a small Transformer model to capture image context features; Step 3.2.3: Design a video modality mapping network: combine temporal LSTM or Transformer to enhance the alignment of temporal features; Step 3.2.4: Design an audio modality mapping network: use a convolutional layer + LSTM layer to extract time and frequency features; Step 3.2.5: Gradually debug the network structure parameters to ensure the projection quality of each modality in the shared semantic space.

6. The method according to claim 5, wherein: In Step 3.3, the specific steps of building an indexing mechanism are as follows: Step 3.3.1: Create a vector collection (Collection) in Milvus to store the aligned multi-modal embeddings; Step 3.3.2: Formulate each aligned embedding, normalize all vectors to ensure the retrieval performance of the embedding vectors, and store them in the vector database; Step 3.3.3: Build an HNSW index for the stored vector fields; Step 3.3.4: Build a dynamic update mechanism to achieve incremental data insertion and index reconstruction.

7. The method according to claim 4, wherein: In Step 4, Step 4.1: When the user inputs a text query, use natural language processing technology to parse the key limiting conditions in the query statement and then perform embedding representation, and jointly filter to achieve retrieval enhancement; Step 4.2: When the user inputs an image query, use OCR technology to extract text content from the input image and then perform embedding representation, and combine the extracted text content with the image features input by the user to enhance the semantic vectorization effect.

8. The method according to claim 7, wherein: In Step 5, Step 5.1: Pre-screen candidate data in the Milvus database according to the parsed limiting conditions, and use SQL-like filtering syntax to match the conditions of the stored data; Step 5.2: Calculate the similarity between the query vector and the candidate embedding vectors based on the shared semantic space; Step 5.3: Use the HNSW index provided by Milvus to retrieve the top-K candidate results that are most similar to the query semantics.

9. The method according to claim 8, wherein: In Step 6, Step 6.1: For the retrieved candidates, fuse the weight scores of multi-modal features and calculate the comprehensive relevance of each candidate file; Step 6.2: Introduce context-based semantic relevance scoring to re-rank the candidate results; use the BERT model to recalculate the semantic relationship between the query and the candidates; Step 6.3: For the retrieved candidate files, according to the modality input by the user, preferentially display the results of the corresponding modality, and at the same time attach information of other modalities in the results.

10. A retrieval system for performing cross-modal retrieval method of semantic and vector fusion in the data space described in any one of claims 1 to 9, characterized in that: The retrieval system includes a preprocessing module, an embedding representation module, a mapping module, a semantic enhancement module, a similarity calculation module, and a retrieval output module; The preprocessing module collects multi-modal data and performs data preprocessing; The embedding representation module generates an embedding representation for each modality through a pre-trained model and saves it in a vector database; The mapping module maps vectors of different modalities to a shared semantic space to ensure that they can be queried and compared in a unified manner; The semantic enhancement module processes user queries and generates embedding representations, enhancing the semantic vectorization effect by combining natural language processing technology and OCR technology for text queries and image queries respectively; The similarity calculation module performs semantic similarity calculation and candidate retrieval; The retrieval output module displays cross-modal relevance ranking and multi-modal retrieval results.

Images