Multi-modal data processing and retrieval method

Abstract

The invention particularly relates to a multi-modal data processing and retrieving method. The multi-modal data processing and retrieval method comprises the following steps: respectively carrying out depth feature extraction on image data and text data to generate an image vector and a text vector; carrying out interaction on the image vector and the text vector, and capturing semantic association information among multiple modes; mapping the vectors after interaction to a unified semantic space to realize semantic alignment among different modes; dividing the unified semantic space vector into fragments, storing the fragments in distributed nodes, and establishing a vector index; and generating a query vector by using an image or text queried by a user, carrying out parallel calculation on the similarity with a storage vector, and returning a retrieval result according to the similarity. According to the multi-modal data processing and retrieval method, the problems that the semantic difference between different modal data such as images and texts is large, the retrieval efficiency is low and storage is difficult to expand are solved, the accuracy and efficiency of multi-modal data retrieval are remarkably improved, and the method has remarkable technical advantages and wide application scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information retrieval, in particular to a multi-modal data processing and retrieval method. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, image, text, audio and other data types dominate the Internet and enterprise data. Traditional single-modal retrieval methods are difficult to fully exploit the semantic relationship between different modalities, resulting in insufficient accuracy and recall rate of the retrieval results.

[0003] Existing multi-modal retrieval techniques rely on single-modal feature matching or shallow joint representation, which cannot achieve efficient storage and fast retrieval in large-scale data scenarios.

[0004] In addition, existing methods lack deep interaction mechanisms when processing semantic mapping between images and text, making it difficult to capture fine-grained semantic information. Therefore, a new method is needed that can support deep multi-modal feature extraction, unified semantic space construction, and distributed vector storage and retrieval simultaneously to improve retrieval accuracy and system expansion capability.

[0005] Based on the above situation, the present application provides a multi-modal data processing and retrieval method. SUMMARY

[0006] The present application provides a simple and efficient multi-modal data processing and retrieval method to overcome the shortcomings of the prior art.

[0007] The present application is achieved by the following technical solutions:

[0008] A multi-modal data processing and retrieval method, characterized by comprising the following steps:

[0009] Step S1, deep feature extraction of image data, using block processing and self-attention mechanism to generate image vector representation;

[0010] In the step S1, in the image feature extraction process, the image encoder adopts a deep network structure based on self-attention mechanism, which processes the image in blocks and maps the features. Each image block aggregates information through multiple layers of attention mechanism, thereby obtaining global image semantic representation.

[0011] Step S2, deep feature extraction of text data, generating text vector representation through word embedding, position encoding and multi-layer attention mechanism;

[0012] In the text feature extraction process in the step S2, the text encoder adopts a self-attention network, maps the text sequence into a high-dimensional vector representation through word embedding and position encoding, captures the context relationship between words through a multi-layer attention mechanism, and forms a semantic-rich text vector through sequence encoding to realize long text dependency modeling.

[0013] In step S3, cross-modal attention mechanism is used to interact the image vector and the text vector to capture the semantic association information between the multi-modalities.

[0014] In the step S3, the image feature vector is taken as a query, and the text feature vector is taken as a key and a value, and the complementary and enhancement of the image and the text semantics are realized through attention distribution.

[0015] Meanwhile, a multi-layer attention mechanism is introduced to enable different semantic dimensions to interact in parallel to capture more rich cross-modal association features.

[0016] In addition, to enhance the robustness and generalization ability of the model, residual connection and normalization operation are added in the interaction layer to make the information flow stable and efficient in the deep network, and to ensure the quality and consistency of feature fusion.

[0017] In step S4, the vector after interaction is mapped to a unified semantic space to realize semantic alignment between different modalities.

[0018] In the step S4, the obtained fusion vector is further mapped to a unified semantic space through a multi-modal interaction layer; the unified semantic space realizes semantic alignment of the image and the text through end-to-end training, so that different modal data of the same semantic content have similar vector representations.

[0019] In the unified semantic space, normalization and vector distance measurement methods are used to uniformly manage the features of different modalities, thereby supporting cross-modal similarity calculation.

[0020] In step S5, the unified semantic space vector is divided into fragments and stored in distributed nodes to establish an efficient vector index.

[0021] In the step S5, based on the distributed storage architecture, the multi-modal vector data is self-defined and divided into multiple fragments and stored on the distributed nodes to support horizontal expansion of large-scale data.

[0022] In the distributed storage architecture, each node is responsible for managing part of the vector data and provides efficient vector insertion, update and deletion interfaces.

[0023] The vector index adopts a combination of hierarchical clustering and inverted index to reduce the amount of calculation during retrieval, supports batch operation to improve storage efficiency, supports dynamic index update, and can take effect immediately when vector data is added or modified.

[0024] In step S6, a query vector is generated using the image or text of the user query, and the similarity between the query vector and the stored vectors is calculated in parallel on the distributed storage nodes. The retrieval results are returned according to the similarity.

[0025] The vector retrieval process and parallel computing strategy achieve low-latency and high-accuracy cross-modal data retrieval.

[0026] In step S6, in the retrieval stage, the user input text or image is used as a query vector, and an approximate nearest neighbor search algorithm is used to calculate the similarity between the query vector and the stored vectors in parallel on the distributed nodes to quickly locate the most relevant vector set to the query. The retrieval results are sorted according to the similarity from high to low and returned to the user.

[0027] A multi-modal data processing and retrieval system for implementing the above method, comprising:

[0028] An image feature vector extraction module is responsible for deep feature extraction of image data, and uses block processing and self-attention mechanism to generate image vector representation;

[0029] A text feature vector extraction module is responsible for deep feature extraction of text data, and generates text vector representation through word embedding, position encoding and multi-layer attention mechanism;

[0030] A cross-modal semantic alignment module uses cross-modal attention mechanism to interact image vectors and text vectors, captures semantic association information between multi-modalities, and maps the interacted vectors to a unified semantic space to achieve semantic alignment between different modalities;

[0031] A vector index module is responsible for dividing the unified semantic space vectors into shards and storing them in distributed nodes to establish an efficient vector index;

[0032] A multi-modal query module is responsible for generating a query vector using the image or text of the user query, and calculating the similarity between the query vector and the stored vectors in parallel on the distributed storage nodes. The retrieval results are returned according to the similarity.

[0033] A multi-modal data processing and retrieval device, characterized by: a memory and a processor; the memory is used to store a computer program, and the processor is used to execute the computer program to realize the method steps described above.

[0034] A readable storage medium, characterized in that: the readable storage medium stores a computer program, the computer program is executed by a processor to realize the method steps.

[0035] The beneficial effects of the present application are: the multi-modal data processing and retrieval method solves the problems of large semantic difference between image and text and other different modal data, low retrieval efficiency and difficult storage expansion, significantly improves the accuracy and efficiency of multi-modal data retrieval, has significant technical advantages, and is suitable for search engines, content recommendation, intelligent question and answer and other application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0037] ATTACHMENT Figure 1 The multi-modal data processing and retrieval method of the present application is shown in the figure. DETAILED DESCRIPTION

[0038] In order to make those skilled in the art better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0039] The multi-modal data processing and retrieval method comprises the following steps:

[0040] Step S1, deep feature extraction is performed on image data, and block processing and self-attention mechanism are used to generate image vector representation;

[0041] In the step S1, in the image feature extraction process, the image encoder adopts a deep network structure based on self-attention mechanism, and performs block processing and feature mapping on the image. Each image block aggregates information through multi-layer attention mechanism, so as to obtain global image semantic representation.

[0042] Step S2, deep feature extraction is performed on text data, and text vector representation is generated through word embedding, position encoding and multi-layer attention mechanism;

[0043] In step S2, during the text feature extraction process, the text encoder also uses a self-attention network to map the text sequence into a high-dimensional vector representation through word embedding and position encoding. Then, it captures the contextual relationships between words through a multi-layer attention mechanism and forms semantically rich text vectors through sequence encoding, thereby realizing the modeling of long text dependency relationships.

[0044] Using the above method, images and text are mapped into high-dimensional feature spaces, providing a foundation for subsequent cross-modal information interaction and the construction of a unified semantic space.

[0045] Step S3: Utilize cross-modal attention mechanism to interact image vectors and text vectors to capture semantic association information between multiple modalities;

[0046] The multimodal interaction layer employs a cross-modal attention mechanism, enabling image features to mutually enhance text features while supporting parallel processing across multiple semantic dimensions.

[0047] In step S3, the image feature vector is used as the query and the text feature vector is used as the key and value. Attention allocation is used to achieve the complementarity and enhancement of image and text semantics.

[0048] Meanwhile, a multi-layer attention mechanism is introduced to enable different semantic dimensions to interact in parallel in order to capture richer cross-modal association features;

[0049] In addition, to enhance the robustness and generalization ability of the model, residual connections and normalization operations are added to the interaction layer to keep the information flow stable and efficient in the deep network, ensuring the quality and consistency of feature fusion.

[0050] Step S4: Map the interacted vectors to a unified semantic space to achieve semantic alignment between different modalities;

[0051] In step S4, the obtained fusion vector is further mapped to a unified semantic space through a multimodal interaction layer; the unified semantic space achieves semantic alignment between images and text through end-to-end training, so that different modal data of the same semantic content have similar vector representations;

[0052] In a unified semantic space, normalization and vector distance metrics are used to manage features of different modalities in a unified manner, thereby supporting cross-modal similarity calculation.

[0053] The unified semantic space not only preserves the feature information of each modality, but also reduces the semantic differences between modalities through alignment mechanisms, enabling the retrieval system to more accurately identify the correlation between user queries and multimodal data in the database.

[0054] Step S5: Divide the unified semantic space vector into fragments and store them on distributed nodes to establish an efficient vector index;

[0055] Distributed vector storage employs data sharding, node redundancy, load balancing, and index optimization strategies to achieve efficient storage and horizontal scaling of massive amounts of data.

[0056] In step S5, based on a distributed storage architecture, the multimodal vector data is custom-divided into multiple shards and stored on distributed nodes to support the horizontal scaling of large-scale data.

[0057] In a distributed storage architecture, each node is responsible for managing a portion of the vector data and provides efficient vector insertion, update, and deletion interfaces;

[0058] The vector index uses a combination of hierarchical clustering and inverted indexes to reduce the amount of computation during retrieval. It also supports batch operations to improve storage efficiency and dynamic index updates, which can take effect immediately when new or modified vector data is added.

[0059] Through load balancing and sharding redundancy mechanisms, the system can maintain high availability and stability when nodes fail or data volume surges, enabling the persistence and fast access of massive multimodal data.

[0060] Step S6: Generate a query vector using the image or text queried by the user, and calculate the similarity with the stored vector in parallel on the distributed storage nodes, and return the search results according to the similarity.

[0061] Vector retrieval process and parallel computing strategy enable low-latency, high-accuracy cross-modal data retrieval.

[0062] In step S6, during the retrieval phase, the user-input text or image is used as the query vector. An approximate nearest neighbor search algorithm is used to calculate the similarity between the query vector and the stored vector in parallel on distributed nodes to quickly locate the set of vectors most relevant to the query. The retrieval results are sorted from high to low similarity and returned to the user.

[0063] Through distributed parallel computing and efficient indexing mechanisms, low-latency and high-accuracy multimodal retrieval is achieved in large-scale data scenarios.

[0064] This multimodal data processing and retrieval method is applicable to various application scenarios such as intelligent search, content recommendation, knowledge question answering, and image retrieval, and can quickly return relevant multimodal data based on the query intent.

[0065] This multimodal data processing and retrieval system, used to implement the above method, includes:

[0066] The image feature vector extraction module is responsible for extracting deep features from image data and generating image vector representations using block processing and a self-attention mechanism.

[0067] The text feature vector extraction module is responsible for deep feature extraction from text data, generating text vector representations through word embedding, positional encoding, and multi-layer attention mechanisms;

[0068] The cross-modal semantic alignment module uses a cross-modal attention mechanism to interact with image vectors and text vectors, captures semantic association information between multiple modalities, and maps the interacted vectors to a unified semantic space to achieve semantic alignment between different modalities.

[0069] The vector indexing module is responsible for dividing the unified semantic space vectors into shards and storing them on distributed nodes to build an efficient vector index.

[0070] The multimodal query module is responsible for generating query vectors from images or text queried by users, calculating the similarity between the query vectors and the stored vectors in parallel on distributed storage nodes, and returning the search results based on the similarity.

[0071] The multimodal data processing and retrieval device includes a memory and a processor; the memory is used to store a computer program, and the processor is used to execute the computer program to implement the above-described method steps.

[0072] The readable storage medium stores a computer program that, when executed by a processor, implements the above-described method steps.

[0073] Compared with existing technologies, this multimodal data processing and retrieval method has the following characteristics:

[0074] 1) Improved cross-modal semantic understanding capabilities: Through deep encoders and multimodal interaction layers, the semantic relationships between images and text can be fully explored, enabling data from different modalities to be effectively aligned in a unified space, thereby improving the relevance of search results.

[0075] 2) Enhanced retrieval efficiency and scalability: Utilizing distributed vector storage and indexing strategies, it enables rapid retrieval of massive multimodal data, supports node expansion and load balancing, and ensures that the system maintains high performance even in large-scale data environments.

[0076] 3) Improved system robustness: Multi-head attention mechanism and residual connection enable the model to remain stable in deep networks, avoid information loss and gradient vanishing, and improve the model's generalization ability and retrieval accuracy.

[0077] 4) Reduced computational resource consumption: By using the approximate nearest neighbor algorithm and hierarchical indexing strategy, the computational overhead during retrieval is effectively reduced, while supporting batch data processing and improving system resource utilization.

[0078] 5) Adaptable to multiple application scenarios: This method is applicable to multiple fields such as intelligent search, content recommendation, knowledge question answering, and image retrieval, and can meet the comprehensive needs of different business scenarios for retrieval accuracy, response speed, and system scalability.

[0079] 6) Supports dynamic updates and maintenance: It can update vector data and indexes in real time, so that newly added or modified data can be used for retrieval immediately, reducing data maintenance costs and enhancing the system's usability.

[0080] In summary, this invention technically solves the problems of large semantic differences, low retrieval efficiency, and difficulty in expanding storage in multimodal data retrieval, and has significant technical advantages and broad application prospects.

[0081] The embodiments described above are merely one specific implementation of the present invention. Ordinary changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-modal data processing and retrieval method, characterized by: The method comprises the following steps: Step S1, deep feature extraction is performed on image data, and a block processing and self-attention mechanism are used to generate an image vector representation; Step S2, deep feature extraction is performed on text data, and a word embedding, position encoding and multi-layer attention mechanism are used to generate a text vector representation; Step S3, a cross-modal attention mechanism is used to interact the image vector and the text vector, and semantic association information between the multi-modalities is captured; Step S4, the vectors after the interaction are mapped to a unified semantic space, and semantic alignment between different modalities is achieved; Step S5, the unified semantic space vectors are divided into fragments, and are stored in distributed nodes, and a vector index is established; Step S6, a query vector is generated by using an image or text of a user query, and a similarity between the query vector and the stored vector is calculated in parallel on the distributed storage nodes, and a retrieval result is returned according to the similarity.

2. The multi-modal data processing and retrieval method according to claim 1, wherein in the step S1, in the image feature extraction process, an image encoder adopts a deep network structure based on a self-attention mechanism, block processing and feature mapping are performed on the image, each image block is aggregated by a multi-layer attention mechanism, and thus global image semantic representation is obtained.

3. The multi-modal data processing and retrieval method according to claim 1, wherein in the step S2, in the text feature extraction process, a text encoder adopts a self-attention network, a text sequence is mapped to a high-dimensional vector representation by word embedding and position encoding, a multi-layer attention mechanism is used to capture the context relationship between words, and a text vector is formed by sequence encoding, so as to model the long text dependency relationship.

4. The multi-modal data processing and retrieval method according to claim 1, wherein in the step S3, the image feature vector is used as a query, the text feature vector is used as a key and a value, and the complement and enhancement of the image and the text semantics are achieved by attention allocation; meanwhile, a multi-layer attention mechanism is introduced, so that different semantic dimensions can be interacted in parallel to capture richer cross-modal association features; in addition, in order to enhance the robustness and generalization ability of the model, a residual connection and a normalization operation are added in the interaction layer, so that the information flow remains stable and efficient in the deep network, and the quality and consistency of the feature fusion are ensured.

5. The multi-modal data processing and retrieval method according to claim 1, wherein in the step S4, the fusion vector obtained by the multi-modal interaction layer is further mapped to a unified semantic space; the unified semantic space realizes semantic alignment of the image and the text through end-to-end training, so that different modal data of the same semantic content have similar vector representations; in the unified semantic space, normalization and vector distance measurement methods are used to uniformly manage the features of different modalities, so as to support cross-modal similarity calculation.

6. The multi-modal data processing and retrieval method according to claim 1, wherein in the step S5, based on a distributed storage architecture, the multi-modal vector data is self-defined and divided into multiple fragments, and is stored in distributed nodes, so as to support horizontal expansion of large-scale data. ​ ​ ​ ​ ​ ​ ​ ​ In the distributed storage architecture, each node is responsible for managing part of the vector data and providing vector insertion, update and deletion interfaces; The vector index adopts a combination of hierarchical clustering and inverted index to reduce the amount of calculation during retrieval, while supporting batch operations to improve storage efficiency, supporting dynamic index updates, and being able to take effect immediately when new or modified vector data is added. The vector retrieval process and parallel computing strategy realize low-latency and high-accuracy cross-modal data retrieval.

7. The multi-modal data processing and retrieval method of claim 1, wherein: In the retrieval stage of step S6, the user input text or image is used as a query vector, and an approximate nearest neighbor search algorithm is used to calculate the similarity between the query vector and the stored vector on the distributed nodes in parallel, to quickly locate the most relevant vector set to the query; the retrieval results are sorted in descending order of similarity and returned to the user.

8. A multi-modal data processing and retrieval system characterized by: A device for implementing the method of any one of claims 1 to 7, comprising An image feature vector extraction module responsible for deep feature extraction of image data, using block processing and self-attention mechanism to generate image vector representation; A text feature vector extraction module responsible for deep feature extraction of text data, generating text vector representation through word embedding, position encoding and multi-layer attention mechanism; A cross-modal semantic alignment module that uses cross-modal attention mechanism to interact image vectors and text vectors, captures semantic association information between multi-modal, and maps the interacted vectors to a unified semantic space, achieving semantic alignment between different modalities; A vector index module responsible for dividing the unified semantic space vectors into shards and storing them in distributed nodes, establishing a vector index; A multi-modal query module responsible for generating query vectors from user queries of images or text, and calculating the similarity between the query vectors and the stored vectors in parallel on the distributed storage nodes, and returning the retrieval results according to the similarity.

9. A multi-modal data processing and retrieval device, characterized by: A device comprising a memory and a processor; the memory is used to store a computer program, and the processor is used to execute the computer program to realize the method of any one of claims 1 to 7.

10. A readable storage medium characterized by: The computer program stored on the readable storage medium is executed by the processor to realize the method of any one of claims 1 to 7.

Images