Dual-engine driven multi-modal data retrieval method, device and system

Abstract

The application discloses a kind of dual-engine driven multimodal data retrieval method, equipment and system, belong to multimodal data comprehensive retrieval field, including steps: S1, with multimodal data as input, judge the semantic label of multimodal data, construct semantic label library;Unified feature space is constructed, and the feature vector under the same feature space of different modal data is extracted, and feature vector library is constructed;S2, with multimodal retrieval condition as input, respectively obtain the semantic label and feature vector of retrieval condition by multimodal data intelligent processing method, based on semantic label library and feature vector library, utilize cross-modal semantic label retrieval engine and cross-modal feature vector retrieval engine in multimodal database Complete the query of multimodal data that meets retrieval condition, after sorting, push the search result.The application supports a kind of modal data input to retrieve all modal data containing result, can effectively realize the comprehensive accurate recall of query result in different types of data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal data integrated retrieval, and more particularly to a dual-engine driven multi-modal data retrieval method, device and system. BACKGROUND

[0002] With the continuous expansion of the data size on the Internet, the data types are increasingly showing the characteristics of diversification, and the data modalities of interest to users are no longer single. The retrieval requirements of users are increasingly showing a development trend from single modality to cross modality. Modality refers to the form of data expression, including text, image, video and audio, etc. In recent years, users usually hope to obtain the required data in their own way. The traditional single modality retrieval cannot solve the compatibility problem of the two, because they can only return the original data for query in the same way, and cannot meet the retrieval requirements. Multi-modal retrieval can query different forms of information by using various data, and the key problem of performing multi-modal retrieval is to consider the semantic similarity between different forms of data.

[0003] Multi-modal retrieval is to establish a network structure of different modalities to realize the retrieval between different modalities of data. The modality of the query and the retrieval result is not necessarily the same, for example, retrieving text by image, retrieving video by audio, etc. Unlike the traditional multi-modal retrieval, the key of the existing research on multi-modal retrieval is to map the data of different modalities to a common space and establish a relationship model between the two. The main problem faced by multi-modal retrieval research is how to fill the "semantic gap" between different modalities of data to establish a general representation of multi-modal data.

[0004] At present, multi-modal retrieval is generally divided into two methods. One is semantic label retrieval, which identifies different modalities of data such as text, image, video and audio, gives text semantic labels and stores them through hash coding. By analyzing the query condition, the implicit label information in the query condition is obtained to query the label in the library, and then the retrieval results of different modalities are obtained. The other is feature vector retrieval, which converts different modalities of data such as text, image, video and audio into feature vectors through feature extraction and storage. When querying, the query condition is also converted into a feature vector, and the retrieval results of different modalities are obtained by calculating the similarity of the feature vectors.

[0005] Semantic label retrieval is a relatively mature multi-modal retrieval method at present. Semantic information is important information retained during query. Different forms of data have different feature spaces, but they have the same semantic space, and data with the same semantics have relevance in various modalities. Semantic information can not only be used to represent the correlation between multi-modalities, but also the connection within each modality. In order to obtain good retrieval results, embedding methods can be used to retain semantic and original feature information at the same time. Some scholars propose a latent semantic sparse hashing method, which combines sparse coding and matrix decomposition to obtain a latent shared semantic space. In the deep method, a convolutional neural network is usually used to generate an image, and a semantic part is used to perform feature embedding on each word and generate text through a text CNN or a recurrent neural network. The lack of semantic information leads to limited retrieval results. Some regression methods focus on retaining semantic information. However, the correlation between labels in multi-modal information is ignored. In addition, they fix the public space as the label space, which is low in efficiency when the data scale is large.

[0006] Vector retrieval is to retrieve the top k vectors closest to the query vector in a given vector data set. The core problem of vector retrieval is how to map data of different modalities to a unified feature space. Multi-modal joint representation is an effective means to solve this problem. Multi-modal learning is to use multi-modal data to mine the correlation between modalities and establish a related model for each modality to extract more discriminative feature representation. Traditional multi-modal representation mostly simply splices the features of each modality, ignoring the correlation information between modalities. Multi-modal learning methods can be divided into methods based on collaborative training and methods based on collaborative regularization. Through multi-modal learning, a unified feature space of modal data such as text, image, audio and video can be obtained, and the closest results can be recalled through vector similarity calculation. Vector retrieval relies more on the quality of joint feature representation in terms of retrieval accuracy.

[0007] At present, the multi-modal retrieval technology based on semantic labels is relatively mature, but there are deficiencies in the scene and event related multi-modal retrieval results. The multi-modal retrieval technology based on feature vectors is widely used in recent years, but the accuracy of the entity type multi-modal retrieval result still needs to be further improved. SUMMARY

[0008] The purpose of the present application is to overcome the deficiencies of the prior art, provide a dual-engine driven multi-modal data retrieval method, device and system, support one of text, image, video and audio modal data input to retrieve all modal data containing results, and effectively realize comprehensive and accurate recall of query results in different types of data such as text, image, audio and video.

[0009] The purpose of the present application is achieved by the following scheme:

[0010] A dual-engine driven multi-modal data retrieval method, comprising the following steps:

[0011] S1, multi-modal data intelligent processing: multi-modal intelligent recognition, taking multi-modal data as input, judging the semantic label of multi-modal data through intelligent recognition method, and constructing semantic label library through inverted index method; multi-modal feature extraction, constructing a unified feature space, extracting feature vectors of different modal data in the same feature space, and constructing a feature vector library;

[0012] S2, dual-engine cross-modal retrieval: taking multi-modal retrieval conditions as input, obtaining the semantic label and feature vector of the retrieval condition through the multi-modal data intelligent processing method, based on the semantic label library and the feature vector library, using the cross-modal semantic label retrieval engine and the cross-modal feature vector retrieval engine to complete the query of multi-modal data meeting the retrieval conditions in the multi-modal database, and pushing the search results after sorting.

[0013] Further, the cross-modal semantic label retrieval engine comprises an inverted index construction step and a semantic label query step;

[0014] The inverted index construction forms an inverted index table through label dictionary construction and inverted list construction method;

[0015] The semantic label query is based on the inverted index table, combines the semantic label of the query condition obtained in step S1 multi-modal data intelligent processing, and completes the multi-modal data search meeting the query condition through label hash coding and label comparison.

[0016] Further, the cross-modal feature vector retrieval engine comprises a hierarchical navigable graph construction step and a feature vector query step;

[0017] The hierarchical navigable graph construction forms a hierarchical navigation graph through near neighbor node number setting, bottom layer near neighbor graph construction, node level calculation, and top layer near neighbor graph construction method, providing a basis for fast feature vector retrieval;

[0018] The feature vector query is based on the hierarchical navigable graph, combines the feature vector of the query condition obtained in step S1 multi-modal data intelligent processing, finds the nearest node through feature vector similarity comparison from the top layer, and queries layer by layer downward according to the rule of gradually decreasing connection distance, to complete the multi-modal data search most similar to the query condition.

[0019] Further, in step S1, the multi-modal data comprises text, image, video and audio.

[0020] Further, in step S1, the intelligent recognition method comprises an intelligent recognition method based on deep neural network.

[0021] Further, in step S1, the intelligent recognition method comprises a key frame extraction method, a target detection method, a target recognition method, a character recognition method, an entity recognition method and a speech recognition method.

[0022] Further, in step S1, a unified feature space is constructed by a joint feature extraction method; and a feature vector library is constructed by a hierarchical navigable small world graph method.

[0023] Further, in step S1, the multi-modal feature extraction first constructs a joint representation model by a contrastive learning method, forms a unified feature space of multi-modal data, completes feature vector extraction on the multi-modal data, and stores the feature vector in a feature vector library.

[0024] A computer device comprising a processor and a memory, the memory storing a computer program, when the computer program is loaded and executed by the processor, the method according to any one of the above.

[0025] A dual-engine driven multi-modal data retrieval system comprising the computer device as described above.

[0026] The beneficial effects of the present application include:

[0027] (1) The technical scheme of the present application realizes multi-modal search under multi-source heterogeneous data, designs two modules of multi-modal data intelligent processing and dual-engine cross-modal retrieval, supports different modal data such as text, image, video and audio as query conditions, retrieves and recalls multi-modal data containing search targets as query results, and achieves the effect of searching all modal data in any modal.

[0028] (2) The technical scheme of the present application realizes comprehensive retrieval of coarse and fine granularity, proposes a dual-engine driven multi-modal retrieval method, realizes multi-modal data search of entity target class fine granularity through a cross-modal semantic label retrieval engine, realizes multi-modal data search of event scene class coarse granularity through a cross-modal feature vector retrieval engine, and provides more comprehensive multi-modal retrieval results through dual-engine combination. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical scheme in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0030] Figure 1A flowchart of a multi-modal data retrieval system driven by dual engines in an embodiment of the present application;

[0031] Figure 2 A flowchart of multi-modal data intelligent processing in an embodiment of the present application;

[0032] Figure 3 A flowchart of a cross-modal semantic label engine workflow in an embodiment of the present application;

[0033] Figure 4 A flowchart of a cross-modal feature vector engine workflow in an embodiment of the present application. DETAILED DESCRIPTION

[0034] All features disclosed in the specification or in any document incorporated by reference herein, or in any method or process disclosed in the specification or in any document incorporated by reference herein, can be combined and / or extended in any way, unless otherwise expressly excluded according to state of the art.

[0035] In view of the limitations of single semantic label retrieval technology or vector retrieval technology in the background art, the existing cross-modal retrieval or multi-modal retrieval method has the technical problems of low retrieval accuracy and incomplete results. The technical solution of the present application aims to provide a solution to the technical problems of low retrieval accuracy and incomplete results existing in the current cross-modal retrieval or multi-modal retrieval method. After creative thinking, the inventors of the present application conceived a multi-modal data retrieval method driven by dual engines of semantic labels and feature vectors, which technically achieved the support of one of text, image, video, and audio modal data input to retrieve all modal data containing results, and effectively realized the comprehensive and accurate recall of query results in different types of data such as text, image, audio, and video.

[0036] As shown in Figure 1 In a further concept of the present application, a dual-engine driven multi-modal data retrieval method is provided, which includes two stages of multi-modal data intelligent processing and dual-engine cross-modal retrieval. In the multi-modal data intelligent processing stage, text, image, video, audio, and other multi-modal data are input, the semantic labels of the multi-modal data are judged by an intelligent recognition method based on a deep neural network, and a semantic label library is constructed by an inverted index method. A unified feature space is constructed by a joint feature extraction method, feature vectors of different modal data in the same feature space are extracted, and a feature vector library is constructed by a hierarchical navigable small-world graph method. In the dual-engine cross-modal retrieval stage, multi-modal retrieval conditions are input, the semantic labels and feature vectors of the retrieval conditions are obtained by a multi-modal data intelligent processing method, and based on the semantic label library and the feature vector library, a cross-modal semantic label retrieval engine and a cross-modal feature vector retrieval engine are used to query multi-modal data that meet the retrieval conditions in a multi-modal database. After automatic relevance sorting, the search results are pushed.

[0037] like Figure 2 As shown, in a further aspect of this invention, the technical solution of this embodiment designs a multimodal data intelligent processing flow. Taking multimodal data such as text, images, videos, and audio as processing objects, it performs two processes: multimodal intelligent recognition and multimodal feature extraction. Multimodal intelligent recognition determines semantic tags for multimodal data, such as ship, aircraft, and person categories, and stores them in a semantic tag library using an inverted index method. Multimodal feature extraction first constructs a joint representation model through a contrastive learning method, forming a unified feature space for multimodal data. Feature vectors are then extracted from the multimodal data and stored in a feature vector library.

[0038] In the specific implementation process, such as Figure 3 As shown, to address the technical problem of querying fine-grained multimodal data of target classes, in a further embodiment of this invention, a cross-modal semantic tag retrieval engine is designed, comprising two parts: inverted index construction and semantic tag-based query. Inverted index construction forms an inverted index table through methods such as tag dictionary construction and inverted list construction. Semantic tag-based query, based on the inverted index table, combines multimodal data intelligent processing to obtain semantic tags for query conditions. The multimodal data intelligent processing, through target detection and various intelligent recognition methods, can complete the fine-grained semantic tag recognition of entities, targets, etc., in the query conditions. Semantic tag-based query uses a hash algorithm to encode the semantic tags of the query conditions and the tags in the semantic tag library, and through tag encoding comparison, completes the search for text, image, video, and audio data types in the database that fully match the fine-grained semantic tags of the query conditions.

[0039] In the specific implementation process, such as Figure 4As shown, in order to solve the technical problem of making up for the deficiency of semantic label retrieval in the search of scene class coarse-grained multi-modal data, in a further concept of the present application, a cross-modal feature vector retrieval engine is designed, including two parts of hierarchical navigable graph construction and feature vector type query. The hierarchical navigable graph construction forms a hierarchical navigable graph through methods such as near neighbor node number setting, bottom layer near neighbor graph construction, node level calculation, and top layer near neighbor graph construction, providing a basis for fast feature vector retrieval. Unlike the partial information of the query condition represented by the semantic label, the feature vector contains all useful information of the query condition, which can effectively represent the information of coarse-grained type data such as events and scenes. The feature vector type query is based on the hierarchical navigable graph, combines the feature vector of the query condition obtained through multi-modal data intelligent processing, finds the nearest node through feature vector similarity comparison from the top layer, and gradually queries down layer by layer according to the rule that the connection distance gradually decreases, and completes the search of multi-modal data most similar to the query condition. The feature vector similarity calculation is fast, and makes full use of all information of the data, which can more accurately retrieve coarse-grained multi-modal data such as events and scenes.

[0040] Embodiment 1

[0041] A dual-engine driven multi-modal data retrieval method, comprising the following steps:

[0042] S1, multi-modal data intelligent processing: multi-modal intelligent recognition, taking multi-modal data as input, judging the semantic label of the multi-modal data through intelligent recognition method, and constructing a semantic label library through inverted index method; multi-modal feature extraction, constructing a unified feature space, extracting feature vectors of different modal data in the same feature space, and constructing a feature vector library;

[0043] S2, dual-engine cross-modal retrieval: taking multi-modal retrieval condition as input, obtaining the semantic label and feature vector of the retrieval condition through multi-modal data intelligent processing method respectively, based on the semantic label library and the feature vector library, using the cross-modal semantic label retrieval engine and the cross-modal feature vector retrieval engine to complete the query of multi-modal data meeting the retrieval condition in the multi-modal database, and pushing the search results after sorting.

[0044] Embodiment 2

[0045] On the basis of embodiment 1, the cross-modal semantic label retrieval engine includes an inverted index construction step and a semantic label type query step;

[0046] The inverted index construction forms an inverted index table through label dictionary construction and inverted list construction methods;

[0047] The semantic label query is based on an inverted index table, and the semantic label of the query condition is obtained by combining the intelligent processing of the multi-modal data in step S1. Through label hash coding and label comparison, multi-modal data search conforming to the query condition is completed.

[0048] Embodiment 3

[0049] On the basis of embodiment 1, the cross-modal feature vector retrieval engine comprises a hierarchical navigable graph construction step and a feature vector query step.

[0050] The hierarchical navigable graph construction forms a hierarchical navigable graph through the near neighbor node number setting, bottom layer near neighbor graph construction, node level calculation, and top layer near neighbor graph construction method, providing a basis for fast feature vector retrieval.

[0051] The feature vector query is based on a hierarchical navigable graph, and the feature vector of the query condition is obtained by combining the intelligent processing of the multi-modal data in step S1. Starting from the top layer, the nearest node is found by comparing the feature vector similarity, and the query is gradually performed layer by layer downward according to the rule of gradually decreasing connection distance, completing the search of multi-modal data most similar to the query condition.

[0052] Embodiment 4

[0053] On the basis of embodiment 1, in step S1, the multi-modal data comprises text, image, video and audio.

[0054] Embodiment 5

[0055] On the basis of embodiment 1, in step S1, the intelligent recognition method comprises an intelligent recognition method based on a deep neural network.

[0056] Embodiment 6

[0057] On the basis of embodiment 1, in step S1, the intelligent recognition method comprises a key frame extraction method, a target detection method, a target recognition method, a character recognition method, an entity recognition method and a speech recognition method.

[0058] Embodiment 7

[0059] On the basis of embodiment 1, in step S1, a unified feature space is constructed by a joint feature extraction method; and a feature vector library is constructed by a hierarchical navigable small world graph method.

[0060] Embodiment 8

[0061] On the basis of embodiment 1, in step S1, the multi-modal feature extraction first constructs a joint representation model by a contrastive learning method, forms a unified feature space of multi-modal data, completes feature vector extraction of multi-modal data, and stores it in a feature vector library.

[0062] Embodiment 9

[0063] A computer device comprising a processor and a memory, the memory storing a computer program, when the computer program is loaded and executed by the processor, the method of any one of Embodiments 1-8 is executed.

[0064] Embodiment 10

[0065] A dual-engine driven multi-modal data retrieval system comprising the computer device of Embodiment 9.

[0066] The units described in the embodiments of the present application can be implemented by software, or by hardware, or by a combination of software and hardware. The units described may also be located in a single processor. In some cases, the units described may also be located in different processors.

[0067] According to an aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method provided in the various optional implementation manners.

[0068] As another aspect, the present application also provides a computer readable medium, which can be included in the electronic device described in the above embodiments, or can exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the method described in the above embodiments.

[0069] The parts of the present application not involved are the same as or can be implemented by the prior art.

[0070] The above technical solution is only one embodiment of the present application. For those skilled in the art, on the basis of the application disclosed application methods and principles, various types of improvements or modifications can be easily made, and are not limited to the methods described in the above embodiments. Therefore, the above description is only preferred, and is not limited in meaning.

[0071] In addition to the above examples, those skilled in the art can obtain other embodiments by being inspired by the above disclosure or by using knowledge or technology in related fields to make modifications. The features of each embodiment can be interchanged or replaced. The modifications and changes made by those skilled in the art do not deviate from the spirit and scope of the present application, and should be within the protection scope of the claims of the present application.

Claims

1. A dual engine driven multi-modal data retrieval method, characterized in that, Comprising the following steps: S1, multi-modal data intelligent processing: multi-modal intelligent recognition, taking multi-modal data as input, judging the semantic label of multi-modal data through intelligent recognition method, and constructing semantic label library through inverted index method; multi-modal feature extraction, constructing a unified feature space, extracting feature vectors of different modal data in the same feature space, and constructing a feature vector library; S2, dual-engine cross-modal retrieval: taking multi-modal retrieval conditions as input, obtaining the semantic label and feature vector of the retrieval condition through multi-modal data intelligent processing method, based on the semantic label library and the feature vector library, using the cross-modal semantic label retrieval engine and the cross-modal feature vector retrieval engine to complete the query of multi-modal data meeting the retrieval conditions in the multi-modal database, and pushing the search results after sorting; The cross-modal feature vector retrieval engine comprises a hierarchical navigable graph construction step and a feature vector query step; The hierarchical navigable graph construction forms a hierarchical navigation graph through near neighbor node number setting, bottom layer near neighbor graph construction, node level calculation and top layer near neighbor graph construction method, providing a basis for fast feature vector retrieval; The feature vector query is based on the hierarchical navigable graph, and the feature vector of the query condition is obtained by combining the multi-modal data intelligent processing in step S1, and the nearest node is found by comparing the feature vector similarity from the top layer, and the query is gradually performed layer by layer downward according to the rule that the connection distance gradually decreases, and the multi-modal data most similar to the query condition is searched.

2. The dual engine driven multi-modal data retrieval method as claimed in claim 1, wherein, The cross-modal semantic label retrieval engine comprises an inverted index construction step and a semantic label query step; The inverted index construction forms an inverted index table through label dictionary construction and inverted list construction method; The semantic label query is based on the inverted index table, and the semantic label of the query condition is obtained by combining the multi-modal data intelligent processing in step S1, and the multi-modal data meeting the query condition is searched through label hash coding and label comparison.

3. The dual engine driven multi-modal data retrieval method as claimed in claim 1, wherein, In step S1, the multi-modal data includes text, image, video and audio.

4. The dual engine driven multi-modal data retrieval method as claimed in claim 1, wherein, In step S1, the intelligent recognition method comprises a deep neural network-based intelligent recognition method.

5. The dual engine driven multi-modal data retrieval method as claimed in claim 1, wherein, In step S1, the intelligent recognition method comprises a key frame extraction method, a target detection method, a target recognition method, a character recognition method, an entity recognition method and a speech recognition method.

6. The dual engine driven multi-modal data retrieval method as claimed in claim 1, wherein, In step S1, a unified feature space is constructed by a joint feature extraction method; and the feature vector library is constructed by a hierarchical navigable small world graph method.

7. The dual engine driven multi-modal data retrieval method as claimed in claim 1, wherein, In step S1, the multi-modal feature extraction first constructs a joint representation model through a contrast learning method to form a unified feature space of multi-modal data, extracts feature vectors of multi-modal data, and stores them in the feature vector library.

8. A computer device, comprising: The computer device comprises a processor and a memory, and the memory stores a computer program, when the computer program is loaded and executed by the processor, the method of any one of claims 1-7 is executed.

9. A dual engine driven multi-modal data retrieval system characterized in that, The computer device of claim 8 is included.

Images