A multi-modal retrieval method based on deep learning and hash algorithm

Abstract

The application provides a multi-modal retrieval method based on deep learning and a hash algorithm, comprising: obtaining multi-modal data to be retrieved; performing text semantic retrieval on the multi-modal data based on a deep learning and hash coding method to obtain text data; performing picture retrieval on the multi-modal data based on a gray scale comparison method and an ElasticSearch technology to obtain picture data; mapping the text data and the picture data into a unified vector subspace by using a model based on a double encoder; obtaining a candidate set by calculating the similarity distance of the text data and the picture data in the vector subspace through a cosine similarity; and calculating the similarity on the candidate set by using a model based on a fusion encoder and performing sorting to obtain a retrieval result of the multi-modal data. By the method, the efficient and high-precision multi-modal retrieval task on a large-scale data set is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of data processing. BACKGROUND

[0002] With the continuous development of the Internet era, the data on the network is growing larger and larger, and the rapid popularization of intelligent terminal devices makes the multi-modal data on the Internet grow explosively, which has made it impossible to quickly find the required information through manual means. Therefore, how to quickly and efficiently and accurately retrieve the information required by users from the vast amount of multi-modal data information is a problem that needs to be solved.

[0003] The current mainstream multi-modal retrieval model architecture is based on an encoder, mainly including a model architecture based on a fusion encoder and a model architecture based on a double encoder.

[0004] The main idea of the model architecture based on the fusion encoder is to first convert the picture data and the text data into features, then perform feature fusion on the picture-text features, then input the picture-text features into the fusion encoder, let the model learn a function that can measure the cross-modal similarity, and finally use the picture-text similarity obtained by the function to realize cross-modal retrieval. This retrieval scheme needs to calculate the similarity between the user input data and all picture-text data in the database, and then sort the similarity to obtain the retrieval result. Therefore, when implementing the retrieval task on a large-scale data set, this scheme has problems such as low retrieval efficiency and large hardware resource consumption.

[0005] The main idea of the model architecture based on the double encoder is to use a picture encoder and a text encoder to encode the picture-text data respectively, map the picture-text data into a unified vector subspace, and finally realize cross-modal retrieval by calculating the cosine similarity. This retrieval scheme can pre-encode the picture-text data and construct a vector database, so it can realize large-scale picture-text retrieval tasks. However, this scheme cannot guarantee sufficient information interaction and information sharing between different modal data during model training, and the picture-text data cannot achieve high-quality "semantic alignment", so it has problems such as low retrieval accuracy.

[0006] The present application aims to solve at least one of the technical problems in the related art. SUMMARY

[0007] The present application aims to solve at least one of the technical problems in the related art.

[0008] To this end, the application aims to provide a multi-modal retrieval method based on deep learning and hash algorithm, which is used to realize efficient and high-precision multi-modal retrieval task on large-scale data sets.

[0009] To achieve the above-mentioned purpose, the application provides a multi-modal retrieval method based on deep learning and hash algorithm, which comprises the following steps of:

[0010] obtaining multi-modal data to be retrieved; performing text semantic retrieval on the multi-modal data based on a deep learning and hash coding method to obtain text data; performing picture retrieval on the multi-modal data based on a gray level comparison method and ElasticSearch technology to obtain picture data;

[0011] mapping the text data and the picture data into a unified vector subspace by using a model based on a double encoder;

[0012] obtaining a similarity distance of the text data and the picture data in the vector subspace by calculating a cosine similarity, and obtaining a candidate set;

[0013] calculating a similarity on the candidate set and performing sorting by using a model based on a fusion encoder to obtain a retrieval result of the multi-modal data.

[0014] In addition, the multi-modal retrieval method based on deep learning and hash algorithm according to the above-mentioned embodiments of the application can further have the following additional technical features:

[0015] Further, in an embodiment of the application, the deep learning and hash coding method for performing text semantic retrieval on the multi-modal data comprises the following steps of:

[0016] performing audio retrieval on the multi-modal data by using a voice recognition technology to obtain text data;

[0017] calculating a distance similarity of the text data and all texts in a database in a Hamming space, recalling L texts with the closest similarity distance to form a recall set;

[0018] calculating a distance similarity of a continuous vector obtained by a BERT pre-training model and a hash coding of the recall set, and taking a result with the smallest distance similarity as an output.

[0019] Further, in an embodiment of the application, the gray level comparison method and the ElasticSearch technology for performing picture retrieval on the multi-modal data comprise the following steps of:

[0020] extracting each frame of video data in the multi-modal data, mapping the frame into a unified LUV color space, and calculating an absolute distance of each frame from a previous frame;

[0021] According to the absolute distance, sort all the extracted frames, and the top several frames are the several pictures that can best represent the video content;

[0022] The picture data in the multi-modal data is converted into a matrix, and then picture retrieval is realized through ElasticSearch.

[0023] Further, in an embodiment of the present application, the use of a dual-encoder-based model to map the text data and the picture data into a unified vector subspace comprises:

[0024] On the basis of the multi-path Transformer pre-training model, a picture encoder and a text encoder are trained using a contrast learning method; the use of the contrast learning method to train the picture encoder and the text encoder comprises:

[0025] Linear transformation and regularization operation are realized in the encoder comprising a multi-head self-attention module, a visual feedforward network and a text feedforward network, to obtain picture vectors and text vectors for calculating the similarity from picture to text and from text to picture:

[0026]

[0027]

[0028] wherein represents the similarity between the picture in the i-th picture-text matching pair and the text in the j-th picture-text matching pair when going from picture to text, represents the similarity between the picture in the i-th picture-text matching pair and the text in the j-th picture-text matching pair when going from text to picture, and respectively represent the vector representation of the text in the i-th matching pair and the picture in the j-th matching pair after regularization, and σ is a temperature parameter, and are the similarities after regularization and softmax operation;

[0029] The contrast learning training of the model is realized using the similarity from picture to text and from text to picture and a cross-entropy loss function, to obtain the final picture encoder and text encoder.

[0030] Further, in an embodiment of the present application, the fine-tuning based on the multi-path Transformer pre-training model comprises:

[0031] Input the picture representation data into the pre-training model to train the visual feedforward network and the multi-head self-attention module;

[0032] Freeze the parameters in the obtained visual feedforward network and multi-head self-attention mechanism module, input the text representation data into the pre-training model, and train the text feedforward network using a masked language model self-supervised learning method;

[0033] The entire pre-training model is trained using the image-text matching pairs to obtain a final pre-training model for subsequent construction of a dual encoder and a fusion encoder.

[0034] Further, in an embodiment of the present application, the using a model based on the fusion encoder to calculate the similarity on the candidate set and perform sorting includes:

[0035] On the basis of the multi-path Transformer pre-training model, the model is optimized using a masking mechanism and a contrast learning method, and a cross-entropy loss function to obtain a final fusion encoder for a sorting task in image-text cross-modal retrieval.

[0036] To achieve the above object, the second aspect embodiment of the present application proposes a multi-modal retrieval device based on deep learning and hash algorithm, comprising the following modules:

[0037] The acquisition module is configured to acquire multi-modal data to be retrieved; a text semantic retrieval is performed on the multi-modal data based on a deep learning and hash coding method to acquire text data; and an image retrieval is performed on the multi-modal data based on a gray level comparison method and an ElasticSearch technology to acquire image data;

[0038] The mapping module is configured to map the text data and the image data into a unified vector subspace using a model based on a dual encoder;

[0039] The calculation module is configured to obtain a similarity distance of the text data and the image data in the vector subspace by calculating a cosine similarity to obtain a candidate set;

[0040] The sorting module is configured to calculate the similarity on the candidate set and perform sorting using a model based on a fusion encoder to obtain a retrieval result of the multi-modal data.

[0041] To achieve the above object, the third aspect embodiment of the present application proposes a computer device, characterized in that it comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a multi-modal retrieval method based on deep learning and hash algorithm as described above when executing the computer program.

[0042] To achieve the above object, the fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the multi-modal retrieval method based on deep learning and hash algorithm.

[0043] The multi-modal retrieval method based on deep learning and hash algorithm provided by the embodiments of the present application uses a pre-training model based on multi-path Transformer, so that different modal data can realize sufficient information interaction and information sharing during encoding. On the basis of the pre-training model, the present application constructs a model based on double encoders to map the image-text data into a unified vector subspace, realizes "rough recall" by calculating the cosine similarity, and guarantees the retrieval efficiency on large-scale data sets. Meanwhile, the present application constructs a model based on fusion encoders to realize "accurate sorting" of the k-candidate set, and guarantees the accuracy of retrieval. In addition, the present application provides a large-scale text single-modal semantic retrieval scheme based on deep learning and hash algorithm and a large-scale picture single-modal accurate retrieval scheme based on gray level comparison representation technology, which are organically combined with the image-text cross-modal technology, and finally realize large-scale, high-efficiency and high-performance multi-modal retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0044] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

[0045] Figure 1 A flowchart of a multi-modal retrieval method based on deep learning and hash algorithm provided by the embodiments of the present application.

[0046] Figure 2 A text semantic retrieval model architecture diagram based on deep learning and hash algorithm provided by the embodiments of the present application.

[0047] Figure 3 An image-text cross-modal retrieval system architecture diagram provided by the embodiments of the present application.

[0048] Figure 4 A model architecture diagram based on double encoders provided by the embodiments of the present application.

[0049] Figure 5 A model architecture diagram based on fusion encoders provided by the embodiments of the present application.

[0050] Figure 6 A multi-modal retrieval system architecture diagram based on deep learning and hash algorithm provided by the embodiments of the present application.

[0051] Figure 7A flowchart of a multi-modal retrieval device based on deep learning and a hash algorithm provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0052] Embodiments of the present application are described in detail below with reference to examples shown in the accompanying drawings, in which the same or similar components or components having the same or similar functions are denoted by the same or similar reference numerals throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0053] A multi-modal retrieval method based on deep learning and a hash algorithm of an embodiment of the present application is described below with reference to the accompanying drawings.

[0054] Figure 1 A flowchart of a multi-modal retrieval method based on deep learning and a hash algorithm provided by an embodiment of the present application is shown.

[0055] As shown in Figure 1 the multi-modal retrieval method based on deep learning and a hash algorithm includes the following steps:

[0056] S101: Obtain multi-modal data to be retrieved; perform text semantic retrieval on the multi-modal data based on a deep learning and hash coding method to obtain text data; perform picture retrieval on the multi-modal data based on a gray level comparison method and an ElasticSearch technology to obtain picture data;

[0057] S102: Map the text data and the picture data to a unified vector subspace using a model based on a double encoder;

[0058] S103: Obtain a similarity distance of the text data and the picture data in the vector subspace by calculating a cosine similarity, to obtain a candidate set;

[0059] S104: Calculate a similarity on the candidate set using a model based on a fusion encoder and perform sorting to obtain a retrieval result of the multi-modal data.

[0060] Further, in an embodiment of the present application, the deep learning and hash coding method for text semantic retrieval on the multi-modal data includes:

[0061] performing audio retrieval on the multi-modal data by a voice recognition technology to obtain text data;

[0062] calculate a distance similarity in a Hamming space of the retrieved text data and all texts in a database, and recall L texts with the closest similarity distance to form a recall set;

[0063] The distance similarity of the continuous vector obtained by the BERT pre-training model and the hash coding of the recall set is calculated, and the result with the minimum distance similarity is taken as the output.

[0064] Specifically, the application proposes a text semantic retrieval based on deep learning and hash algorithm. The BERT pre-training model is used to encode the text content into a continuous vector to extract the semantic information in the text, but this consumes a large amount of hardware resources and affects the retrieval efficiency, so the application constructs a hash coding layer based on the scalable tanh function, integrates it into the BERT pre-training model, and uses dense binary coding instead of continuous vectors, so that the retrieval efficiency is greatly improved without losing retrieval accuracy, and the required hardware resources are reduced. The text semantic retrieval model architecture is as shown in the figure. Figure 2

[0065] In order to improve the efficiency as much as possible without losing accuracy and reduce the consumption of hardware resources during retrieval, the application divides the retrieval process into two stages: "candidate set generation" and "re-ranking". In the "candidate set generation" stage, the application calculates the distance similarity of the retrieval input text and all text in the database in the Hamming space, and recalls the L closest text data. In the "re-ranking" stage, the application calculates the distance similarity of the continuous vector obtained by BERT and the hash coding of the recall set, and takes the K results with the minimum distance similarity as the output.

[0066] The application uses a hash coding-based method to recall the candidate set to ensure retrieval speed and reduce hardware resource consumption, and then uses a deep learning-based method to accurately sort the recalled candidate set to ensure retrieval accuracy.

[0067] Further, in an embodiment of the application, a gray level comparison-based method and ElasticSearch technology are used for picture retrieval of the multi-modal data, including:

[0068] Each frame of video data in the multi-modal data is extracted and mapped to a unified LUV color space, and the absolute distance of each frame from the previous frame is calculated.

[0069] All extracted frames are sorted according to the absolute distance, and the top several frames are the pictures that best represent the video content.

[0070] The picture data in the multi-modal data is converted into a matrix, and ElasticSearch is used to realize picture retrieval.

[0071] The application uses a gray level comparison-based method and ElasticSearch technology to realize large-scale and efficient picture retrieval tasks, and the details of this module are described below. ​

[0072] Firstly, the picture is divided into a 10x10 grid block, at this time there are 9x9 grid points on the picture; then, a 5x5 pixel rectangular region is fixed around each grid point and the average gray level thereof is calculated; for each rectangular region, an 8-element array is calculated, which respectively represents the comparison difference of the average gray level of the rectangular region and the surrounding 8 rectangular regions, and uses -2, -1, 0, 1 and 2 to represent the difference level of the average gray level. At this time, a picture can be represented as an 81x8 matrix.

[0073] The application uses the above picture feature extraction method to convert each picture into an 81x8 matrix, and then stores it in ElasticSearch to build a picture retrieval database. When a user inputs a picture, the picture is first converted into an 81x8 matrix, and then the powerful retrieval function of ElasticSearch is used to realize accurate picture retrieval.

[0074] The application uses video intelligent frame extraction technology to realize video retrieval task. Firstly, each frame of the video is extracted and mapped to a unified LUV color space, and the absolute distance between each frame and the previous frame is calculated. The greater the absolute distance, the more dramatic the change of the frame compared to the previous frame; finally, all the extracted frames are sorted according to the calculated absolute distance, and the top few frames are considered as the few pictures that best represent the video content, which are used to realize video retrieval.

[0075] The application constructs a double encoder and a fusion encoder based on a multi-path Transformer pre-training model to ensure that data of different modalities can realize sufficient information interaction and sharing. The model based on the double encoder is used to map the picture and text data to a unified vector subspace, and the cosine similarity is calculated to realize "rough recall", and then the model based on the fusion encoder is used to realize "accurate sorting", and finally the retrieval results are returned to the user for display. The architecture of the picture-text cross-modal retrieval system is shown in Figure 3 .

[0076] The application uses a multi-path Transformer pre-training model, which can realize sufficient information interaction and sharing of data of different modalities during encoding. The multi-path Transformer model is composed of a shared multi-head self-attention module and multiple feedforward networks. The visual feedforward network and the text feedforward network process picture data and text data respectively to realize the double encoder, and the visual-text feedforward network is used to process picture-text matching pairs to realize the fusion encoder. The architecture learns the features of different modalities of data through the shared multi-head self-attention module, and aligns the features between different modalities, so that the multi-modal information fusion is more closely.

[0077] During model training: 1) input the picture representation into the model, and train the visual feedforward network and the multi-head self-attention module; 2) freeze the parameters in the visual feedforward network and the multi-head self-attention module, and then input the text representation data into the model, and train the text feedforward network using the masked language modeling (MLM) self-supervised learning method; and 3) train the entire model using the image-text matching pairs to obtain a final pre-training model, which is used for subsequent construction of a dual encoder and a fusion encoder.

[0078] The present application constructs a dual-encoder-based model to encode image-text data respectively, and realizes the "rough recall" task of image-text cross-modal retrieval. The dual-encoder-based model architecture is shown in Figure 4 To realize the encoder, the present application fine-tunes a multi-path Transformer pre-training model, and trains the image encoder and the text encoder using the contrastive learning method.

[0079] Suppose there are N image-text matching pairs in a training batch. The goal of contrastive learning is to predict the correct image-text matching pair from N x N possible matching pairs, where only N pairs of image-text matching pairs are positive matching pairs, and N 2 N pairs of image-text matching pairs are negative matching pairs. The present application realizes linear transformation and regularization operations in the encoder containing the multi-head self-attention module, the visual feedforward network and the text feedforward network, to obtain the image vector and the text vector for calculating the similarity from image to text and from text to image.

[0080] Further, in an embodiment of the present application, the dual-encoder-based model is used to map the text data and the image data into a unified vector subspace, which includes:

[0081] On the basis of the multi-path Transformer pre-training model, the image encoder and the text encoder are trained using the contrastive learning method; training the image encoder and the text encoder using the contrastive learning method includes:

[0082] Realizing linear transformation and regularization operations in the encoder containing the multi-head self-attention module, the visual feedforward network and the text feedforward network, to obtain the image vector and the text vector for calculating the similarity from image to text and from text to image.

[0083]

[0084]

[0085] wherein, denotes the similarity between the image in the i-th image-text matching pair and the text in the j-th image-text matching pair from image to text, denotes the similarity between the image in the i-th image-text matching pair and the text in the j-th image-text matching pair from text to image, and denote the vector representations of the text in the i-th matching pair and the image in the j-th matching pair after regularization, respectively, and σ is a temperature parameter, and is the similarity after regularization and softmax operation;

[0086] The contrastive learning training of the model is implemented by using the similarities from image to text and from text to image and a cross-entropy loss function, and the final image encoder and text encoder are obtained.

[0087] The image encoder and the text encoder respectively encode the image and text data and map them into a common subspace, and then the similarity distance of the image and the text in the subspace is obtained by calculating the cosine similarity, so as to realize the cross-modal retrieval between images and texts. This scheme has the characteristics of high efficiency, and is used for the "rough recall" task in a large-scale image-text retrieval system.

[0088] Further, in an embodiment of the present application, fine-tuning is performed on the basis of a multi-path Transformer pre-training model, including:

[0089] The image representation data is input into the pre-training model, and the visual feedforward network and the multi-head self-attention module are trained;

[0090] The parameters in the obtained visual feedforward network and multi-head self-attention mechanism module are frozen, and then the text representation data is input into the pre-training model, and the text feedforward network is trained using a masked language model self-supervised learning method.

[0091] The image-text matching pair is used to train the entire pre-training model, and the final pre-training model is obtained, which is used for subsequent construction of a dual encoder and a fusion encoder.

[0092] Further, in an embodiment of the present application, a model based on the fusion encoder is used to calculate the similarity on the candidate set and perform sorting, including:

[0093] On the basis of the multi-path Transformer pre-training model, the model is optimized using a masking mechanism and a contrastive learning method, and a cross-entropy loss function, and the final fusion encoder is obtained, which is used for the sorting task in image-text cross-modal retrieval.

[0094] The model architecture based on the fusion encoder needs to jointly encode all possible combinations of image-text pairs, obtain a similarity score, reorder, and finally obtain the retrieval result when implementing cross-modal retrieval between images and texts. When the data volume is large, this scheme is too inefficient to be implemented, but the precision is higher than that of the model architecture based on the double encoder. Therefore, the fusion encoder is obtained by using the mask mechanism and the contrast learning method, and the cross-entropy loss function is used to fine-tune the model based on the multi-path Transformer pre-training model, which is used for the "precision ranking" task in cross-modal retrieval between images and texts. The model architecture based on the fusion encoder is as shown in Figure 5 .

[0095] The above is the complete multi-modal retrieval method based on deep learning and hash algorithm, Figure 6 which is the overall architecture diagram of the present application.

[0096] The multi-modal retrieval method based on deep learning and hash algorithm proposed in the embodiment of the present application uses a method based on deep learning and hash coding to implement a large-scale text semantic retrieval task, and uses a speech recognition technology to implement an audio retrieval task; uses a method based on gray level comparison and an ElasticSearch technology to implement a large-scale picture precision retrieval task, and uses a video intelligent frame extraction technology to implement a video retrieval task; uses a method based on a multi-path Transformer pre-training model and an encoder to implement a large-scale cross-modal retrieval between images and texts, while ensuring the precision and speed of retrieval. Compared with the current mainstream retrieval technology, the advantages of the present application are:

[0097] 1) The current retrieval method based on deep learning has high precision but low efficiency. The present application proposes a scheme combining the BERT pre-training model and the hash encoder based on the scalable tanh function to realize text semantic retrieval with unchanged precision and higher efficiency. Picture data has the characteristics of high dimensionality, so there is a problem of low efficiency when implementing picture retrieval. The present application proposes a scheme combining the gray level comparison method and the ElasticSearch technology to realize picture precision retrieval with higher precision and faster speed.

[0098] 2) The present application proposes a cross-modal retrieval scheme between images and texts based on a multi-path Transformer pre-training model and an encoder, which solves the problems of insufficient precision and low efficiency in the current cross-modal retrieval system between images and texts. The multi-path Transformer pre-training model can enable different modal data to realize sufficient information interaction and sharing. The present application first uses the pre-training model to construct a double encoder and a fusion encoder, then uses the double encoder to implement a "rough recall" task to obtain a k-candidate set, greatly improving the retrieval efficiency, and finally uses the fusion encoder to calculate the similarity on the k-candidate set to complete a "precision ranking" task, ensuring the retrieval precision.

[0099] To achieve the above embodiments, the present invention also proposes a multimodal retrieval device based on deep learning and hash algorithms.

[0100] Figure 7 This is a schematic diagram of a multimodal retrieval device based on deep learning and hash algorithm provided in an embodiment of the present invention.

[0101] like Figure 7 As shown, the multimodal retrieval device based on deep learning and hash algorithms includes: an acquisition module 100, a mapping module 200, a calculation module 300, and a sorting module 400, wherein...

[0102] The acquisition module is used to acquire multimodal data to be retrieved; perform text semantic retrieval on the multimodal data based on deep learning and hash encoding methods to acquire text data; and perform image retrieval on the multimodal data based on grayscale comparison methods and ElasticSearch technology to acquire image data.

[0103] The mapping module is used to map the text data and the image data to a unified vector subspace using a dual-encoder-based model;

[0104] The calculation module is used to obtain the similarity distance between text data and image data in the vector subspace by calculating cosine similarity, and to obtain the candidate set;

[0105] The ranking module is used to calculate similarity and rank the candidate set using a model based on a fusion encoder, so as to obtain the retrieval results of the multimodal data.

[0106] To achieve the above objectives, a third aspect of the present invention provides a computer device, characterized in that it includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the multimodal retrieval method based on deep learning and hash algorithm as described above.

[0107] To achieve the above objectives, a fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the multimodal retrieval method based on deep learning and hash algorithms as described above.

[0108] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0109] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0110] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and cannot be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. A multi-modal retrieval method based on deep learning and hash algorithm, characterized in that, The method comprises the following steps: acquiring multi-modal data to be searched; performing text semantic search on the multi-modal data based on a deep learning and hash coding method to acquire text data; performing picture search on the multi-modal data based on a gray level comparison method and an ElasticSearch technology to acquire picture data; mapping the text data and the picture data into a unified vector subspace using a dual-encoder-based model; obtaining a similarity distance of the text data and the picture data in the vector subspace by calculating a cosine similarity to obtain a candidate set; calculating a similarity on the candidate set using a fusion-encoder-based model and performing sorting to obtain a search result of the multi-modal data; wherein the step of mapping the text data and the picture data into a unified vector subspace using a dual-encoder-based model comprises: training a picture encoder and a text encoder using a contrast learning method on the basis of a multi-path Transformer pre-training model; the step of training the picture encoder and the text encoder using the contrast learning method comprises: Linear transformations and regularization operations are implemented in an encoder comprising a multi-headed self-attention module, a visual feedforward network, and a textual feedforward network, resulting in a picture vector and a textual vector for computing similarity from graph to text and from text to graph: ， ， ， wherein denotes the similarity between the image in the i-th image-text pair and the text in the j-th image-text pair from image to text, denotes the similarity between the image in the i-th image-text pair and the text in the j-th image-text pair from text to image, and denote the vector representation of the text in the i-th pair and the image in the j-th pair after regularization, respectively, is a temperature parameter, and is the similarity after regularization and softmax operation; implementing contrast learning training of the model by using a similarity from a picture to a text and a similarity from a text to a picture and a cross-entropy loss function to obtain the final picture encoder and the text encoder; the step of fine-tuning on the basis of the multi-path Transformer pre-training model comprises: inputting picture representation data into the pre-training model to train a visual feedforward network and a multi-head self-attention module; freezing the parameters of the visual feedforward network and the multi-head self-attention mechanism module, and then inputting text representation data into the pre-training model to train a text feedforward network using a masked language model self-supervised learning method; training the entire pre-training model using picture-text matching pairs to obtain the final pre-training model for subsequent construction of a dual encoder and a fusion encoder.

2. The method of claim 1, wherein, the step of performing text semantic search on the multi-modal data based on the deep learning and hash coding method comprises: performing audio search on the multi-modal data by a speech recognition technology to acquire text data; calculating a distance similarity in a Hamming space of the searched text data and all texts in a database, and recalling L texts with the closest similarity distance to form a recall set; calculating a distance similarity of a continuous vector obtained by a BERT pre-training model and hash coding of the recall set to obtain a result with the minimum distance similarity as an output.

3. The method of claim 1, wherein, the step of performing picture search on the multi-modal data based on the gray level comparison method and the ElasticSearch technology comprises: extracting each frame of video data in the multi-modal data, mapping the frame into a unified LUV color space, and calculating an absolute distance of each frame from a previous frame; sorting all the extracted frames according to the absolute distance, and the frames at the front of the sorting are a number of pictures that can best represent the video content; converting picture data in the multi-modal data into a matrix, and then performing picture search through ElasticSearch.

4. The method of claim 1, wherein, the step of calculating a similarity on the candidate set using a fusion-encoder-based model and performing sorting comprises: On the basis of a multi-path Transformer pre-training model, a mask mechanism and a contrast learning method, and a cross-entropy loss function are used to optimize the model to obtain a final fusion encoder for a ranking task in cross-modal retrieval of images and texts.

5. A multi-modal retrieval device based on deep learning and hash algorithm, characterized in that, The method comprises the following modules: An acquisition module is configured to acquire multi-modal data to be retrieved, perform text semantic retrieval on the multi-modal data based on a deep learning and hash coding method to obtain text data, and perform image retrieval on the multi-modal data based on a gray level comparison method and an ElasticSearch technology to obtain image data. A mapping module is configured to map the text data and the image data to a unified vector subspace by using a dual-encoder-based model. A calculation module is configured to obtain a similarity distance of the text data and the image data in the vector subspace by calculating a cosine similarity to obtain a candidate set. A ranking module is configured to calculate a similarity on the candidate set by using a fusion-encoder-based model and perform ranking to obtain a retrieval result of the multi-modal data. The mapping of the text data and the image data to the unified vector subspace by using the dual-encoder-based model comprises the following steps: On the basis of a multi-path Transformer pre-training model, a contrast learning method is used to train an image encoder and a text encoder. Linear transformations and regularization operations are implemented in an encoder comprising a multi-headed self-attention module, a visual feedforward network, and a textual feedforward network, resulting in a picture vector and a textual vector for computing similarity from graph to text and from text to graph: ， ， ， wherein denotes the similarity between the image in the i-th image-text pair and the text in the j-th image-text pair from image to text, denotes the similarity between the image in the i-th image-text pair and the text in the j-th image-text pair from text to image, and denote the regularized vector representation of the text in the i-th pair and the image in the j-th pair, respectively, is a temperature parameter, and is the similarity after regularization and softmax operation; The contrast learning method for training the image encoder and the text encoder comprises the following steps: The similarity from images to texts and from texts to images and a cross-entropy loss function are used to realize contrast learning training of the model to obtain the final image encoder and the text encoder. The fine-tuning on the basis of the multi-path Transformer pre-training model comprises the following steps: Image representation data is input into the pre-training model to train a visual feedforward network and a multi-head self-attention module. The parameters in the visual feedforward network and the multi-head self-attention module are frozen, and then text representation data is input into the pre-training model, and a masked language model self-supervised learning method is used to train a text feedforward network.

6. A computer device, comprising: Image-text matching pairs are used to train the entire pre-training model to obtain the final pre-training model for subsequent construction of a dual encoder and a fusion encoder.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the multi-modal retrieval method based on deep learning and a hash algorithm as claimed in any one of claims 1-5. The computer program is executed by the processor to implement the multi-modal retrieval method based on deep learning and a hash algorithm as claimed in any one of claims 1-5.

Images