Quick retrieval method and device for multi-modal data and storage medium

Abstract

The invention discloses a quick retrieval method and device for multi-modal data and a storage medium. The quick retrieval method and device for the multi-modal data are used for quickly retrieving the multi-modal data. The method comprises the following steps: collecting a plurality of public data sets, wherein the data sets comprise data types of at least two different modalities; labeling the data set to generate labeling information related to the multi-modal data; performing feature extraction on the labeling information in the data set by using a multi-modal feature extraction model to generate a multi-modal feature vector; according to the feature vectors, a cross-modal feature fusion algorithm is used, weighted average of all modal feature vectors is calculated based on an attention mechanism, and fusion feature vectors are generated; according to the related multi-modal data, calculating the similarity of the feature vectors by using a weighted cosine similarity algorithm; retrieving related multi-modal data from the fused feature vectors through the similarity of the feature vectors according to the feature vectors input and queried by the user; and sorting the multi-modal data according to the correlation to obtain a retrieval result.

Description

Technical Field

[0001] The present application relates to the field of data retrieval, and in particular to a method, device and storage medium for rapid retrieval of multimodal data. Background Art

[0002] With the rapid development of information technology, the forms of data have become increasingly diverse. In today's digital environment, data in multiple modes such as images, text and audio have emerged in large quantities and are widely present in various fields. For example, in multimedia news reports, news pictures, news text content and possible interview audio are often included at the same time; in the field of education, electronic textbooks may have audio of explanations, text content in textbooks and related illustrations at the same time. Therefore, how to quickly retrieve the required information in multimodal data has become a key research topic.

[0003] In the prior art, data of different modalities are simply spliced ​​together to extract feature vectors of data of different modalities, and then the two feature vectors are directly connected to form a new vector, and then the two vectors are connected at the data level. In addition, some basic fusion strategies are adopted. This processing method ignores the differences in semantics and importance of data of different modalities, and directly adds the extracted feature vectors of different modalities. Simple addition fusion does not require complex algorithm design or a large number of parameter adjustments.

[0004] However, simply splicing multimodal data does not take into account the semantic differences and logical relationships between multimodal data. When calculating feature vectors, the retrieval system with a simple fusion strategy does not reasonably allocate the weights of each modality according to the specific circumstances of the multimodal data, resulting in the possibility that some modal information may be over-emphasized or ignored during the retrieval process, thereby affecting the accuracy and relevance of the retrieval results. Therefore, the existing retrieval methods are prone to poor relevance and accuracy of retrieval results, and cannot meet the user's needs for accurate and comprehensive retrieval in a multimodal data environment. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a method, device and storage medium for rapid retrieval of multimodal data.

[0006] The technical solution provided in this application is described below: The first aspect of the present application provides a method for rapid retrieval of multimodal data, the method comprising: Collect multiple public data sets, wherein the data sets contain data types of at least two different modalities, wherein the modalities include images, texts, and audios, and the data of each modality are interrelated; Annotating the data set to generate annotation information related to the multimodal data, the annotation information including image description, text label, and audio transcription; Using a multimodal feature extraction model to extract features from the annotation information in the data set to generate a multimodal feature vector, the model comprising a convolutional neural network, a bidirectional encoder, and a long short-term memory network; According to the feature vector, a cross-modal feature fusion algorithm is used to calculate the weighted average of the feature vectors of each modality based on the attention mechanism to generate a fused feature vector; Calculating the similarity of the feature vectors using a weighted cosine similarity algorithm based on the relevant multimodal data; According to the feature vector of the user input query, relevant multimodal data is retrieved from the fused feature vector by similarity of the feature vector; The multimodal data is sorted according to relevance to obtain search results.

[0007] Optionally, before using a multimodal feature extraction model to extract features from the annotation information in the data set to generate a multimodal feature vector, the model includes a convolutional neural network, a bidirectional encoder, and a long short-term memory network, including: The dataset is randomly divided into training set, validation set and test set.

[0008] Optionally, after randomly dividing the data set into a training set, a validation set, and a test set, the method further includes: Inputting the training set data into the convolutional neural network, the bidirectional encoder and the long short-term memory network for pre-training; Adjusting the model parameters of the convolutional neural network, the bidirectional encoder, and the long short-term memory network according to the pre-training results and the verification set; The test set is used to evaluate the multimodal feature extraction model after adjusting parameters.

[0009] Optionally, the multimodal feature extraction model is used to extract features from the annotation information in the data set to generate a multimodal feature vector, and the model includes a convolutional neural network, a bidirectional encoder, and a long short-term memory network, including: Load the image and input the preprocessed image into the convolutional neural network model to extract image features; Use a bidirectional encoder to encode the text and obtain the pooled vector as the text feature; Mel-frequency cepstral coefficient features are extracted, and the Mel-frequency cepstral coefficient features are input into a long short-term memory network model to extract audio features.

[0010] Optionally, the step of using a cross-modal feature fusion algorithm according to the feature vector to calculate a weighted average of the feature vectors of each modality based on an attention mechanism to generate a fused feature vector includes: Perform a linear transformation on each modality feature vector and apply a nonlinear activation function; Normalizing the linearly transformed feature vector to obtain the weight of each mode; The feature vector of each modality is multiplied by its corresponding weight and the sum is calculated to obtain a fused feature vector.

[0011] Optionally, the similarity of the feature vectors is calculated using a weighted cosine similarity algorithm, which is calculated using the following formula:

[0012] in, and is the feature vector, is the corresponding weight vector, n is the dimension of the feature vector, and is the eigenvector and The value in the i-th dimension, is the weight value associated with the i-th dimension feature, and the numerator represents the weighted inner product, the denominator and Calculate the norm of the weighted eigenvectors respectively.

[0013] Optionally, sorting the multimodal data by relevance to obtain search results includes: A sorting function is used to obtain a tuple of score and index, and the multimodal data is sorted from high to low according to the similarity score.

[0014] A second aspect of the present application provides a fast retrieval device for multimodal data, the fast retrieval device comprising: A collecting unit, configured to collect a plurality of public data sets, wherein the data sets include data types of at least two different modalities, wherein the modalities include images, texts, and audios, and the data of each modality are interrelated; A labeling unit, used to label the data set and generate labeling information related to the multimodal data, wherein the labeling information includes image description, text label and audio transcription; An extraction unit, configured to extract features from the annotation information in the data set using a multimodal feature extraction model to generate a multimodal feature vector, wherein the model includes a convolutional neural network, a bidirectional encoder, and a long short-term memory network; A first calculation unit is used to calculate the weighted average of the feature vectors of each modality based on the attention mechanism according to the feature vector and use a cross-modal feature fusion algorithm to generate a fused feature vector; A second calculation unit, configured to calculate the similarity of the feature vectors using a weighted cosine similarity algorithm according to the relevant multimodal data; A retrieval unit, configured to retrieve relevant multimodal data from the fused feature vector according to the feature vector of the user input query by similarity of the feature vector; The sorting unit is used to sort the multimodal data according to relevance to obtain search results.

[0015] The third aspect of the present application provides a fast retrieval device for multimodal data, the fast retrieval device comprising: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the first aspect and any optional method in the first aspect.

[0016] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the program executes the first aspect and any optional method in the first aspect.

[0017] It can be seen from the above technical solutions that this application has the following advantages: The retrieval method in this application reasonably allocates the weight of each mode according to the specific circumstances of the multimodal data, avoiding the over-emphasis or neglect of information of certain modes during the retrieval process. Make full use of the synergistic relationship between multimodal data to provide more relevant retrieval results and meet the user's needs for accurate and comprehensive retrieval in a multimodal data environment. In a multimodal data environment, whether it is multimedia news reports or various application scenarios such as electronic textbooks, users can be provided with comprehensive retrieval results covering multiple modalities such as images, texts, and audio. With a deeper understanding of the semantic features and logical relationships of multimodal data, the retrieval system can also provide personalized retrieval results based on the user's historical retrieval behavior and preferences. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A schematic diagram of an embodiment of a method for rapid retrieval of multimodal data provided by the present application; Figure 2 A schematic diagram of an embodiment of a flow chart of dividing a data set in a method for rapid retrieval of multimodal data provided in the present application; Figure 3 A schematic diagram of an embodiment of a flow chart of extracting data set features in a rapid retrieval method for multimodal data provided in the present application; Figure 4 A schematic diagram of an embodiment of a process for generating a fused feature vector in a fast retrieval method for multimodal data provided in the present application; Figure 5 A schematic diagram of an embodiment of sorting multimodal data by relevance in a fast retrieval method for multimodal data provided by the present application; Figure 6 A schematic diagram of the structure of a fast retrieval device for multimodal data provided by this application; Figure 7 Another structural schematic diagram of the device for rapid retrieval of multimodal data provided in this application. DETAILED DESCRIPTION

[0020] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0021] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0022] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0023] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0024] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0025] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0026] The present application provides a method for quickly retrieving multimodal data, which can quickly retrieve multimodal data. It should be noted that the method for quickly retrieving multimodal data of the present application is applied to a terminal.

[0027] See also Figure 1 The present application first provides an embodiment of a method for rapid retrieval of multimodal data, the embodiment comprising: S101. Collect multiple public data sets, where the data sets contain data types of at least two different modalities, including images, text, and audio, and the data of each modality are interrelated; The collected datasets cover at least two modal data types among images, text and audio to meet the needs of cross-modal research. For example, a dataset can contain both images and corresponding text descriptions, or audio and related text transcriptions. There should be a clear correlation between the modal data. The collected data sets must be publicly available and follow the corresponding open source agreements or data usage terms to ensure that the data can be used legally.

[0028] Specifically, use keywords to search on the corresponding websites and platforms, conduct preliminary screening of search results, check the introduction, documents and sample data of the dataset, determine whether it meets the requirements of multimodality and data relevance, further evaluate the quality, data format and use license of the dataset, and exclude datasets that do not meet the screening criteria. Establish a special storage directory for datasets that meet the standards, and classify and store them according to the category and source of the dataset to ensure orderly management of data.

[0029] S102, annotating the data set to generate annotation information related to the multimodal data, where the annotation information includes image description, text label, and audio transcription; A detailed and accurate text description of the image content in the dataset can clearly express the key information such as the objects, scenes, actions, attributes, etc. presented in the image, helping the model understand the semantic content of the image. Appropriate tags are added to the text data to classify and annotate the text's subject, category, and other features, facilitating the classification, retrieval, and analysis of text data. Audio data is converted into corresponding text content, accurately recording the voice information in the audio so that the audio data can be processed and analyzed in text form.

[0030] Specifically, for images, use concise and clear language to describe the image content. The description should be from the whole to the part, first describing the main scene or theme of the image, and then describing the specific objects and details. For text, select appropriate labels for the text according to the label classification system defined in the annotation guide. The label should be able to accurately reflect the characteristics and category of the text. For some complex texts, multiple labels can be used for annotation to more comprehensively describe the characteristics of the text. For audio, convert the voice content in the audio into the corresponding text content. When transcribing, the original words of the voice should be recorded as accurately as possible, including punctuation and modal particles.

[0031] The annotated image descriptions, text labels, and audio transcription information are sorted in correspondence with the original data to ensure the relevance of the annotated information and the data. The annotated data is stored according to a certain directory structure and naming rules to facilitate data management and retrieval.

[0032] S103, using a multimodal feature extraction model to extract features from the annotation information in the data set to generate a multimodal feature vector, the model including a convolutional neural network, a bidirectional encoder, and a long short-term memory network; Normalize the image and scale the pixel values ​​of the image to a specific range to improve the training effect and convergence speed of the model. At the same time, adjust the size and dimension of the image according to the input requirements of the model.

[0033] The text annotation information is segmented to divide the text into individual words or sub-word units. For image descriptions and audio transcriptions, word vector encoding can also be performed to convert the text into vector form so that the model can process it.

[0034] The audio transcribed text is subjected to the same preprocessing operations as the text data, including word segmentation and word vector encoding.

[0035] Specifically, the preprocessed image data is input into the convolutional neural network model. The dimension of the image data should meet the input requirements of the convolutional neural network model. The convolution layer of the convolutional neural network slides the convolution kernel on the image, performs convolution operations on the local area, and extracts the local features of the image. Parameters such as the size, step size, and number of convolution kernels can be adjusted according to the specific task and data set. The feature map output by the convolution layer is downsampled using the pooling layer to reduce the size of the feature map while retaining important feature information. Common pooling operations include maximum pooling and average pooling, which flatten the feature map processed by multiple convolutional layers and pooling layers into a one-dimensional vector and input it into the fully connected layer. The fully connected layer combines and maps the extracted local features to generate a high-level semantic feature vector of the image. The convolutional neural network model then outputs a feature vector of the image, whose dimension depends on the structure and parameter settings of the model.

[0036] The preprocessed text annotation information is input into the bidirectional encoder model. The text data is represented in the form of word vectors or word indices, and special markers need to be added to indicate the beginning and end of the text.

[0037] The bidirectional encoder model first performs word embedding on the input text data, converting each word into a low-dimensional vector representation. The word embedding vector contains the semantic information and contextual information of the word. The encoder of the bidirectional encoder consists of multiple Transformer blocks, each of which contains a multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism can simultaneously focus on the relationship between words at different positions in the text and capture the semantic information of the text; the feedforward neural network performs a nonlinear transformation on the output of the attention mechanism to further extract the features of the text. After being processed by multiple encoder layers, the bidirectional encoder model will output a feature vector at the beginning of the text, which contains the semantic information of the entire text. For shorter texts such as text labels, the output vector of the last layer can also be used directly as a feature representation.

[0038] The preprocessed audio transcription text data is input into the LSTM network model. The audio transcription text is represented in the form of a sequence, with each time step corresponding to a word or subword unit. Similar to text feature extraction, the LSTM network model first performs word embedding on the input audio transcription text and converts each word into a vector representation. The LSTM network layer processes the input sequence data through a gating mechanism, which can effectively capture the long-term dependencies and timing information in the audio text. At each time step, the LSTM network unit updates the hidden state and cell state at the current moment based on the current input and the hidden state at the previous moment. After being processed by multiple LSTM network layers, the hidden state of the last time step is taken as the feature vector of the audio transcription text. This feature vector contains the semantic and timing information of the audio text.

[0039] S104, using a cross-modal feature fusion algorithm according to the feature vector, calculating a weighted average of the feature vectors of each modality based on an attention mechanism, and generating a fused feature vector; The attention mechanism is a method that simulates human visual attention and can automatically assign different weights according to the importance of different parts of the input data. In cross-modal feature fusion, the attention mechanism is used to calculate the importance weight of each modal feature vector in the fusion process, thereby achieving adaptive fusion of different modal information. Based on the weights calculated by the attention mechanism, the feature vectors of each modality are weighted averaged, so that important modal features occupy a larger proportion in the fused feature vector. This weighted average method can make full use of information from different modalities, while highlighting the role of key modalities and improving the expressiveness of the fused feature vector.

[0040] Specifically, an attention network is designed to calculate the weight of each modal feature vector. The attention network is usually composed of multiple fully connected layers and activation functions. The sorted feature vectors of each modality are input into the attention network. The attention network obtains the attention score of each modality by performing nonlinear transformation and calculation on the feature vector, and normalizes the calculated attention score so that its sum is 1. The normalized attention weight is multiplied element by element with the corresponding modal feature vector to obtain the weighted modal feature vector. The weighted feature vectors of each modality are added to obtain the fused feature vector.

[0041] S105, calculating the similarity of feature vectors using a weighted cosine similarity algorithm according to relevant multimodal data; First, the different modal feature vectors of each sample in the multimodal data are sorted to ensure that the dimensions and data types of the feature vectors of different modalities are consistent. For example, the image feature vectors, text feature vectors, etc. are arranged in a matrix form according to the sample order, where each row represents the feature vector of a sample. Secondly, the weight vector W is determined according to specific needs. Each element of the weight vector corresponds to a feature in the feature vector, and its value can be set according to the importance of the feature.

[0042] The weighted cosine similarity algorithm is used to calculate the similarity of feature vectors, which is calculated by the following formula: ; in, and is the feature vector, is the corresponding weight vector, n is the dimension of the feature vector, and is the eigenvector and The value in the i-th dimension, is the weight value associated with the i-th dimension feature, and the numerator represents the weighted inner product, the denominator and Calculate the norm of the weighted eigenvectors respectively.

[0043] S106, according to the feature vector of the user input query, retrieve relevant multimodal data from the fused feature vector by similarity of the feature vector; S107. Sort the multimodal data by relevance to obtain search results.

[0044] The user provides a query feature vector according to his or her needs. The feature vector should have the same dimension and feature representation as the feature vector of the fused feature vector dataset to ensure effective similarity calculation. Specifically, each fused feature vector in the fused feature vector dataset is traversed, and its similarity with the query feature vector input by the user is calculated in turn. The similarity values ​​of each fused feature vector and the query feature vector calculated are sorted. The sorted results can be represented as an ordered list, in which each element contains the identifier of the fused feature vector and the corresponding similarity value. According to the sorted similarity results, the first N (N is the number of search results specified by the user) fused feature vectors with the highest similarity values ​​are selected. Through the identifier of the fused feature vector, the corresponding multimodal data is retrieved in the multimodal dataset. These multimodal data are the search results most relevant to the query feature vector input by the user.

[0045] In this embodiment, the retrieval method in this application reasonably allocates the weight of each mode according to the specific circumstances of the multimodal data, avoiding the over-emphasis or neglect of information of certain modes during the retrieval process. Make full use of the collaborative relationship between multimodal data to provide more relevant retrieval results and meet the user's needs for accurate and comprehensive retrieval in a multimodal data environment. In a multimodal data environment, whether it is multimedia news reports or various application scenarios such as electronic teaching materials, users can be provided with comprehensive retrieval results covering multiple modes such as images, texts, and audios. With a deeper understanding of the semantic features and logical relationships of multimodal data, the retrieval system can also provide personalized retrieval results based on the user's historical retrieval behavior and preferences.

[0046] See also Figure 2 , Figure 2 An embodiment of dividing a data set in a fast retrieval method for multimodal data provided by the present application includes: S201, randomly divide the data set into a training set, a validation set and a test set; First, determine the proportion of the training set, validation set, and test set. For example, the common proportion is 60% for the training set, 20% for the validation set, and 20% for the test set. Then use a random number generator to shuffle the index of the data set to ensure the randomness of the data. Finally, according to the set proportion, the data is allocated to different subsets in the order of the shuffled indexes.

[0047] It should be noted that the training set is used to train multimodal feature extraction models and cross-modal feature fusion algorithms, etc., so that the model can learn the features and patterns in the data. The validation set is used to evaluate the performance of the model during the model training process, adjust the model's hyperparameters, and monitor whether the model is overfitting. The test set is used to objectively evaluate the performance of the final model after the model training and hyperparameter adjustment are completed, reflecting the model's generalization ability on unseen data.

[0048] S202, inputting the training set data into the convolutional neural network, the bidirectional encoder, and the long short-term memory network for pre-training; The main goal of pre-training is to allow the model to learn general features and patterns in the data so that it can learn and predict faster and more accurately in subsequent specific tasks. Specifically, the goals of pre-training include but are not limited to: For convolutional neural networks, learn local features and spatial structure information in image data, such as edges, textures, etc. For bidirectional encoders, learn semantic information, lexical information, and contextual information in text data to better understand the meaning of the text. For long short-term memory networks, learn long-term dependencies and time series information in sequence data.

[0049] Pre-training is to initialize the convolutional neural network, bidirectional encoder and long short-term memory network respectively, and set the model's hyperparameters, such as learning rate, batch size, number of training rounds, etc. Specifically, image data, text data and audio sequences are extracted from the training set and input into the corresponding models respectively, and then the output of the model is calculated through forward propagation, the loss of the model is calculated according to the predefined loss function (such as the cross entropy loss function), and the back-propagation algorithm is used to update the model's parameters to minimize the loss function until the preset training effect is achieved.

[0050] S203, adjusting the model parameters of the convolutional neural network, the bidirectional encoder, and the long short-term memory network according to the pre-training results and the verification set; After loading the validation set data, input the validation set data into the pre-trained model to obtain the model's prediction results. According to the prediction results and the true labels, calculate relevant performance indicators, such as accuracy, recall, F1 value, etc. Record the performance of the initial model on the validation set, select the appropriate model parameter adjustment method to modify the model parameters according to the performance indicator analysis results, and retrain the model on the training set using the adjusted model parameters. Input the validation set data into the retrained model to obtain new prediction results. According to the new prediction results and the true labels, calculate the relevant performance indicators and compare them with the performance indicators of the initial model to determine whether the model performance has been improved. If the model performance has been improved, continue to adjust the model parameters according to the above steps until the model performance no longer improves or reaches the expected goal. If the model performance has not been improved, analyze the possible reasons and try other adjustment methods or parameters. During the model parameter adjustment process, record the performance of the model on the validation set after each adjustment. When the model performance reaches the best, save the model parameters at this time as the final optimization result.

[0051] S204. Use the test set to evaluate the multimodal feature extraction model after adjusting the parameters.

[0052] Load the multimodal feature extraction model after adjusting the parameters, ensure that the model weights and parameters have been loaded correctly, set the model to evaluation mode, that is, turn off some specific operations in the model used in the training process, traverse the test set data, input the multimodal data of each sample into the model for prediction, and record the output of the model, that is, the predicted value. According to the predicted results and the true labels of the test set data, calculate the value of each indicator according to the selected evaluation indicator calculation formula, analyze the calculated evaluation indicator results, compare the performance between different indicators, and evaluate the advantages and disadvantages of the model.

[0053] In this embodiment, the data set is randomly divided into a training set, a validation set, and a test set, which can effectively prevent the model from overfitting the training data during the training process, so that the model is trained, tuned, and evaluated on different data subsets, which helps to improve the generalization ability of the model. According to the results of pre-training and the validation set, the model parameters are adjusted to make the model better adapt to the needs of specific tasks. In addition, the multimodal feature extraction model after adjusting the parameters is evaluated using the test set, and the performance of the model on unseen data can be obtained, thereby accurately evaluating the generalization ability of the model.

[0054] See also Figure 3 , Figure 3 An embodiment of extracting data set features in a fast retrieval method for multimodal data provided by the present application includes: S301, loading an image, and inputting the preprocessed image into a convolutional neural network model to extract image features; Resize the loaded image to the size required for model input and convert the image's pixel values ​​to a specific range. Convolutional neural network models usually process data in batches, so a batch dimension needs to be added to the image data dimension. Then import the deep learning framework and model definition. When performing feature extraction, set the model to evaluation mode to ensure that the model's parameters are not updated during the inference process. According to the requirements of the deep learning framework, convert the preprocessed image data into the corresponding tensor type, input the image tensor into the model, obtain the model's output, and obtain the image's features based on the model's output structure.

[0055] S302, encoding the text using a bidirectional encoder, and obtaining a pooled vector as a text feature; Load the word segmenter and model of the bidirectional encoder separately. The word segmenter is used to segment the text into individual tokens, and the model is used to encode the text. Perform necessary preprocessing on the original text, such as removing extra spaces, punctuation marks, etc. Use the word segmenter to segment the preprocessed text into tokens, convert the tokens after word segmentation into corresponding token IDs, convert the token ID list into a tensor, and create an attention mask tensor at the same time. Input the input tensor and the attention mask tensor are input into the bidirectional encoder model for encoding. The model returns a tuple containing multiple outputs, where the first output is the hidden state of all tokens, and the second output is the pooled vector, which is the text feature.

[0056] S303, extracting Mel-frequency cepstral coefficient features, and inputting the Mel-frequency cepstral coefficient features into a long short-term memory network model to extract audio features.

[0057] The Mel-frequency cepstral coefficient features are extracted, and the shape of the Mel-frequency cepstral coefficient features is adjusted to a shape that meets the input requirements of the long short-term memory network model. The Mel-frequency cepstral coefficient features that meet the requirements are then converted into torch tensors. The preprocessed Mel-frequency cepstral coefficient tensor is input into the long short-term memory network model for forward propagation to obtain audio features.

[0058] In this embodiment, by extracting features from three different modalities, image, text, and audio, more comprehensive and rich information can be obtained. Data in different modalities have different characteristics and distributions. The combination of multimodal features can enable the model to learn more universal patterns and laws, thereby enhancing the generalization ability of the model in various complex scenarios. By extracting features from data in three different modalities, image, text, and audio, using different models, data in different modalities can be processed in a targeted manner, making the extracted features more accurate and highlighting important task-related information in multimodal data.

[0059] See also Figure 4 , Figure 4 An embodiment of generating a fused feature vector in a fast retrieval method for multimodal data provided by the present application includes: S401, performing a linear transformation on each modal feature vector and applying a nonlinear activation function; S402, normalizing the linearly transformed feature vector to obtain the weight of each mode; S403: Multiply the feature vector of each modality by its corresponding weight and sum them up to obtain a fused feature vector.

[0060] First, the original feature vector is mapped to a new feature space through linear transformation, so as to extract more representative and distinguishing features. A nonlinear activation function is introduced to increase the nonlinear expression ability of the model. Secondly, the feature vector after linear transformation and activation function processing is standardized so that its numerical range is within a reasonable range and the feature vectors of each mode are comparable. Finally, the feature vector of each mode is multiplied by its corresponding weight to weight its features according to the importance of each mode.

[0061] Specifically, the feature vectors extracted from image, text and audio modalities are expressed as: Fi: image feature vector Ft: text feature vector Fa: audio feature vector A simple weighted average method is used to dynamically adjust the weights of each modal feature, that is, a simple feedforward neural network is used to calculate an attention weight for each modal feature vector. Specifically, each modal feature vector is linearly transformed and a nonlinear activation function is applied:

[0062]

[0063]

[0064] in, , , is the weight matrix, , , is the bias term.

[0065] These results are then normalized to obtain the weight of each mode:

[0066]

[0067]

[0068] Finally, the eigenvector of each modality is multiplied by its corresponding weight and summed to obtain the fused eigenvector:

[0069] In this embodiment, the feature vector of each modality is multiplied by its corresponding weight and summed to obtain a fused feature vector. This weighted fusion method can fully consider the importance of each modality and effectively integrate information from different modalities. Data of different modalities often have different characteristics and advantages. Through reasonable weight distribution, the fused feature vector can contain both the key information of each modality and highlight the contribution of important modalities. For example, in the multimodal fusion of images and texts, the image modality may provide visual detail information, and the text modality may provide semantic description information. Through weighted fusion, the advantages of the two can be combined to obtain a more comprehensive and richer feature representation. The fusion of multimodal information can significantly improve the performance and accuracy of the model, making the model more capable and effective in handling complex tasks.

[0070] See also Figure 5 , Figure 5 An embodiment of sorting multimodal data by relevance in a method for rapid retrieval of multimodal data provided by the present application includes: S501. Use a sorting function to obtain a tuple of score and index, and sort the multimodal data from high to low according to the similarity score.

[0071] In order to ensure that the sorting function sorts the tuples in descending order of similarity scores, you need to specify a sorting rule. The sorting rule is usually based on the comparison of the score elements in the tuple, that is, the tuple is sorted in descending order according to the score of the first element. When using the sorting function, the sorting rule is specified to achieve descending order according to the similarity score. Extract the index information from the tuple list processed by the sorting function, and re-sort the original multimodal data according to the sorted index list, so that the multimodal data is arranged in descending order according to the similarity score.

[0072] This can be achieved through the following function: #Calculate the similarity score and construct a tuple of score and index score_index_tuples = [] for i, d in enumerate(data, start=1): score = cosine_similarity(d, target_vector) score_index_tuples.append((score, i)) # Sort the tuples using the sort function sorted_tuples = sorted(score_index_tuples, key=lambda x: x[0],reverse=True) #Get the sorted index sorted_indices = [t[1] for t in sorted_tuples] # Sort multimodal data sorted_data = [data[i - 1] for i in sorted_indices] print("Sorted multimodal data:", sorted_data) In this embodiment, by sorting from high to low according to the similarity score, the multimodal data that is most similar to the target or reference data can be placed in front. This allows the user or system to focus on the most relevant and valuable data first when processing a large amount of data, quickly obtain key information, and the sorted multimodal data can be classified and archived according to the similarity score, which facilitates data management and storage.

[0073] See also Figure 6 The present application provides an embodiment of a fast retrieval device for multimodal data, comprising: A collecting unit 601 is used to collect multiple public data sets, where the data sets include data types of at least two different modalities, where the modalities include images, texts, and audios, and the data of each modality are interrelated; Annotation unit 602, used to annotate the data set and generate annotation information related to the multimodal data, the annotation information including image description, text label and audio transcription; A partitioning unit 603 is used to randomly partition the data set into a training set, a validation set, and a test set; An input unit 604 is used to input the training set data into the convolutional neural network, the bidirectional encoder, and the long short-term memory network for pre-training; An adjustment unit 605 is used to adjust the model parameters of the convolutional neural network, the bidirectional encoder, and the long short-term memory network according to the pre-training results and the verification set; An evaluation unit 606, used to evaluate the multimodal feature extraction model after adjusting parameters using a test set; An extraction unit 607 is used to extract features from the annotation information in the data set using a multimodal feature extraction model to generate a multimodal feature vector, wherein the model includes a convolutional neural network, a bidirectional encoder, and a long short-term memory network; A first calculation unit 608 is used to calculate the weighted average of the feature vectors of each modality based on the feature vector and the attention mechanism using a cross-modal feature fusion algorithm to generate a fused feature vector; A second calculation unit 609 is used to calculate the similarity of feature vectors using a weighted cosine similarity algorithm according to the relevant multimodal data; A retrieval unit 610 is used to retrieve relevant multimodal data from the fused feature vector based on the similarity of the feature vector according to the feature vector input by the user; The sorting unit 611 is used to sort the multimodal data according to relevance to obtain search results.

[0074] Optionally, the extraction unit 607 is further configured to: Load the image and input the preprocessed image into the convolutional neural network model to extract image features; Use a bidirectional encoder to encode the text and obtain the pooled vector as the text feature; Extract the Mel-frequency cepstral coefficient features and input the Mel-frequency cepstral coefficient features into the long short-term memory network model to extract audio features.

[0075] Optionally, the first computing unit 608 is further configured to: Perform a linear transformation on each modality feature vector and apply a nonlinear activation function; Normalize the linearly transformed feature vector to obtain the weight of each mode; The feature vector of each modality is multiplied by its corresponding weight and summed to obtain the fused feature vector.

[0076] Optionally, a weighted cosine similarity algorithm is used to calculate the similarity of feature vectors, using the following formula: ; in, and is the feature vector, is the corresponding weight vector, n is the dimension of the feature vector, and is the eigenvector and The value in the i-th dimension, is the weight value associated with the i-th dimension feature, and the numerator represents the weighted inner product, the denominator and Calculate the norm of the weighted eigenvectors respectively.

[0077] Optionally, the sorting unit 611 is further configured to: Use the sorting function to obtain a tuple of score and index, and sort the multimodal data from high to low according to the similarity score.

[0078] See also Figure 7 , the present application provides another fast retrieval device for multimodal data, comprising: Processor 701, memory 702, input and output unit 703, bus 704; The processor 701 is connected to the memory 702, the input and output unit 703 and the bus 704; The memory 702 stores a program, and the processor 701 calls the program to execute any of the above methods.

[0079] The present application also relates to a computer-readable storage medium on which a program is stored, wherein when the program is run on a computer, the computer is caused to execute any of the above methods.

[0080] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0081] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0082] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0083] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0084] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A fast retrieval method for multimodal data, characterized in that: The method comprises: Collect multiple public data sets, wherein the data sets contain data types of at least two different modalities, wherein the modalities include images, texts, and audios, and the data of each modality are interrelated; Annotating the data set to generate annotation information related to the multimodal data, the annotation information including image description, text label, and audio transcription; Using a multimodal feature extraction model to extract features from the annotation information in the data set to generate a multimodal feature vector, the model comprising a convolutional neural network, a bidirectional encoder, and a long short-term memory network; According to the feature vector, a cross-modal feature fusion algorithm is used to calculate the weighted average of the feature vectors of each modality based on the attention mechanism to generate a fused feature vector; Calculating the similarity of the feature vectors using a weighted cosine similarity algorithm based on the relevant multimodal data; According to the feature vector of the user input query, relevant multimodal data is retrieved from the fused feature vector by similarity of the feature vector; The multimodal data is sorted according to relevance to obtain search results.

2. The method for rapid retrieval of multimodal data according to claim 1, characterized in that: Before using a multimodal feature extraction model to extract features from the annotation information in the data set to generate a multimodal feature vector, the model includes a convolutional neural network, a bidirectional encoder, and a long short-term memory network, including: The dataset is randomly divided into training set, validation set and test set.

3. The method for rapid retrieval of multimodal data according to claim 2, characterized in that: After randomly dividing the data set into a training set, a validation set, and a test set, the following steps are further included: Inputting the training set data into the convolutional neural network, the bidirectional encoder and the long short-term memory network for pre-training; Adjusting the model parameters of the convolutional neural network, the bidirectional encoder, and the long short-term memory network according to the pre-training results and the verification set; The test set is used to evaluate the multimodal feature extraction model after adjusting parameters.

4. The method for rapid retrieval of multimodal data according to claim 1, characterized in that: The multimodal feature extraction model is used to extract features from the annotation information in the data set to generate a multimodal feature vector, wherein the model includes a convolutional neural network, a bidirectional encoder, and a long short-term memory network, including: Load the image and input the preprocessed image into the convolutional neural network model to extract image features; Use a bidirectional encoder to encode the text and obtain the pooled vector as the text feature; Mel-frequency cepstral coefficient features are extracted, and the Mel-frequency cepstral coefficient features are input into a long short-term memory network model to extract audio features.

5. The method for rapid retrieval of multimodal data according to claim 1, characterized in that: The method of using a cross-modal feature fusion algorithm based on the feature vector to calculate the weighted average of the feature vectors of each modality based on the attention mechanism to generate a fused feature vector includes: Perform a linear transformation on each modality feature vector and apply a nonlinear activation function; Normalizing the linearly transformed feature vector to obtain the weight of each mode; The feature vector of each modality is multiplied by its corresponding weight and the sum is calculated to obtain a fused feature vector.

6. The method for rapid retrieval of multimodal data according to claim 1, characterized in that: The weighted cosine similarity algorithm is used to calculate the similarity of the feature vectors, and is calculated by the following formula: ； in, and is the feature vector, is the corresponding weight vector, n is the dimension of the feature vector, and is the eigenvector and The value in the i-th dimension, is the weight value associated with the i-th dimension feature, and the numerator represents the weighted inner product, the denominator and Calculate the norm of the weighted eigenvectors respectively.

7. The method for rapid retrieval of multimodal data according to claim 1, characterized in that: The step of sorting the multimodal data by relevance to obtain search results includes: A sorting function is used to obtain a tuple of score and index, and the multimodal data is sorted from high to low according to the similarity score.

8. A fast retrieval device for multimodal data, characterized in that: The quick retrieval device comprises: A collecting unit, configured to collect a plurality of public data sets, wherein the data sets include data types of at least two different modalities, wherein the modalities include images, texts, and audios, and the data of each modality are interrelated; A labeling unit, used to label the data set and generate labeling information related to the multimodal data, wherein the labeling information includes image description, text label and audio transcription; An extraction unit, configured to extract features from the annotation information in the data set using a multimodal feature extraction model to generate a multimodal feature vector, wherein the model includes a convolutional neural network, a bidirectional encoder, and a long short-term memory network; A first calculation unit is used to calculate the weighted average of the feature vectors of each modality based on the feature vector and the attention mechanism by using a cross-modal feature fusion algorithm to generate a fused feature vector; A second calculation unit, configured to calculate the similarity of the feature vectors using a weighted cosine similarity algorithm according to the relevant multimodal data; A retrieval unit, configured to retrieve relevant multimodal data from the fused feature vector according to the feature vector of the user input query by similarity of the feature vector; The sorting unit is used to sort the multimodal data according to relevance to obtain search results.

9. A fast retrieval device for multimodal data, characterized in that: The quick retrieval device comprises: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the fast retrieval method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon, wherein the program, when executed on a computer, performs the rapid retrieval method according to any one of claims 1 to 7.

Images