Retrieval index construction method, device and equipment and computer readable storage medium

Abstract

The invention relates to the technical field of data processing, and discloses a retrieval index construction method and device, equipment and a computer readable storage medium. The method comprises the following steps: respectively carrying out preprocessing and feature extraction on each piece of data in a multi-modal database to obtain a feature vector of each piece of data; performing cross-modal alignment processing on the feature vector of each piece of data to obtain a target feature vector corresponding to the feature vector of each piece of data in a target vector space; and constructing a target retrieval index based on the target feature vector of each piece of data. According to the method, cross-modal alignment processing is carried out on feature vectors of each piece of data in a multi-modal database, the feature vectors of the multi-modal data are converted into target feature vectors in a target vector space, and then a target retrieval index is constructed based on the target feature vectors; the feature loss caused by converting various modal data into text data is avoided, the accuracy of the constructed retrieval index is improved, and the multi-modal data retrieval precision can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a retrieval index construction method, apparatus, device and computer-readable storage medium. Background Art

[0002] Multimodal data retrieval refers to the cross-modal associative retrieval of multiple types of data such as text, images, audio, and video. It aims to break the limitations of a single modality and achieve semantic matching and information retrieval between data of different modalities.

[0003] The core of multimodal data retrieval is the construction of retrieval indexes. Existing retrieval index construction methods rely on textual conversion, that is, converting descriptions of data such as video, images, and audio into text data, and then constructing retrieval indexes based on the converted text data. However, reliance on textual conversion will, to a certain extent, lead to the loss of visual or audio features of data such as video, images, and audio, causing semantic deviation problems, resulting in reduced accuracy of the constructed retrieval index, and thus reduced precision of multimodal data retrieval. Summary of the Invention

[0004] In view of this, the purpose of this application is to overcome the deficiencies in the prior art and provide a method for constructing a search index, the method comprising:

[0005] Preprocess and extract features for each piece of data in the multimodal database to obtain the feature vector of each piece of data;

[0006] Performing cross-modal alignment processing on the feature vector of each piece of data to obtain a target feature vector corresponding to the feature vector of each piece of data in a target vector space;

[0007] Based on the target feature vector of each piece of data, a target retrieval index is constructed.

[0008] In one embodiment, the step of performing cross-modal alignment processing on the feature vector of each piece of data to obtain a target feature vector corresponding to the feature vector of each piece of data in a target vector space includes:

[0009] Determine a kernel matrix corresponding to each mode based on the kernel function corresponding to each mode and the eigenvectors of all data corresponding to each mode;

[0010] Centralizing the kernel matrix corresponding to each mode according to the order of the kernel matrix corresponding to each mode to obtain a centralized kernel matrix corresponding to each mode;

[0011] Based on the centralized kernel matrices corresponding to all modes, determine the projection vector corresponding to each mode in the target vector space;

[0012] Based on the projection vector corresponding to each modality and the eigenvectors of all data corresponding to each modality, a target eigenvector corresponding to the eigenvector of each piece of data corresponding to each modality in the target vector space is obtained.

[0013] In one embodiment, the step of determining the projection vector corresponding to each modality in the target vector space based on the centralized kernel matrices corresponding to all modalities includes:

[0014] Divide all modes into reference modes and non-reference modes;

[0015] Based on the centralized kernel matrix corresponding to each non-reference mode and the centralized kernel matrix corresponding to the reference mode, a projection vector corresponding to each non-reference mode in the target vector space and a projection vector corresponding to the reference mode in the target vector space are determined.

[0016] In one embodiment, the step of constructing a target retrieval index based on the target feature vector of each piece of data includes:

[0017] Clustering the target feature vectors of all data to obtain a cluster set;

[0018] Calculating the distance between the cluster centers of every two clusters in the cluster set, and sorting each cluster in the cluster set according to the distance;

[0019] Based on the sorting results and the cluster center corresponding to each cluster, a target retrieval index is constructed.

[0020] In one embodiment, the method further comprises:

[0021] When detecting that new data exists in the multimodal database, determining a new target feature vector corresponding to the new data in the target vector space;

[0022] Calculating the distance between the newly added target feature vector and each cluster center in the target retrieval index, and adding the newly added target feature vector to the cluster corresponding to the cluster center with the smallest distance;

[0023] Counting the number of target feature vectors of each node in the target retrieval index and the total number of target feature vectors of all nodes in the target retrieval index, if there is a node whose number of target feature vectors is greater than a first preset threshold, or if the total number of target feature vectors of all nodes is greater than a second preset threshold, reconstructing the target retrieval index.

[0024] In one embodiment, after the step of constructing a target retrieval index based on the target feature vector of each piece of data, the following steps are included:

[0025] Upon receiving the search data, determining a search target feature vector corresponding to the search data in the target vector space;

[0026] Determining a target candidate cluster set based on the retrieval target feature vector and the target retrieval index;

[0027] For each target candidate cluster in the target candidate cluster set, calculating a distance between the retrieval target feature vector and each target feature vector in the target candidate cluster set, and determining a target feature vector associated with the retrieval target feature vector based on the distance;

[0028] According to the target feature vector associated with the retrieval target feature vector, corresponding data is determined in the multimodal database and a retrieval result is returned.

[0029] In one embodiment, the step of determining a target candidate cluster set based on the retrieval target feature vector and the target retrieval index includes:

[0030] Calculating the distance between the retrieval target feature vector and each cluster center in the target retrieval index, and determining a candidate cluster set based on the distance;

[0031] Counting the frequency of each target feature vector in each candidate cluster being selected as a neighbor by the retrieval target feature vector;

[0032] Target feature vectors whose frequency in each candidate cluster is greater than a preset frequency are filtered to obtain a target candidate cluster set.

[0033] The present application also provides a retrieval index construction device, the retrieval index construction device comprising:

[0034] The first processing module is used to preprocess and extract features for each piece of data in the multimodal database to obtain a feature vector for each piece of data;

[0035] A second processing module is configured to perform cross-modal alignment processing on the feature vector of each piece of data to obtain a target feature vector corresponding to the feature vector of each piece of data in a target vector space;

[0036] A construction module is used to construct a target retrieval index based on the target feature vector of each data.

[0037] The present application also provides a computer device, which includes a processor and a memory, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the above-mentioned retrieval index construction method.

[0038] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is run on a processor, the computer program executes the above-mentioned retrieval index construction method.

[0039] The embodiments of the present application have the following beneficial effects:

[0040] The embodiment of the present application preprocesses and extracts features for each piece of data in a multimodal database to obtain a feature vector for each piece of data; performs cross-modal alignment on the feature vector of each piece of data to obtain a target feature vector corresponding to the feature vector of each piece of data in a target vector space; and constructs a target retrieval index based on the target feature vector of each piece of data. This method performs cross-modal alignment on the feature vector of each piece of data in a multimodal database, converts the feature vector of the multimodal data into a target feature vector in a target vector space, and then constructs a target retrieval index based on the target feature vector, thereby avoiding feature loss caused by converting various modal data into text data, improving the accuracy of the constructed retrieval index, and thus helping to improve the accuracy of multimodal data retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] To more clearly illustrate the technical solution of this application, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of this application and should not be considered as limiting the scope of protection of this application. Those skilled in the art can also derive other relevant drawings based on these drawings without inventive effort.

[0042] Figure 1 A flowchart of the first embodiment of the search index construction method provided by this application;

[0043] Figure 2 A flowchart of a second embodiment of the search index construction method provided by this application;

[0044] Figure 3 A flowchart of a third embodiment of the search index construction method provided by this application;

[0045] Figure 4 A flowchart of a fourth embodiment of the search index construction method provided by this application;

[0046] Figure 5 A flowchart of a fifth embodiment of the search index construction method provided by this application;

[0047] Figure 6 A flowchart of a sixth embodiment of the search index construction method provided by this application;

[0048] Figure 7This is a flowchart of the seventh embodiment of the search index construction method provided by this application;

[0049] Figure 8 This is a schematic diagram of the structure of the retrieval index construction device provided in this application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0051] The components of the embodiments of the present application generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but rather merely represents selected embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0052] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present application, are intended only to indicate specific features, numbers, steps, operations, elements, components or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing items or the possibility of adding one or more features, numbers, steps, operations, elements, components or combinations of the foregoing items.

[0053] Furthermore, the terms “first,” “second,” “third,” etc., are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0054] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meaning as in the context of the relevant technical field and will not be interpreted as having an idealized meaning or an overly formal meaning unless clearly defined in the various embodiments of the present application.

[0055] It is understandable that the method of the present application is applied to a multimodal data retrieval device, which may be a smart terminal, a PC terminal, a mobile terminal, etc., and is not limited here.

[0056] The following describes some embodiments of the present application in detail with reference to the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments may be combined with each other.

[0057] Please refer to Figure 1 , Figure 1 This is a flowchart of a first embodiment of the search index construction method provided by this application, the method comprising:

[0058] Step S101 : Preprocess and extract features for each piece of data in the multimodal database to obtain a feature vector for each piece of data.

[0059] In this embodiment, a large amount of multimodal data is stored in advance in the multimodal database in the multimodal data retrieval device. Each piece of data in the multimodal database is preprocessed and feature extracted to obtain a feature vector for each piece of data.

[0060] Among them, multimodal data refers to data of multiple modalities, and the modalities include: text, video, image, audio, etc. at least two. Correspondingly, multimodal data includes: text data, video data, image data, audio data, etc. at least two.

[0061] In one embodiment, during the preprocessing process, stop words, punctuation marks, etc. are removed from text data, blurred frames and noise samples are filtered from video data or image data, and silent segments and background noise are removed from audio data to ensure the quality of data in various modalities.

[0062] In one embodiment, during the feature extraction process, feature extraction is performed on each piece of data that has undergone the above preprocessing, and a feature vector of each piece of data is obtained. Among them, pre-trained language models such as BERT and ERNIE are used to extract semantic feature vectors of text data, retaining contextual semantic information; visual feature vectors of video data or image data are extracted through ResNet-50 and CLIP models, and visual representation vectors of video data or image data are extracted through the DINO self-supervised model; VGGish is used to extract acoustic feature vectors of audio data, or audio embedding vectors are generated through self-supervised learning. Furthermore, since video feature vectors are high-dimensional feature vectors compared to other vectors, principal component analysis (PCA) or random projection is used to reduce the dimension of video feature vectors to balance the representation capabilities between feature vectors corresponding to each modal data, while reducing the vector dimension can improve computational efficiency.

[0063] Step S102 : performing cross-modal alignment processing on the feature vector of each piece of data to obtain a target feature vector corresponding to the feature vector of each piece of data in a target vector space.

[0064] In this embodiment, after obtaining the feature vector corresponding to each data in the multimodal database, the multimodal data retrieval device performs cross-modal alignment processing on the feature vector of each data to obtain the target feature vector corresponding to the feature vector of each data in the target vector space.

[0065] Exemplarily, the multimodal data retrieval device first performs mean centering and variance normalization on the eigenvectors corresponding to all data of each modality to eliminate dimensional differences; then, the multimodal data retrieval device calculates the kernel matrix corresponding to each modality based on the corresponding kernel function and the eigenvectors of all data of the modality; finally, the multimodal data retrieval device takes a certain modality as the reference modality, pairs each of the remaining modalities with the reference modality, constructs an objective function that maximizes the correlation between the projected modalities based on the kernel matrices of the two paired modalities, and solves the generalized eigenvalue problem to determine the projection vector corresponding to each modality in the target vector space, and based on the projection vector corresponding to each modality and the eigenvectors of all data corresponding to each modality, obtains the target eigenvector corresponding to the eigenvector of each data corresponding to each modality in the target vector space.

[0066] Step S103: construct a target retrieval index based on the target feature vector of each piece of data.

[0067] In this embodiment, after obtaining the target feature vector of each piece of data, the multimodal data retrieval device constructs a target retrieval index based on the target feature vector of each piece of data. In one embodiment, the multimodal data retrieval adopts a semi-ordered linked list index construction technology to cluster all target feature vectors and generate a preset number of clusters. The cluster center of each cluster is used as a node in the target retrieval index, and the nodes are sorted by the Euclidean distance between the cluster centers to ensure that the cluster centers of adjacent nodes are closest in the cross-modal semantic space to construct the target retrieval index; wherein, each node in the target retrieval index contains multiple data, and the multiple data are arranged in disorder.

[0068] The multimodal data retrieval device of this embodiment preprocesses and extracts features from each piece of data in a multimodal database to obtain a feature vector for each piece of data; performs cross-modal alignment on the feature vector of each piece of data to obtain a target feature vector corresponding to the feature vector of each piece of data in a target vector space; and constructs a target retrieval index based on the target feature vector of each piece of data. This method performs cross-modal alignment on the feature vector of each piece of data in a multimodal database, converts the feature vector of the multimodal data into a target feature vector in a target vector space, and then constructs a target retrieval index based on the target feature vector. This avoids feature loss caused by converting various modal data into text data, improves the accuracy of the constructed retrieval index, and thus helps improve the accuracy of multimodal data retrieval.

[0069] Please refer to Figure 2 , Figure 2This is a flow chart of a second embodiment of the retrieval index construction method provided by the present application. The difference between the second embodiment and the first embodiment is that the step of performing cross-modal alignment processing on the feature vector of each piece of data to obtain the target feature vector corresponding to the feature vector of each piece of data in the target vector space includes:

[0070] Step S201 : determining a kernel matrix corresponding to each modality based on the kernel function corresponding to each modality and the eigenvectors of all data corresponding to each modality.

[0071] In this embodiment, after obtaining the feature vector corresponding to each piece of data in the multimodal database, the multimodal data retrieval device determines a kernel matrix corresponding to each modality based on the kernel function corresponding to that modality and the feature vectors of all data corresponding to that modality. In one embodiment, a polynomial kernel function can be used for text modalities, and an RBF kernel function can be used for video, audio, or image modalities.

[0072] In one embodiment, the formula for calculating the kernel matrix corresponding to the text modality is as follows:

[0073]

[0074] Among them, k(x i ,x j ) is the matrix unit of the kernel matrix corresponding to the text modality, x i , x j Represents the feature vector of the i-th text data and the feature vector of the j-th text data in the text modality. α is the weight of the adjustment parameter to scale the original feature inner product, affecting the strength of feature interaction. Usually α = 1 / p, p is the dimension of the feature vector of the text data, c is a constant term, usually 1 for text data, d is the highest order term, generally 2 or 3 for text data; in practical applications, it can be adjusted according to the data set situation.

[0075] In one embodiment, the formula for calculating the kernel matrix corresponding to the video modality, audio modality, or image modality is as follows:

[0076] k(x i , x j )=exp(-γ||x i -x j || 2 )

[0077] Among them, k(x i ,x j ) is the matrix unit in the kernel matrix corresponding to the video modality, audio modality or image modality, x i , x jRepresents the eigenvector of the i-th data and the j-th data in the video modality, audio modality, or image modality. γ controls the similarity decay rate. The smaller γ is, the smoother the model is. Generally, γ = 1 / p, where p is the dimension of the eigenvector of the video data, audio data, or image data.

[0078] Among them, the kernel matrix of each mode is as follows:

[0079]

[0080] Among them, K is the kernel matrix, and each matrix unit in the kernel matrix is ​​calculated by the above formula.

[0081] Step S202 : performing centralization processing on the kernel matrix corresponding to each mode according to the order of the kernel matrix corresponding to each mode, to obtain a centralized kernel matrix corresponding to each mode.

[0082] In this embodiment, after obtaining the kernel matrix corresponding to each modality, the multimodal data retrieval device performs centralization processing on the kernel matrix corresponding to each modality according to the order of the kernel matrix corresponding to each modality to obtain a centralized kernel matrix corresponding to each modality.

[0083] In one embodiment, the formula for calculating the centralized kernel matrix corresponding to each mode is:

[0084]

[0085] in, is the centralized kernel matrix, n represents the order of the kernel matrix K, 1 n represents a matrix of all ones.

[0086] Step S203 : Based on the centralized kernel matrices corresponding to all the modes, determine the projection vector corresponding to each mode in the target vector space.

[0087] In this embodiment, after obtaining the centralized kernel matrix corresponding to each modality, the multimodal data retrieval device takes a certain modality as the reference modality, pairs each of the remaining modalities with the reference modality, constructs an objective function that maximizes the correlation between the modalities after projection based on the centralized kernel matrices of the two paired modalities, and solves the generalized eigenvalue problem to determine the projection vector corresponding to each modality in the target vector space.

[0088] Step S204 : Based on the projection vector corresponding to each modality and the feature vectors of all data corresponding to each modality, a target feature vector corresponding to the feature vector of each piece of data corresponding to each modality in a target vector space is obtained.

[0089] In this embodiment, after obtaining the projection vector corresponding to each modality in the target vector space, the multimodal data retrieval device obtains the target feature vector corresponding to the feature vector of each data corresponding to each modality in the target vector space based on the projection vector corresponding to each modality and the feature vectors of all data corresponding to each modality.

[0090] In one embodiment, the multimodal data retrieval device first multiplies the centralized kernel matrix of each modality by the projection vector corresponding to each modality to obtain the target centralized kernel matrix corresponding to each modality in the target vector space, and then determines the target feature vector corresponding to the feature vector of each data corresponding to each modality in the target vector space based on the target centralized kernel matrix.

[0091] The multimodal data retrieval device in this embodiment determines the kernel matrix corresponding to each modality based on the kernel function corresponding to each modality and the eigenvectors of all data corresponding to each modality; performs centralization processing on the kernel matrix corresponding to each modality according to the order of the kernel matrix corresponding to each modality to obtain the centralized kernel matrix corresponding to each modality; determines the projection vector corresponding to each modality in the target vector space based on the centralized kernel matrix corresponding to all modalities; obtains the target eigenvector corresponding to the eigenvector of each piece of data corresponding to each modality in the target vector space based on the projection vector corresponding to each modality and the eigenvectors of all data corresponding to each modality. By converting the eigenvectors of multimodal data to the target eigenvectors in the target vector space, the feature loss caused by converting various modal data into text data is avoided, the accuracy of the cross-modal alignment processing of multimodal data is improved, and the accuracy of the constructed retrieval index is thereby improved.

[0092] Please refer to Figure 3 , Figure 3 This is a flow chart of the third embodiment of the retrieval index construction method provided in this application. The difference between the third embodiment and the first to second embodiments is that the step of determining the projection vector corresponding to each modality in the target vector space based on the centralized kernel matrix corresponding to all modalities includes:

[0093] Step S301: Divide all modes into reference modes and non-reference modes.

[0094] In this embodiment, the multimodal data retrieval device divides all modalities into baseline modalities and non-baseline modalities. In one embodiment, the multimodal data retrieval device can divide the base and non-baseline modalities based on user needs. For example, if the user needs to determine that text data accounts for a large proportion of the multimodal database, while video data, audio data, and image data account for a small proportion, the multimodal data retrieval device will classify the text modality as the baseline modality and the video modality, audio modality, and image modality as non-baseline modalities.

[0095] Step S302 : Determine, based on the centralized kernel matrix corresponding to each non-reference mode and the centralized kernel matrix corresponding to the reference mode, a projection vector corresponding to each non-reference mode in the target vector space and a projection vector corresponding to the reference mode in the target vector space.

[0096] In this embodiment, the multimodal data retrieval device determines the projection vector corresponding to each non-baseline modality in the target vector space and the projection vector corresponding to the baseline modality in the target vector space based on the centralized kernel matrix corresponding to each non-baseline modality and the centralized kernel matrix corresponding to the baseline modality. In one embodiment, the multimodal database includes four modalities: text modality, video modality, audio modality, and image modality. The text modality is divided into the base modality, and the video modality, audio modality, and image modality are divided into non-baseline modalities. The multimodal data retrieval device first pairs the text modality and the video modality, and constructs an objective function that maximizes the correlation between the modalities after projection based on the centralized kernel matrix of the paired text modality and the centralized kernel matrix of the video modality, and solves the generalized eigenvalue problem, thereby determining the projection vector corresponding to the text modality in the target vector space and the projection vector corresponding to the video modality in the target vector space. Then, the multimodal data retrieval device pairs the text modality and the audio modality. , constructs an objective function for maximizing the inter-modal correlation after projection based on the paired text modality’s centralized kernel matrix, the audio modality’s centralized kernel matrix and the corresponding projection vector of the text modality in the target vector space, and solves the generalized eigenvalue problem to determine the corresponding projection vector of the audio modality in the target vector space; the multimodal data retrieval device pairs the text modality and the image modality, constructs an objective function for maximizing the inter-modal correlation after projection based on the paired text modality’s centralized kernel matrix, the image modality’s centralized kernel matrix and the corresponding projection vector of the text modality in the target vector space, and solves the generalized eigenvalue problem to determine the corresponding projection vector of the image modality in the target vector space.

[0097] In one embodiment, the objective function for maximizing the projected inter-modality correlation is:

[0098]

[0099] Among them, η X , η Y is the regularization coefficient, I is the identity matrix, is the centralized kernel matrix corresponding to the benchmark mode, is the centralized kernel matrix corresponding to the non-reference mode, α is the projection vector corresponding to the reference mode in the target vector space, and β is the projection vector corresponding to the non-reference mode in the target vector space.

[0100] In one embodiment, the formula for solving the generalized eigenvalue problem is:

[0101]

[0102] Among them, C XY =K X K Y , where C XY It is to solve the cross covariance matrix constructed for convenience.

[0103] The multimodal data retrieval device of this embodiment divides all modalities into a reference modality and a non-reference modality. Based on the centralized kernel matrix corresponding to each non-reference modality and the centralized kernel matrix corresponding to the reference modality, the projection vector corresponding to each non-reference modality in the target vector space and the projection vector corresponding to the reference modality in the target vector space are determined. Using a particular modality as a reference can improve the efficiency of determining the projection vector corresponding to each modality in the target vector space.

[0104] Please refer to Figure 4 , Figure 4 This is a flow chart of a fourth embodiment of the retrieval index construction method provided by the present application. The fourth embodiment differs from the first to third embodiments in that the step of constructing a target retrieval index based on the target feature vector of each piece of data includes:

[0105] Step S401: cluster the target feature vectors of all data to obtain a cluster set.

[0106] In this embodiment, after determining the target feature vector of each data item in the multimodal database in the target vector space, the multimodal data retrieval device clusters the target feature vectors of all data items to obtain a cluster set. In one embodiment, the multimodal data retrieval device first initializes all target feature vectors into an initial cluster, then uses a K-means classification method to split the initial cluster into two sub-clusters. The sub-cluster with the larger dispersion is selected from the two sub-clusters, and the K-means classification method is used to split the sub-cluster into two sub-clusters again. This cycle is repeated until a preset number of clusters are obtained, thereby obtaining a cluster set.

[0107] Step S402 : calculating the distance between the cluster centers of every two clusters in the cluster set, and sorting each cluster in the cluster set according to the distance.

[0108] In this embodiment, after obtaining a cluster set, the multimodal data retrieval device calculates the mean vector of all target feature vectors in each cluster in the cluster set, and uses the mean vector as the cluster center of the cluster. The multimodal data retrieval device calculates the distance between the cluster centers of every two clusters in the cluster set and sorts each cluster in the cluster set based on the distance to ensure that the cluster centers of adjacent clusters are closest in the target vector space after sorting, where the target vector space is a cross-modal semantic space.

[0109] Step S403: construct a target search index based on the sorting result and the cluster center corresponding to each cluster.

[0110] In this embodiment, after sorting the clusters, the multimodal data retrieval device constructs a target retrieval index based on the sorting results and the cluster centers corresponding to each cluster. Each cluster center serves as a node in the target retrieval index, and each node in the target retrieval index contains multiple data, which are arranged in a disordered manner.

[0111] The multimodal data retrieval device of this embodiment determines the target feature vector of each data item in the target vector space in the multimodal database, then clusters the target feature vectors of all data items to form a cluster set. The device then calculates the distance between the cluster centers of each pair of clusters in the cluster set and sorts each cluster in the cluster set based on the distance. A target retrieval index is constructed based on the sorting results and the cluster centers corresponding to each cluster. This target retrieval index is constructed using a semi-ordered linked list index construction technique, improving the efficiency and accuracy of index construction.

[0112] Please refer to Figure 5 , Figure 5 This is a flowchart of the fifth embodiment of the search index construction method provided by the present application. The difference between the fifth embodiment and the first to fourth embodiments is that the method further includes:

[0113] Step S501 : When it is detected that new data exists in the multimodal database, a new target feature vector corresponding to the new data in the target vector space is determined.

[0114] In this embodiment, after constructing a target retrieval index, the multimodal data retrieval device determines a new target feature vector corresponding to the new data in the target vector space if it detects that new data has been added to the multimodal database. The multimodal data retrieval device first preprocesses and extracts features on the new data to obtain a new feature vector corresponding to the new data. The device then determines the modality of the new data and, based on the projection vector of the modality in the target vector space, the new feature vector and the projection vector, determines the new target feature vector corresponding to the new data in the target vector space.

[0115] Step S502 : calculating the distance between the newly added target feature vector and each cluster center in the target search index, and adding the newly added target feature vector to the cluster corresponding to the cluster center with the smallest distance.

[0116] In this embodiment, the multimodal data retrieval device calculates the distance from the newly added target feature vector to each cluster center in the target retrieval index, determines the target cluster as the cluster corresponding to the cluster center with the smallest distance to the newly added target feature vector, and adds the newly added target feature vector to the target cluster.

[0117] Step S503, counting the number of target feature vectors of each node in the target retrieval index and the total number of target feature vectors of all nodes in the target retrieval index, if there is a node whose number of target feature vectors is greater than a first preset threshold, or if the total number of target feature vectors of all nodes is greater than a second preset threshold, the target retrieval index is reconstructed.

[0118] In this embodiment, the multimodal data retrieval device counts the number of target feature vectors for each node in the target retrieval index and the total number of target feature vectors for all nodes in the target retrieval index. If there is a node whose number of target feature vectors is greater than a first preset threshold, or if the total number of target feature vectors for all nodes is greater than a second preset threshold, the target retrieval index is reconstructed. In one embodiment, the process of reconstructing the target retrieval index is the same as the process of constructing the target retrieval index described above and is not further described here.

[0119] When the multimodal data retrieval device of this embodiment detects new data, it adds the newly added target feature vector to the cluster corresponding to the cluster center with the smallest distance. The target retrieval index is rebuilt only when there are nodes with a number of target feature vectors greater than a first preset threshold, or when the total number of target feature vectors for all nodes exceeds a second preset threshold. In general, only a partial index can be updated, avoiding a full rebuild and improving the efficiency of target retrieval index updates.

[0120] Please refer to Figure 6 , Figure 6 This is a flow chart of a sixth embodiment of the retrieval index construction method provided by the present application. The sixth embodiment differs from the first to fifth embodiments in that, after the step of constructing a target retrieval index based on the target feature vector of each piece of data, the method includes:

[0121] Step S601: upon receiving search data, determining a search target feature vector corresponding to the search data in a target vector space.

[0122] In this embodiment, after completing the construction of the target retrieval index, the multimodal data retrieval device, upon receiving retrieval data input by the user, determines the retrieval target feature vector corresponding to the retrieval data in the target vector space. In one embodiment, the multimodal data retrieval device first preprocesses and extracts features on the retrieval data to obtain a retrieval feature vector corresponding to the retrieval data, then determines the modality of the retrieval data, and determines the retrieval target feature vector corresponding to the retrieval data in the target vector space based on the projection vector of the modality in the target vector space, the retrieval feature vector, and the projection vector.

[0123] In one embodiment, the retrieval data input by the user can be unimodal data or multimodal data; for example: when the retrieval data input by the user includes only text data, it is only necessary to determine the retrieval target feature vector corresponding to the text data in the target vector space; when the retrieval data input by the user includes only text data and video data, it is necessary to separately determine the retrieval target feature vector corresponding to the text data in the target vector space and the retrieval target feature vector corresponding to the video data in the target vector space.

[0124] Step S602: Determine a target candidate cluster set based on the retrieval target feature vector and the target retrieval index.

[0125] In this embodiment, the multimodal data retrieval device determines a target candidate cluster set based on the retrieval target feature vector and the target retrieval index. In one embodiment, the multimodal data retrieval device first selects a candidate cluster set based on the retrieval target feature vector and the target retrieval index, and then removes high-frequency common target feature vectors from each candidate cluster in the candidate cluster set to obtain the target candidate cluster set.

[0126] Step S603 : for each target candidate cluster in the target candidate cluster set, calculate the distance between the retrieval target feature vector and each target feature vector in the target candidate cluster set, and determine a target feature vector associated with the retrieval target feature vector based on the distance.

[0127] In this embodiment, the multimodal data retrieval device calculates the distance between the retrieval target feature vector and each target feature vector in the target candidate cluster set for each target candidate cluster in the target candidate cluster set, and determines the target feature vector associated with the retrieval target feature vector based on the distance.

[0128] Exemplarily, the target candidate cluster set includes three target candidate clusters, and the multimodal data retrieval device needs to respectively determine the target feature vectors associated with the retrieval target feature vector in the three target candidate clusters.

[0129] In one embodiment, the retrieval data input by the user can be unimodal data or multimodal data; when the retrieval data is unimodal data, for each target candidate cluster in the target candidate cluster set, the multimodal data retrieval device only needs to calculate the distance between the retrieval target feature vector and each target feature vector in the target candidate cluster set, and can determine the target feature vector associated with the retrieval target feature vector based on the distance. When the retrieval data is multimodal data, for each target candidate cluster in the target candidate cluster set, the multimodal data retrieval device needs to calculate the distance between the retrieval target feature vector of each modality and each target feature vector in the target candidate cluster set, then weighted average the calculated target distances, and then determine the target feature vector associated with the retrieval target feature vector based on the target distances.

[0130] Exemplarily, when the retrieval data input by the user includes only text data, the preset number is 4, and the target cluster includes 10 target feature vectors, the distances from the retrieval target feature vector to each target feature vector in the target candidate cluster are calculated to be 3 distances of 1, 1 distance of 1.5, 3 distances of 1.53, and 3 distances of 1.6. At this time, the 4 target feature vectors with distances of 1 and 1.5 are determined as target feature vectors associated with the retrieval target feature vector.

[0131] For example, when the retrieval data input by the user includes text data and video data, the preset number is 4, and the target cluster includes 10 target feature vectors, for one of the target feature vectors, the multimodal data retrieval device first calculates a first distance between the retrieval target feature vector corresponding to the text data in the target vector space and the target feature vector, where the first distance is assumed to be 2. The multimodal data retrieval device first calculates a second distance between the retrieval target feature vector corresponding to the video data in the target vector space and the target feature vector, where the second distance is assumed to be 3. When the weight of the text data is 0.9 and the weight of the video data is 0.1, the target distance between the retrieval data and the target feature vector is: 2*0.9+3*0.1=2.1. The distances between the retrieval data and each target feature vector in the target candidate cluster are 3 distances of 1.6, 1 distance of 2.1, 3 distances of 2.2, and 3 distances of 2.3. At this time, the four target feature vectors with distances of 1.6 and 2.1 are determined as target feature vectors associated with the retrieval target feature vector.

[0132] It should be noted that the weight value corresponding to each modality in the retrieval data is determined based on actual conditions and is not limited here.

[0133] Furthermore, to further speed up the search, we use HNSW (Hierarchical Navigable SmallWorld graphs) to accelerate the search: the HNSW graph structure is used to quickly locate the approximate nearest neighbor in the target cluster with the highest number of visits, and the target feature vector associated with the search target feature vector is returned. The specific steps are as follows:

[0134] Initialize the layers and randomly assign a level to the target feature vectors within the target cluster, following a Poisson distribution. A "small-world graph" is constructed layer by layer, starting from the highest level and constructing an undirected graph for each level from top to bottom. Each target feature vector is connected to its nearest neighbor nodes within its local range at the current level. Adjacency lists are used to record the adjacent target feature vectors and their corresponding distances at each level for each target feature vector. During retrieval, the search begins at the highest level, selecting any target feature vector from the highest level as the search starting point. The target feature vector x that is closest to the retrieval target feature vector is found. If the current level is the lowest, a refined search begins. Otherwise, x is used as the starting point for the next level. The current optimal neighbor set is continuously updated during retrieval, and the nearest neighbor set is returned as the result after the retrieval. During each jump, only adjacent target feature vectors with a distance less than the current optimal distance are retained to avoid invalid searches.

[0135] Step S604: According to the target feature vector associated with the search target feature vector, corresponding data is determined in the multimodal database and a search result is returned.

[0136] In this embodiment, after determining the target feature vectors associated with the retrieval target feature vector, the multimodal data retrieval device determines the data corresponding to these target feature vectors in the multimodal database and returns the retrieval results. It can be understood that the returned retrieval results may include text data, video data, image data and audio data.

[0137] The multimodal data retrieval device of this embodiment dynamically weights and fuses multimodal similarities during the retrieval process, using the HNSW graph structure to accelerate intra-cluster retrieval. Dynamically adjusting strategies based on query modality, combining filtering with graph structure acceleration, improves the efficiency of multimodal data retrieval.

[0138] Please refer to Figure 7 , Figure 7 This is a flow chart of the seventh embodiment of the retrieval index construction method provided in the present application. The difference between the seventh embodiment and the first to sixth embodiments is that the step of determining the target candidate cluster set based on the retrieval target feature vector and the target retrieval index includes:

[0139] Step S701 : Calculate the distance between the retrieval target feature vector and each cluster center in the target retrieval index, and determine a candidate cluster set based on the distance.

[0140] In this embodiment, the multimodal data retrieval device calculates the distance from the retrieval target feature vector to each cluster center in the target retrieval index, and determines the candidate cluster set based on the distance; in one embodiment, the multimodal data retrieval device sorts the distance from the retrieval target feature vector to each cluster center in the target retrieval index from small to large, and generates a candidate cluster set based on the clusters corresponding to the first preset number of cluster centers in the sorting results.

[0141] Step S702 : Counting the frequency of each target feature vector in each candidate cluster being selected as a neighbor by the search target feature vector.

[0142] In this embodiment, the multimodal data retrieval device counts the frequency of each target feature vector in each candidate cluster being selected as a neighbor by the retrieval target feature vector. In one embodiment, the KNN method is used to calculate and count the frequencies.

[0143] Step S703 , filtering the target feature vectors in each candidate cluster whose frequency is greater than a preset frequency to obtain a target candidate cluster set.

[0144] In this embodiment, the multimodal data retrieval device filters target feature vectors within each candidate cluster whose frequency exceeds a preset frequency, thereby obtaining a set of target candidate clusters. Specifically, target feature vectors whose frequency exceeds the preset frequency are determined to be normal target feature vectors and are filtered out. Target feature vectors whose frequency does not exceed the preset frequency are determined to be normal target feature vectors and borderline target feature vectors and are retained in the candidate cluster. This process is repeated for each candidate cluster to obtain a set of target candidate clusters.

[0145] The multimodal data retrieval device of this embodiment, after retrieving a set of candidate clusters, screens the target feature vectors in each candidate cluster, filters out common target feature vectors and retains boundary target feature vectors, which greatly reduces the target feature vectors with less correlation in the candidate clusters, facilitates subsequent rapid retrieval to obtain target feature vectors with correlation, and improves the efficiency of multimodal data retrieval.

[0146] refer to Figure 8 , Figure 8 : is a schematic diagram of the structure of the retrieval index construction device provided by this application. The retrieval index construction device includes:

[0147] The first processing module 10 is used to preprocess and extract features from each piece of data in the multimodal database to obtain a feature vector for each piece of data;

[0148] A second processing module 20 is configured to perform cross-modal alignment processing on the feature vector of each piece of data to obtain a target feature vector corresponding to the feature vector of each piece of data in a target vector space;

[0149] The construction module 30 is used to construct a target retrieval index based on the target feature vector of each data.

[0150] It can be understood that the retrieval index construction device of this embodiment corresponds to the retrieval index construction method of the above embodiment, and the optional items in the above embodiment are also applicable to this embodiment, so they will not be described again here.

[0151] The present application also provides a computer device. Exemplarily, the computer device includes a processor and a memory, wherein the memory stores a computer program, and the processor runs the computer program to enable the computer device to execute the functions of the various modules in the above-mentioned retrieval index construction method or the above-mentioned retrieval index construction device.

[0152] Among them, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU) and a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or at least one of other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application.

[0153] The memory may be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory is used to store a computer program, and the processor may execute the computer program accordingly after receiving an execution instruction.

[0154] The present application also provides a computer storage medium for storing the computer program used in the above-mentioned computer device. The computer storage medium may be a readable storage medium, a non-volatile storage medium, or a volatile storage medium. For example, the computer storage medium may include, but is not limited to, various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or flowchart, and the combination of boxes in the structure diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0156] In addition, the functional modules or units in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0157] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a smart phone, personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0158] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A method for constructing a retrieval index, characterized in that: The method comprises: Preprocess and extract features for each piece of data in the multimodal database to obtain the feature vector of each piece of data; Performing cross-modal alignment processing on the feature vector of each piece of data to obtain a target feature vector corresponding to the feature vector of each piece of data in a target vector space; Based on the target feature vector of each piece of data, a target retrieval index is constructed.

2. The method for constructing a search index according to claim 1, wherein: The step of performing cross-modal alignment processing on the feature vector of each piece of data to obtain a target feature vector corresponding to the feature vector of each piece of data in a target vector space includes: Determine a kernel matrix corresponding to each mode based on the kernel function corresponding to each mode and the eigenvectors of all data corresponding to each mode; Centralizing the kernel matrix corresponding to each mode according to the order of the kernel matrix corresponding to each mode to obtain a centralized kernel matrix corresponding to each mode; Based on the centralized kernel matrices corresponding to all modes, determine the projection vector corresponding to each mode in the target vector space; Based on the projection vector corresponding to each modality and the eigenvectors of all data corresponding to each modality, a target eigenvector corresponding to the eigenvector of each piece of data corresponding to each modality in the target vector space is obtained.

3. The method for constructing a search index according to claim 2, wherein: The step of determining the projection vector corresponding to each mode in the target vector space based on the centralized kernel matrix corresponding to all modes includes: Divide all modes into reference modes and non-reference modes; Based on the centralized kernel matrix corresponding to each non-reference mode and the centralized kernel matrix corresponding to the reference mode, a projection vector corresponding to each non-reference mode in the target vector space and a projection vector corresponding to the reference mode in the target vector space are determined.

4. The method for constructing a search index according to claim 1, wherein: The step of constructing a target retrieval index based on the target feature vector of each piece of data includes: Clustering the target feature vectors of all data to obtain a cluster set; Calculating the distance between the cluster centers of every two clusters in the cluster set, and sorting each cluster in the cluster set according to the distance; Based on the sorting results and the cluster center corresponding to each cluster, a target retrieval index is constructed.

5. The method for constructing a search index according to claim 4, wherein: The method further comprises: When detecting that new data exists in the multimodal database, determining a new target feature vector corresponding to the new data in the target vector space; Calculating the distance between the newly added target feature vector and each cluster center in the target retrieval index, and adding the newly added target feature vector to the cluster corresponding to the cluster center with the smallest distance; Counting the number of target feature vectors of each node in the target retrieval index and the total number of target feature vectors of all nodes in the target retrieval index, if there is a node whose number of target feature vectors is greater than a first preset threshold, or if the total number of target feature vectors of all nodes is greater than a second preset threshold, reconstructing the target retrieval index.

6. The method for constructing a retrieval index according to any one of claims 1 to 5, characterized in that: After the step of constructing a target retrieval index based on the target feature vector of each piece of data, the method further includes: Upon receiving the search data, determining a search target feature vector corresponding to the search data in the target vector space; Determining a target candidate cluster set based on the retrieval target feature vector and the target retrieval index; For each target candidate cluster in the target candidate cluster set, calculating a distance between the retrieval target feature vector and each target feature vector in the target candidate cluster set, and determining a target feature vector associated with the retrieval target feature vector based on the distance; According to the target feature vector associated with the retrieval target feature vector, corresponding data is determined in the multimodal database and a retrieval result is returned.

7. The method for constructing a search index according to claim 6, wherein: The step of determining a target candidate cluster set based on the retrieval target feature vector and the target retrieval index includes: Calculating the distance between the retrieval target feature vector and each cluster center in the target retrieval index, and determining a candidate cluster set based on the distance; Counting the frequency of each target feature vector in each candidate cluster being selected as a neighbor by the retrieval target feature vector; Target feature vectors whose frequency in each candidate cluster is greater than a preset frequency are filtered to obtain a target candidate cluster set.

8. A retrieval index construction device, characterized in that: The retrieval index building device includes: The first processing module is used to preprocess and extract features for each piece of data in the multimodal database to obtain a feature vector for each piece of data; A second processing module is configured to perform cross-modal alignment processing on the feature vector of each piece of data to obtain a target feature vector corresponding to the feature vector of each piece of data in a target vector space; A construction module is used to construct a target retrieval index based on the target feature vector of each data.

9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the retrieval index construction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is run on a processor, the method for constructing a retrieval index according to any one of claims 1 to 7 is executed.

Images