Image search method based on multi-modal algorithm

Abstract

The invention relates to the technical field of information retrieval, in particular to an image search method based on a multi-modal algorithm, which comprises the following steps: S1, acquiring image data and corresponding text description data, and generating a text feature vector; s2, constructing a multi-modal feature alignment model, and outputting an alignment feature vector; s3, performing feature fusion processing on the aligned feature vectors output in the S2; s4, hierarchical clustering is carried out, and a hierarchical feature index structure is generated; s5, receiving query data input by a user, and generating a query feature vector; and S6, matching the query feature vector generated in the step S5 with the hierarchical feature index generated in the step S4. According to the cross-modal image search method and device, through the multi-modal feature alignment, deep fusion and hierarchical index construction technologies, the precision, efficiency and robustness of cross-modal image search are remarkably improved, and the problems that in the prior art, modals are inconsistent, relevance is insufficient, and the query speed is low are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of information retrieval, and in particular to an image search method based on a multimodal algorithm. Background Art

[0002] With the rapid growth of multimedia information, image search technology is increasingly used in the fields of artificial intelligence and big data. Traditional image search methods usually rely on single modal features, such as using convolutional neural networks to extract image features for visual search, or analyzing text descriptions through natural language processing technology for semantic retrieval. However, single modal features are difficult to fully express the association information between images and texts, especially when it comes to cross-modal scenarios, such as querying related images through text or finding corresponding descriptions through images. The accuracy and robustness of single modal search methods are difficult to meet the needs. At the same time, with the exponential growth of the scale of image and text data, how to efficiently process and match multimodal data has become an important technical challenge in the current image search field.

[0003] The existing technologies have the following main problems when processing multimodal data: first, image and text modal features are usually distributed in different feature spaces, and the existing methods lack an effective alignment mechanism, resulting in poor matching performance between features; second, the existing feature fusion methods are mostly simple splicing or average calculation, which fails to fully explore the correlation between image and text features; in addition, the fast matching and efficient index construction technologies for large-scale data sets are still imperfect, and the query process needs to traverse a large amount of feature data, resulting in a significant increase in query time. These problems significantly limit the practical application of multimodal image search technology. Therefore, there is an urgent need for an image search method that can achieve feature alignment, fully integrate multimodal features, and support efficient indexing and fast query to meet the application needs of large-scale multimodal data scenarios. Summary of the invention

[0004] Based on the above objectives, the present invention provides an image search method based on a multimodal algorithm.

[0005] An image search method based on a multimodal algorithm comprises the following steps: S1: Obtain image data and its corresponding text description data, and input the image data into a convolutional neural network to extract image feature vectors; perform word segmentation and embedding processing on the text description data to generate text feature vectors; S2: Build a multimodal feature alignment model, and input the image feature vector and text feature vector generated by S1 into the feature alignment model. Optimize the feature distribution through adversarial training, align the image feature vector and the text feature vector in the same vector space, and output the aligned feature vector. S3: Perform feature fusion processing on the aligned feature vector output by S2, use the bidirectional attention mechanism to calculate the weights between image features and text features, and generate a multimodal fusion feature vector based on the weighted calculation; S4: Perform hierarchical clustering on the multimodal fusion feature vector generated by S3, establish an index using a tree-based clustering method, map the fusion feature vector to the corresponding index node, and generate a hierarchical feature index structure; S5: receiving query data input by the user, and sequentially processing the query data through the same processing methods in S1, S2 and S3 above to generate a query feature vector; S6: Match the query feature vector generated by S5 with the hierarchical feature index generated by S4, calculate the similarity score between the query feature and the index feature, filter the candidate feature vectors according to the similarity score, and output the image result corresponding to the query data.

[0006] Optionally, the S1 specifically includes: S11: Acquire image data containing a target image and its corresponding text description data from an image database, wherein the image data is a digital image in RGB format, and the text description data is a natural language text related to the image content, and the two are stored in a one-to-one correspondence through data tags; S12: input the acquired image data into a pre-trained convolutional neural network for processing, and output an image feature vector of a fixed dimension; S13: segmenting the acquired text description data word by word according to lexical rules to form a word segmentation sequence containing a single word or phrase; S14: input the word segmentation sequence generated in S13 into a pre-trained embedding model, where the embedding model is built based on word vector representation technology, and uses a word embedding method to map each word into a vector of a fixed dimension, thereby generating a text feature vector of the text description data; S15: Normalize the image feature vector generated by S12 and the text feature vector generated by S14, and map the values ​​of all feature vectors to the range of [0,1].

[0007] Optionally, the S2 specifically includes: S21: constructing a multimodal feature alignment model, wherein the multimodal feature alignment model consists of a generator and a discriminator, wherein the generator is used to generate a feature representation consistent with the target modality, and the discriminator is used to distinguish between generated features and real features; S22: using the image feature vector and text feature vector generated in S1 as training data, wherein the image feature vector is a first modality feature of the model input, and the text feature vector is a second modality feature; S23: Conduct adversarial training. First, input the image feature vector into the generator to generate the alignment features of the target modality. At the same time, input the text feature vector and the features generated by the generator into the discriminator. The discriminator will update the discriminant function according to the distribution difference between the real features and the generated features. S24: After the adversarial training is completed, the image feature vector and text feature vector generated by S1 are input into the trained generator again to generate an aligned feature vector of the image modality and the text modality.

[0008] Optionally, the S23 specifically includes: S231: Input the image feature vector generated by S1 to the generator, which maps the input features through a deep neural network to obtain an aligned feature vector of the target modality; the output of the generator is represented by the following formula: ,in, is the image feature vector; Align features for the generated target modality; For the generator; is the parameter of the generator; S232: Input the real text feature vector generated by S1 and the alignment feature generated by the generator into the discriminator; the discriminator extracts the distribution characteristics of the input features through a convolutional neural network and outputs a probability value , used to indicate whether the input feature is a real feature; the output of the discriminator is expressed by the following formula: ,in, is the real text feature vector; Aligned feature vectors generated by the generator; is the feature of the input discriminator; is the discriminator; are the parameters of the discriminator; S233: The goal of the discriminator is to distinguish between real text features and generated features. Its loss function uses binary cross entropy loss, which is defined as: ,in, is the loss of the discriminator; For the expected operation; is a logarithmic operation; Output probability value for the discriminator; S234: The goal of the generator is to generate aligned features that are as close to the real text feature distribution as possible, and its loss function is defined as: ,in, is the loss of the generator; S235: Based on the min and max adversarial optimization strategies, the parameters of the generator and discriminator are updated using the stochastic gradient descent method: The expression for updating the discriminator parameters is: ; Update the generator parameter expression to: ;in, is the learning rate; This is the gradient calculation operation.

[0009] Optionally, the S3 specifically includes: S31: construct a bidirectional attention mechanism, and set the input to be the aligned feature vector output by S2, wherein the aligned feature vector includes an image feature vector and a text feature vector, wherein the image feature vector is used to express the visual features of the image, and the text feature vector is used to express the semantic features of the text; S32: Calculate the correlation between each image feature and all text features through the bidirectional attention mechanism, normalize the correlation, and generate a weight matrix corresponding to each image feature; S33: performing weighted processing on the text feature vector according to the weight matrix, performing weighted summation on all text features according to their weights with each image feature, and generating a weighted text feature vector associated with the image feature; S34: concatenate the image feature vector with the weighted text feature vector, and process the concatenation result through a nonlinear activation function to generate a multimodal fusion feature vector.

[0010] Optionally, the S4 specifically includes: S41: Analyze the multimodal fusion feature vector generated by S3, select the initial cluster center through the distance-based algorithm, and divide the entire feature vector space into multiple subspaces, each subspace corresponds to an initial cluster center; S42: Allocate the multimodal fusion feature vectors to the corresponding subspaces according to the distances from each cluster center; each feature vector belongs to the cluster center with the closest distance, completing the preliminary feature division; S43: clustering the feature vectors in each subspace again, and dividing each subspace into smaller subspaces by repeating the steps of selecting cluster centers and allocating features, until a preset hierarchical depth is reached or the number of subspace feature vectors meets a specified threshold; S44: according to the result of hierarchical clustering, a tree-like hierarchical structure is established for the subspace of each layer and the corresponding cluster center node; the root node represents the global feature space, and each leaf node represents a subspace; S45: Map the multimodal fusion feature vectors in the hierarchical clustering results to corresponding leaf nodes, each leaf node stores the feature vector belonging to the corresponding subspace, and forms a hierarchical feature index structure for subsequent query and matching.

[0011] Optionally, the S43 specifically includes: S431: Select new cluster centers in each subspace using a distance-based initialization method. Cluster centers; S432: According to the cluster center selected in S431, the feature vectors in the subspace are assigned to the cluster to which the closest cluster center belongs; the attribution of each feature vector is determined by the following formula: ,in, is the eigenvector in the current subspace; For the sub-clusters; is the cluster center; Represents the distance function, which is used to measure the similarity between the feature vector and the cluster center; S433: Recalculate the cluster center according to the feature vectors in each sub-cluster. The new cluster center is the mean of all feature vectors in the sub-cluster. The calculation formula is: ,in, For the Cluster centers; For the The number of eigenvectors in each subcluster; For the The feature vectors of the subclusters; S434: Repeat the above steps S431, S432 and S433 until the cluster centers converge or reach a preset recursive depth threshold, completing the recursive hierarchical clustering in the subspace.

[0012] Optionally, the S44 specifically includes: S441: Initialize the root node of the tree structure according to the hierarchical clustering result, the root node represents the global feature space, and the cluster center is used as the feature value of the root node; create a child node for each subspace, the child node represents the first-level subspace, and each child node stores the cluster center of the corresponding subspace; S442: clustering the feature vectors in each child node again, generating its child nodes under the child node, and recursively repeating the process until one of the following conditions is met; Condition 1: the number of eigenvectors in the current subspace is less than the preset threshold; Condition 2: reaching the preset maximum recursion depth; S443: In each level, the association weight between the parent node and its child nodes is calculated to represent the similarity relationship between the parent and child nodes; the weight value is calculated by the following formula: ,in, Represents the parent node and child nodes The association weight of Represents the parent node cluster center Cluster centers with child nodes distance; is the regularization parameter; S444: All feature vectors in the current subspace are stored in each leaf node, and the feature vectors in the leaf node are no longer divided. The path from each leaf node to the root node represents the storage path of the feature vector.

[0013] Optionally, the S6 specifically includes: S61: According to the query feature vector generated in S5, locate the root node of the hierarchical feature index in S4, and traverse the hierarchical tree structure starting from the root node; S62: in the candidate nodes of the current layer, calculating the similarity score between the query feature vector and each index feature vector in turn; S63: Based on the similarity scores calculated in S62, the top The index feature vectors are filtered into a set of candidate feature vectors; S64: If the number of candidate features in the current layer is less than the preset When the number of nodes is greater than 1, it will go back to the unselected child nodes in the previous layer, repeat steps S61 to S63, and continue to screen candidate features until the preset number conditions are met or all relevant nodes are traversed; S65: According to the final screened candidate feature vector set, the corresponding image results are extracted, sorted from high to low according to the similarity score, and the sorted image results are output as the final result of the query.

[0014] Optionally, the S65 specifically includes: S651: for each feature vector in the selected candidate feature vector set, query its corresponding image identifier in the hierarchical feature index; each feature vector in the candidate feature vector set corresponds to a unique image identifier one-to-one, and a candidate image identifier set is determined through a mapping relationship; S652: sorting the candidate image identifier set in descending order of similarity scores according to the similarity scores of each candidate feature vector calculated in S62; after sorting, prioritizing image identifiers having higher similarity scores with the query feature vector; S653: Retrieve corresponding image data from the image database according to the sorted image identifier set, each image identifier uniquely corresponds to an image, and the retrieved image data set is stored in sorted order to form an ordered image result set; S654: Outputting the sorted image result set as the final result of the query, and arranging the output results in descending order according to the similarity score with the query feature vector for the user to view or use.

[0015] Beneficial effects of the present invention: The present invention realizes the mapping of image modality and text modality features in the same vector space by constructing a multimodal feature alignment model, solves the problem of inconsistent modal feature distribution in the prior art, and utilizes the adversarial training mechanism of the generator and the discriminator to generate aligned features that are close to the real feature distribution, ensuring consistency and high correlation between features. At the same time, the aligned features are deeply fused through a bidirectional attention mechanism to effectively mine the correlation between image and text features, thereby generating multimodal fusion features with stronger expressive power. This feature fusion method improves the accuracy and robustness of cross-modal queries and provides a high-quality feature foundation for image search.

[0016] The present invention constructs a tree-structured feature index through a hierarchical clustering algorithm, which greatly improves the query efficiency. The hierarchical index structure can quickly locate the target feature subspace, and combines cosine similarity calculation and candidate feature screening mechanism to achieve efficient search in large-scale data scenarios. The query results finally output are sorted by relevance, ensuring the accuracy of the retrieval results and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic diagram of an image search method according to an embodiment of the present invention; Figure 2 Schematic diagram of a method for performing hierarchical clustering according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. At the same time, it is explained here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments, and those skilled in the art may also adopt other alternatives to implement some known technologies; and the accompanying drawings are only for more specific description of the embodiments, and are not intended to specifically limit the present invention.

[0020] It should be noted that the references to "one embodiment", "embodiment", "exemplary embodiments", "some embodiments" and the like in the specification indicate that the embodiments described may include specific features, structures or characteristics, but not every embodiment may include the specific features, structures or characteristics. In addition, when a specific feature, structure or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the art to implement such feature, structure or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0021] In general, a term can be understood, at least in part, from its use in context. For example, depending, at least in part, on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending, at least in part, on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0022] like Figure 1-Figure 2 As shown, an image search method based on a multimodal algorithm comprises the following steps: S1: Obtain image data and its corresponding text description data, and input the image data into a convolutional neural network to extract image feature vectors; perform word segmentation and embedding processing on the text description data to generate text feature vectors; S2: Build a multimodal feature alignment model, and input the image feature vector and text feature vector generated by S1 into the feature alignment model. Optimize the feature distribution through adversarial training, align the image feature vector and the text feature vector in the same vector space, and output the aligned feature vector. S3: Perform feature fusion processing on the aligned feature vector output by S2, use the bidirectional attention mechanism to calculate the weights between image features and text features, and generate a multimodal fusion feature vector based on the weighted calculation; S4: Perform hierarchical clustering on the multimodal fusion feature vector generated by S3, establish an index using a tree-based clustering method, map the fusion feature vector to the corresponding index node, and generate a hierarchical feature index structure; S5: receiving query data input by the user, and sequentially subjecting the query data to the same processing methods as in the feature extraction step in S1, the feature alignment step in S2, and the feature fusion step in S3, to generate a query feature vector, ensuring that the query feature vector and the hierarchical feature index in S4 are in the same vector space; S6: Match the query feature vector generated by S5 with the hierarchical feature index generated by S4, calculate the similarity score between the query feature and the index feature, filter the candidate feature vectors according to the similarity score, and output the image result corresponding to the query data.

[0023] S1 specifically includes: S11: Acquire image data containing a target image and its corresponding text description data from an image database, wherein the image data is a digital image in RGB format, and the text description data is a natural language text related to the image content, and the two are stored in a one-to-one correspondence through data tags; S12: Input the acquired image data into a pre-trained convolutional neural network for processing. The convolutional neural network includes multiple convolutional layers, pooling layers, and fully connected layers. First, the low-level features of the edge, texture, and color of the image are extracted through the convolutional layer, and then the feature dimension is reduced through the pooling layer. Finally, the image feature vector of fixed dimension is output in the fully connected layer. S13: Segmenting the acquired text description data word by word according to lexical rules to form a word segmentation sequence containing a single word or phrase; the word segmentation rule is constructed based on the dictionary and grammatical rules of the language to ensure the accuracy of the word segmentation sequence; S14: Input the word segmentation sequence generated in S13 into a pre-trained embedding model. The embedding model is built based on word vector representation technology. It uses a word embedding method (such as Word2Vec or BERT) to map each word into a vector of fixed dimension and generate a text feature vector of the text description data. S15: Normalize the image feature vectors generated by S12 and the text feature vectors generated by S14, map the values ​​of all feature vectors to the range of [0,1], and form a standardized feature data set for feature alignment in subsequent steps; by clearly dividing the steps of acquiring image data and text description data, feature extraction, and standardization, the standardization and consistency of feature extraction are ensured; the pre-trained convolutional neural network and text embedding model are used to effectively extract the low-level features of the image and the semantic information of the text, providing high-quality feature representation for subsequent feature alignment and feature fusion.

[0024] S2 specifically includes: S21: Construct a multimodal feature alignment model. The multimodal feature alignment model consists of a generator and a discriminator. The generator is used to generate feature representations consistent with the target modality, and the discriminator is used to distinguish generated features from real features. The generator uses a deep neural network, including an input layer, multiple hidden layers, and an output layer. The discriminator uses a convolutional neural network, including a convolutional layer and a classification layer. S22: Use the image feature vector and text feature vector generated in S1 as training data, where the image feature vector is the first modality feature of the model input, and the text feature vector is the second modality feature, and the two are aligned and stored through the label correspondence relationship; S23: Conduct adversarial training. First, input the image feature vector into the generator to generate the alignment features of the target modality. At the same time, input the text feature vector and the features generated by the generator into the discriminator. The discriminator will update the discriminant function according to the distribution difference between the real features and the generated features. Through the adversarial optimization of the generator and the discriminator, the distribution difference between the generated features and the real features is gradually reduced. S24: After the adversarial training is completed, the image feature vector and text feature vector generated by S1 are input into the trained generator again to generate the aligned feature vectors of the image modality and the text modality, and ensure that the aligned feature vectors are in a unified high-dimensional vector space for multimodal feature fusion in subsequent steps.

[0025] The adversarial training in S23 specifically includes: S231: The image feature vector generated by S1 is input to the generator, and the generator maps the input features through a deep neural network to obtain an aligned feature vector of the target modality; the output of the generator is expressed by the following formula: ,in, is the image feature vector, extracted by S1; Align features for the generated target modality; For the generator; is the parameter of the generator; S232: The real text feature vector generated by S1 and the alignment feature generated by the generator are input into the discriminator together; the discriminator extracts the distribution characteristics of the input features through the convolutional neural network and outputs the probability value , used to indicate whether the input feature is a real feature; the output of the discriminator is expressed by the following formula: ,in, is the real text feature vector, extracted by S1; Aligned feature vectors generated by the generator; is the feature of the input discriminator (can be or ); is the discriminator; are the parameters of the discriminator; S233: The goal of the discriminator is to distinguish between real text features and generated features. Its loss function uses binary cross entropy loss, which is defined as: ,in, is the loss of the discriminator; For the expected operation; is a logarithmic operation; Output probability value for the discriminator; S234: The goal of the generator is to generate aligned features that are as close to the real text feature distribution as possible, and its loss function is defined as: ,in, is the loss of the generator; S235: Based on the min and max adversarial optimization strategies, the parameters of the generator and discriminator are updated using the stochastic gradient descent method: The expression for updating the discriminator parameters is: ; Update the generator parameter expression to: ;in, is the learning rate; is a gradient calculation operation; through the above sub-steps, the adversarial training of the generator and the discriminator is utilized to make the generated aligned feature vector gradually approach the distribution characteristics of the real text features, ensuring that the image features and the text features are aligned in a unified vector space, thereby providing high-quality input features for multimodal feature fusion, and improving the accuracy of feature alignment and the robustness of the search system.

[0026] S3 specifically includes: S31: Construct a bidirectional attention mechanism and set the input to the aligned feature vector output by S2. The aligned feature vector includes an image feature vector and a text feature vector. The image feature vector is used to express the visual features of the image, and the text feature vector is used to express the semantic features of the text. The bidirectional attention mechanism is used to calculate the correlation weight between the two features. S32: Through the bidirectional attention mechanism, the correlation between each image feature and all text features is calculated, and the correlation is normalized to generate a weight matrix corresponding to each image feature. The weight matrix is ​​used to describe the relative importance between image features and text features. S33: performing weighted processing on the text feature vector according to the weight matrix, performing weighted summation on all text features according to their weights with each image feature, and generating a weighted text feature vector associated with the image feature; S34: concatenate the image feature vector with the weighted text feature vector, and process the concatenation result through a nonlinear activation function to generate a multimodal fusion feature vector, which is used to simultaneously express information of the image and text modalities.

[0027] The specific calculation steps for feature fusion processing are as follows: Initialization of the bidirectional attention mechanism: Construct a bidirectional attention mechanism to calculate the correlation weight between image features and text features; set the input to the image feature vector output by S2 and text feature vector ; Weight matrix calculation: Calculate the correlation weight matrix of image features and text features , the similarity of each pair of features is calculated by the following formula: ,in, is the image feature vector and text feature vector The normalized weight of is the first feature vector of the image Features is the first Features is the dimension of the text feature vector; represents the dot product operation; Exponential function; Weighted feature calculation: using weight matrix The text feature vector is weighted to obtain the weighted image-associated text feature vector. The calculation formula is: ;in, is the weighted image-associated text feature vector Features Multimodal feature fusion: The image feature vector and the weighted text feature vector are concatenated and processed through a nonlinear activation function to generate a multimodal fusion feature vector, which is expressed as: ,in, is the multimodal fusion feature vector; is the concatenation vector of image features and weighted text features; is a nonlinear activation function (such as ReLU or Tanh); the above steps calculate the correlation weights between image features and text features through a bidirectional attention mechanism, and use the weights to weightedly calculate text features, thereby effectively fusing image and text features; the generated multimodal fusion feature vector accurately expresses the correlation information of image and text modalities, and provides high-quality feature representation for index construction and query matching in subsequent steps.

[0028] S4 specifically includes: S41: Analyze the multimodal fusion feature vector generated by S3, select the initial cluster center through the distance-based algorithm, and divide the entire feature vector space into multiple subspaces, each subspace corresponds to an initial cluster center; S42: Allocate the multimodal fusion feature vectors to the corresponding subspaces according to the distances from each cluster center; each feature vector belongs to the cluster center with the closest distance, completing the preliminary feature division; S43: clustering the feature vectors in each subspace again, and dividing each subspace into smaller subspaces by repeating the steps of selecting cluster centers and allocating features, until a preset hierarchical depth is reached or the number of subspace feature vectors meets a specified threshold; S44: according to the result of hierarchical clustering, a tree-shaped hierarchical structure is established for the subspace of each layer and the corresponding cluster center node; the root node represents the global feature space, each leaf node represents a subspace, and the number of feature vectors in the leaf node meets the preset distribution condition; S45: Map the multimodal fusion feature vectors in the hierarchical clustering results to the corresponding leaf nodes, each leaf node stores the feature vector belonging to the corresponding subspace, and forms a hierarchical feature index structure for subsequent query and matching; the above steps divide the multimodal fusion feature vectors into different feature subspaces layer by layer through a hierarchical clustering method based on a tree structure, and store the index nodes in a tree structure, so that the feature space has good hierarchy and accessibility; the generated hierarchical feature index structure can quickly locate the index node corresponding to the query feature, greatly improving the efficiency and accuracy of query matching.

[0029] The clustering again in S43 specifically includes: S431: Select new cluster centers in each subspace using a distance-based initialization method. Cluster centers; The distance-based initialization method steps include: S4311: Randomly select a feature vector from the subspace feature vector as the first cluster center; S4312: For the remaining feature vectors, calculate their distances from the selected cluster center, and select the feature vector with the farthest distance as the next cluster center; S4313: Repeat the above steps until you select Cluster centers; the initialization formula of cluster centers is as follows: ,in, is the new cluster center; are all eigenvectors in the current subspace; is the feature vector With cluster center The distance between Represents the maximum value operation; S432: According to the cluster center selected in S431, the feature vectors in the subspace are assigned to the cluster to which the closest cluster center belongs; the attribution of each feature vector is determined by the following formula: ,in, is the eigenvector in the current subspace; For the sub-clusters; is the cluster center; Represents the distance function, which is used to measure the similarity between the feature vector and the cluster center; S433: Recalculate the cluster center according to the feature vectors in each sub-cluster. The new cluster center is the mean of all feature vectors in the sub-cluster. The calculation formula is: ,in, For the Cluster centers; For the The number of eigenvectors in each subcluster; For the The feature vectors of the subclusters; S434: Repeat the above steps S431, S432 and S433 until the cluster centers converge or reach a preset recursive depth threshold, and complete the recursive hierarchical clustering in the subspace; through the above recursive hierarchical clustering method, the feature vectors in the subspace are finely divided to ensure that the features in each subcluster have higher similarity; the distance-based cluster center selection method and dynamic update mechanism are adopted to maximize the accuracy and stability of the clustering effect, providing high-quality subcluster division results for the subsequent construction of a hierarchical feature index structure.

[0030] S44 specifically includes: S441: Initialize the root node of the tree structure according to the hierarchical clustering result, the root node represents the global feature space, and the cluster center is used as the feature value of the root node; create a child node for each subspace, the child node represents the first-level subspace, and each child node stores the cluster center of the corresponding subspace; S442: clustering the feature vectors in each child node again, generating its child nodes under the child node, and recursively repeating the process until one of the following conditions is met; Condition 1: the number of eigenvectors in the current subspace is less than the preset threshold; Condition 2: reaching the preset maximum recursion depth; in each level, the parent node of the child node stores the link of the cluster center of its child node; S443: In each level, the association weight between the parent node and its child nodes is calculated to represent the similarity relationship between the parent and child nodes; the weight value is calculated by the following formula: ,in, Represents the parent node and child nodes The association weight of Represents the parent node cluster center Cluster centers with child nodes distance; is a regularization parameter used to control the distribution range of the association weight; S444: All feature vectors in the current subspace are stored in each leaf node. The feature vectors in the leaf nodes are no longer divided, and the path from each leaf node to the root node represents the storage path of the feature vector. The above steps establish a tree hierarchy according to the hierarchical clustering results, organize each layer of subspace and the corresponding cluster center node into a parent-child relationship, and form an efficient feature indexing system. By calculating the association weight and storing feature vectors in leaf nodes, the access efficiency of the tree structure and the rationality of feature storage are enhanced, providing strong support for fast query.

[0031] S6 specifically includes: S61: According to the query feature vector generated in S5, locate the root node of the hierarchical feature index in S4, and traverse the hierarchical tree structure starting from the root node; S62: In the candidate nodes of the current layer, the similarity scores between the query feature vector and each index feature vector are calculated in turn. The similarity scores are measured by cosine similarity, and the formula is: ,in, The query feature vector with indexed feature vector Similarity score of is the query feature vector; is the index feature vector; norm of a vector; S63: Based on the similarity scores calculated in S62, the top The index feature vectors are screened as a set of candidate feature vectors to ensure that these feature vectors have the highest matching degree with the query feature vector; S64: If the number of candidate features in the current layer is less than the preset When the number of nodes is greater than 1, it will go back to the unselected child nodes in the previous layer, repeat steps S61 to S63, and continue to screen candidate features until the preset number conditions are met or all relevant nodes are traversed; S65: According to the final screened candidate feature vector set, the corresponding image results are extracted, sorted from high to low according to the similarity score, and the sorted image results are output as the final result of the query.

[0032] S65 specifically includes: S651: for each feature vector in the selected candidate feature vector set, query its corresponding image identifier in the hierarchical feature index; each feature vector in the candidate feature vector set corresponds to a unique image identifier one-to-one, and the candidate image identifier set is determined through the mapping relationship; S652: sorting the candidate image identifier set in descending order of similarity scores according to the similarity scores of each candidate feature vector calculated in S62; after sorting, prioritizing image identifiers having higher similarity scores with the query feature vector; S653: Retrieve corresponding image data from the image database according to the sorted image identifier set, each image identifier uniquely corresponds to an image, and the retrieved image data set is stored in sorted order to form an ordered image result set; S654: Output the sorted image result set as the final result of the query, and arrange the output results in descending order according to the similarity score with the query feature vector for the user to view or use; the above steps can efficiently locate the candidate image results through the mapping relationship between the candidate feature vector and the image identifier; combined with the sorting operation of the similarity score, it is ensured that the output image results have the best correlation with the query features, which improves the accuracy of the image search results and the query efficiency, and provides users with a fast and accurate retrieval experience.

[0033] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0034] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An image search method based on a multimodal algorithm, characterized in that: The following steps are involved: S1: Obtain image data and its corresponding text description data, and input the image data into a convolutional neural network to extract image feature vectors; perform word segmentation and embedding processing on the text description data to generate text feature vectors; S2: Build a multimodal feature alignment model, and input the image feature vector and text feature vector generated by S1 into the feature alignment model. Optimize the feature distribution through adversarial training, align the image feature vector and the text feature vector in the same vector space, and output the aligned feature vector. S3: Perform feature fusion processing on the aligned feature vector output by S2, use the bidirectional attention mechanism to calculate the weights between image features and text features, and generate a multimodal fusion feature vector based on the weighted calculation; S4: Perform hierarchical clustering on the multimodal fusion feature vector generated by S3, establish an index using a tree-based clustering method, map the fusion feature vector to the corresponding index node, and generate a hierarchical feature index structure; S5: receiving query data input by the user, and sequentially processing the query data through the same processing methods in S1, S2 and S3 above to generate a query feature vector; S6: Match the query feature vector generated by S5 with the hierarchical feature index generated by S4, calculate the similarity score between the query feature and the index feature, filter the candidate feature vectors according to the similarity score, and output the image result corresponding to the query data.

2. The image search method based on a multimodal algorithm according to claim 1, characterized in that: The S1 specifically includes: S11: Acquire image data containing a target image and its corresponding text description data from an image database, wherein the image data is a digital image in RGB format, and the text description data is a natural language text related to the image content, and the two are stored in a one-to-one correspondence through data tags; S12: input the acquired image data into a pre-trained convolutional neural network for processing, and output an image feature vector of a fixed dimension; S13: segmenting the acquired text description data word by word according to lexical rules to form a word segmentation sequence containing a single word or phrase; S14: input the word segmentation sequence generated in S13 into a pre-trained embedding model, where the embedding model is built based on word vector representation technology, and uses a word embedding method to map each word into a vector of a fixed dimension, thereby generating a text feature vector of the text description data; S15: Normalize the image feature vector generated by S12 and the text feature vector generated by S14, and map the values ​​of all feature vectors to the range of [0,1].

3. The image search method based on a multimodal algorithm according to claim 1, characterized in that: The S2 specifically includes: S21: constructing a multimodal feature alignment model, wherein the multimodal feature alignment model consists of a generator and a discriminator, wherein the generator is used to generate a feature representation consistent with the target modality, and the discriminator is used to distinguish between generated features and real features; S22: using the image feature vector and text feature vector generated in S1 as training data, wherein the image feature vector is a first modality feature of the model input, and the text feature vector is a second modality feature; S23: Conduct adversarial training. First, input the image feature vector into the generator to generate the alignment features of the target modality. At the same time, input the text feature vector and the features generated by the generator into the discriminator. The discriminator will update the discriminant function according to the distribution difference between the real features and the generated features. S24: After the adversarial training is completed, the image feature vector and text feature vector generated by S1 are input into the trained generator again to generate an aligned feature vector of the image modality and the text modality.

4. The image search method based on a multimodal algorithm according to claim 3, characterized in that: The S23 specifically includes: S231: Input the image feature vector generated by S1 to the generator, which maps the input features through a deep neural network to obtain an aligned feature vector of the target modality; the output of the generator is represented by the following formula: ,in, is the image feature vector; Align features for the generated target modality; For the generator; is the parameter of the generator; S232: Input the real text feature vector generated by S1 and the alignment feature generated by the generator into the discriminator; the discriminator extracts the distribution characteristics of the input features through a convolutional neural network and outputs a probability value , used to indicate whether the input feature is a real feature; the output of the discriminator is expressed by the following formula: ,in, is the real text feature vector; Aligned feature vectors generated by the generator; is the feature of the input discriminator; is the discriminator; are the parameters of the discriminator; S233: The goal of the discriminator is to distinguish between real text features and generated features. Its loss function uses binary cross entropy loss, which is defined as: ,in, is the loss of the discriminator; For the expected operation; is a logarithmic operation; Output probability value for the discriminator; S234: The goal of the generator is to generate aligned features that are as close to the real text feature distribution as possible, and its loss function is defined as: ,in, is the loss of the generator; S235: Based on the min and max adversarial optimization strategies, the parameters of the generator and discriminator are updated using the stochastic gradient descent method: The expression for updating the discriminator parameters is: ; Update the generator parameter expression to: ;in, is the learning rate; This is the gradient calculation operation.

5. The image search method based on a multimodal algorithm according to claim 1, characterized in that: The S3 specifically includes: S31: construct a bidirectional attention mechanism, and set the input to be the aligned feature vector output by S2, wherein the aligned feature vector includes an image feature vector and a text feature vector, wherein the image feature vector is used to express the visual features of the image, and the text feature vector is used to express the semantic features of the text; S32: Calculate the correlation between each image feature and all text features through the bidirectional attention mechanism, normalize the correlation, and generate a weight matrix corresponding to each image feature; S33: performing weighted processing on the text feature vector according to the weight matrix, performing weighted summation on all text features according to their weights with each image feature, and generating a weighted text feature vector associated with the image feature; S34: concatenate the image feature vector with the weighted text feature vector, and process the concatenation result through a nonlinear activation function to generate a multimodal fusion feature vector.

6. The image search method based on a multimodal algorithm according to claim 1, characterized in that: The S4 specifically includes: S41: Analyze the multimodal fusion feature vector generated by S3, select the initial cluster center through the distance-based algorithm, and divide the entire feature vector space into multiple subspaces, each subspace corresponds to an initial cluster center; S42: Allocate the multimodal fusion feature vectors to the corresponding subspaces according to the distances from each cluster center; each feature vector belongs to the cluster center with the closest distance, completing the preliminary feature division; S43: clustering the feature vectors in each subspace again, and dividing each subspace into smaller subspaces by repeating the steps of selecting cluster centers and allocating features, until a preset hierarchical depth is reached or the number of subspace feature vectors meets a specified threshold; S44: according to the result of hierarchical clustering, a tree-like hierarchical structure is established for the subspace of each layer and the corresponding cluster center node; the root node represents the global feature space, and each leaf node represents a subspace; S45: Map the multimodal fusion feature vectors in the hierarchical clustering results to corresponding leaf nodes, each leaf node stores the feature vector belonging to the corresponding subspace, and forms a hierarchical feature index structure for subsequent query and matching.

7. The image search method based on a multimodal algorithm according to claim 6, characterized in that: The S43 specifically includes: S431: Select new cluster centers in each subspace using a distance-based initialization method. Cluster centers; S432: According to the cluster center selected in S431, the feature vectors in the subspace are assigned to the cluster to which the closest cluster center belongs; the attribution of each feature vector is determined by the following formula: ,in, is the eigenvector in the current subspace; For the sub-clusters; is the cluster center; Represents the distance function, which is used to measure the similarity between the feature vector and the cluster center; S433: Recalculate the cluster center according to the feature vectors in each sub-cluster. The new cluster center is the mean of all feature vectors in the sub-cluster. The calculation formula is: ,in, For the Cluster centers; For the The number of eigenvectors in each subcluster; For the The feature vectors of the subclusters; S434: Repeat the above steps S431, S432 and S433 until the cluster centers converge or reach a preset recursive depth threshold, completing the recursive hierarchical clustering in the subspace.

8. The image search method based on a multimodal algorithm according to claim 7, characterized in that: The S44 specifically includes: S441: Initialize the root node of the tree structure according to the hierarchical clustering result, the root node represents the global feature space, and the cluster center is used as the feature value of the root node; create a child node for each subspace, the child node represents the first-level subspace, and each child node stores the cluster center of the corresponding subspace; S442: clustering the feature vectors in each child node again, generating its child nodes under the child node, and recursively repeating the process until one of the following conditions is met; Condition 1: the number of eigenvectors in the current subspace is less than the preset threshold; Condition 2: reaching the preset maximum recursion depth; S443: In each level, the association weight between the parent node and its child nodes is calculated to represent the similarity relationship between the parent and child nodes; the weight value is calculated by the following formula: ,in, Represents the parent node and child nodes The association weight of Represents the parent node cluster center Cluster centers with child nodes distance; is the regularization parameter; S444: All feature vectors in the current subspace are stored in each leaf node, and the feature vectors in the leaf node are no longer divided. The path from each leaf node to the root node represents the storage path of the feature vector.

9. The image search method based on a multimodal algorithm according to claim 1, characterized in that: The S6 specifically includes: S61: According to the query feature vector generated in S5, locate the root node of the hierarchical feature index in S4, and traverse the hierarchical tree structure starting from the root node; S62: in the candidate nodes of the current layer, calculating the similarity score between the query feature vector and each index feature vector in turn; S63: Based on the similarity scores calculated in S62, the top The index feature vectors are filtered into a set of candidate feature vectors; S64: If the number of candidate features in the current layer is less than the preset When the number of nodes is greater than 1, it will go back to the unselected child nodes in the previous layer, repeat steps S61 to S63, and continue to screen candidate features until the preset number conditions are met or all relevant nodes are traversed; S65: According to the final screened candidate feature vector set, the corresponding image results are extracted, sorted from high to low according to the similarity score, and the sorted image results are output as the final result of the query.

10. The image search method based on multimodal algorithm according to claim 9, characterized in that: The S65 specifically includes: S651: for each feature vector in the selected candidate feature vector set, query its corresponding image identifier in the hierarchical feature index; each feature vector in the candidate feature vector set corresponds to a unique image identifier one-to-one, and a candidate image identifier set is determined through a mapping relationship; S652: sorting the candidate image identifier set in descending order of similarity scores according to the similarity scores of each candidate feature vector calculated in S62; after sorting, prioritizing image identifiers having higher similarity scores with the query feature vector; S653: Retrieve corresponding image data from the image database according to the sorted image identifier set, each image identifier uniquely corresponds to an image, and the retrieved image data set is stored in sorted order to form an ordered image result set; S654: Outputting the sorted image result set as the final result of the query, and arranging the output results in descending order according to the similarity score with the query feature vector for the user to view or use.

Images