Image retrieval method and system based on large model

Abstract

The invention provides an image retrieval method and system based on a large model. The method comprises the following steps: extracting text information of an original picture; converting the original picture into a visual feature vector, and converting the text information into a character feature vector; respectively calculating the similarity of the two feature vectors in each set to obtain a visual feature fusion similarity and a character feature fusion similarity; determining similarity measurement of visual feature and character feature fusion; constructing data corresponding to the original picture into an initial picture structure; allocating a weight for each edge of the initial graph structure by using the metric to obtain a simplified coarsened graph structure; performing hierarchical segmentation on the coarsened image structure to obtain a plurality of initial cut edge sets; training the automatic edge cutting set optimizer by using the training data; re-dividing the initial cut edge set by using the trained automatic cut edge set optimizer; and taking the nearest partition as a retrieval result. According to the method, the retrieval accuracy can be improved while the retrieval efficiency is ensured.

Description

Technical Field

[0001] This invention relates to the field of image retrieval technology, and in particular to an image retrieval method and system based on a large model. Background Technology

[0002] Image retrieval is widely used in scenarios such as facial recognition, autonomous driving, and medical imaging. Current image retrieval methods primarily organize data by constructing graph structures, which include nodes and edges. Utilizing the topological relationships of the graph can improve query efficiency. Graph-based indexing methods have advantages in retrieval accuracy; however, as the data scale increases, the computational and storage overhead of constructing the graph structure becomes significant, and the graph partitioning process is prone to missegmentation of similar samples, leading to a decrease in retrieval efficiency and accuracy. Furthermore, most existing image retrieval methods rely on visual feature vectors of images for distance calculations, failing to fully utilize the textual information contained within the images, thus reducing retrieval accuracy. Summary of the Invention

[0003] This invention provides an image retrieval method and system based on a large model to solve the technical problems of low efficiency and accuracy in existing image retrieval methods.

[0004] To address the aforementioned technical problems, this invention provides an image retrieval method based on a large model, comprising the following steps:

[0005] Using a multimodal large model, text information is extracted from each original image;

[0006] Each of the original images is converted into a visual feature vector, and the text information corresponding to each original image is converted into a text feature vector, resulting in a set of visual feature vectors and a set of text feature vectors corresponding to multiple original images;

[0007] Calculate the similarity between two feature vectors in each set to obtain the visual feature fusion similarity and the text feature fusion similarity;

[0008] The similarity measure of visual feature fusion and text feature fusion is determined based on the mean of the similarity of visual feature fusion and text feature fusion.

[0009] The data corresponding to the original image is used to construct an initial graph structure;

[0010] The initial graph structure is simplified by assigning weights to each edge using the metric and merging nearest neighbor pairs based on the weights to obtain a simplified, coarsened graph structure.

[0011] The coarsened graph structure is hierarchically segmented, and based on the segmentation, iteratively backprojected onto the initial graph structure to obtain several initial cut edge sets.

[0012] The automatic edge-cutting set optimizer is trained using training data to obtain the trained automatic edge-cutting set optimizer. The training data consists of query points and their nearest neighbor sets. The automatic edge-cutting set optimizer includes multiple fully connected layers and minimizes the KL divergence between the predicted partition distribution and the target distribution during training.

[0013] The initial edge set is repartitioned using the trained automatic edge set optimizer, so that each boundary node is assigned to the partition corresponding to the maximum distribution probability predicted by the trained automatic edge set optimizer, thus obtaining the optimized index structure.

[0014] Based on the input query vector, the distance between the query vector and the center node of each partition of the index structure is calculated, and the partition with the closest distance is taken as the retrieval result.

[0015] Preferably, the visual feature fusion similarity and the text feature fusion similarity are obtained through multiple similarity fusion methods.

[0016] Preferably, the visual feature fusion similarity and the text feature fusion similarity are each equal to the average of multiple similarities.

[0017] Preferably, the multiple similarities include Euclidean distance similarity, cosine similarity, Jaccard similarity, and semantic feature similarity.

[0018] Preferably, the step of determining the similarity measure of visual feature and text feature fusion includes the following steps: using the average of the visual feature fusion similarity and the text feature fusion similarity as the similarity measure of visual feature and text feature fusion.

[0019] Preferably, the text information includes keywords and a brief description of the original image.

[0020] This invention also provides an image retrieval system based on a large model, comprising the following modules:

[0021] The text extraction module uses a multimodal large model to extract the text information from each original image.

[0022] The vector conversion module is used to convert each of the original images into visual feature vectors and to convert the text information corresponding to each of the original images into text feature vectors, thereby obtaining a set of visual feature vectors and a set of text feature vectors corresponding to multiple original images.

[0023] The similarity calculation module is used to calculate the similarity between two feature vectors in each set, and obtain the visual feature fusion similarity and the text feature fusion similarity.

[0024] The similarity measurement calculation module is used to determine the similarity measure of visual feature fusion and text feature fusion based on the mean of the visual feature fusion similarity and text feature fusion similarity.

[0025] An initial graph structure construction module is used to construct an initial graph structure from the data corresponding to the original image.

[0026] The coarsening graph structure construction module assigns weights to each edge of the initial graph structure using the metric, and merges nearest neighbor node pairs based on the weights to gradually simplify the initial graph structure, resulting in a simplified coarsening graph structure.

[0027] An initial cut edge set construction module is used to hierarchically segment the coarsened graph structure and iteratively backproject it onto the initial graph structure based on the segmentation to obtain several initial cut edge sets.

[0028] An automatic edge-cutting set optimizer is trained using training data to obtain a trained automatic edge-cutting set optimizer. The training data consists of query points and their nearest neighbor sets. The automatic edge-cutting set optimizer includes multiple fully connected layers and minimizes the KL divergence between the predicted partition distribution and the target distribution during training.

[0029] The optimization module uses the trained automatic edge-cutting set optimizer to repartition the initial edge-cutting set, so that each boundary node is assigned to the partition corresponding to the maximum distribution probability predicted by the trained automatic edge-cutting set optimizer, thus obtaining the optimized index structure.

[0030] The output module is used to calculate the distance between the input query vector and the center node of each partition of the index structure, and take the partition with the closest distance as the retrieval result.

[0031] Preferably, the visual feature fusion similarity and the text feature fusion similarity are obtained through multiple similarity fusion methods.

[0032] Preferably, the visual feature fusion similarity and the text feature fusion similarity are each equal to the average of multiple similarities.

[0033] Preferably, the multiple similarities include Euclidean distance similarity, cosine similarity, Jaccard similarity, and semantic feature similarity.

[0034] This invention provides an image retrieval method and system based on a large model. It utilizes a multimodal large model to extract textual information from the original image, vectorizes it to calculate semantic distance, and then weights and fuses it with visual feature distance to construct a retrieval graph driven by both semantic and visual features. Furthermore, a trained automatic edge-cutting set optimizer is used to repartition the initial edge-cutting set, assigning each boundary node to the partition corresponding to the maximum distribution probability predicted by the trained automatic edge-cutting set optimizer, resulting in an optimized index structure. This approach improves retrieval accuracy while maintaining retrieval efficiency. Attached Figure Description

[0035] Figure 1 This is a flowchart of an image retrieval method based on a large model provided in an embodiment of the present invention. Detailed Implementation

[0036] To make the objectives, advantages, and features of this invention clearer, the following detailed description of the image retrieval method and system based on a large model, in conjunction with the accompanying drawings, is provided. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of this invention.

[0037] In the description of this invention, the terms "first," "second," and other qualifiers are added for convenience of description and reference, and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with qualifiers such as "first" and "second" may explicitly or implicitly include one or more of that feature.

[0038] like Figure 1 As shown, this invention provides an image retrieval method based on a large model, comprising the following steps:

[0039] S1. Using a multimodal large model, extract the text information for each original image; wherein the multimodal large model can be a model such as GPT, Claude, or Deepseek. Preferably, the text information includes keywords and brief descriptions corresponding to the original image. These keywords and brief descriptions can be text inherent in the original image or text generated by the multimodal large model to describe the original image.

[0040] S2. Convert each of the original images into visual feature vectors and convert the text information corresponding to each original image into text feature vectors, thereby obtaining a set of visual feature vectors and a set of text feature vectors corresponding to multiple original images. Models such as CLIP or Transformer can be used to convert the original images and corresponding text information to obtain visual feature vectors and text feature vectors, with each original image corresponding to one visual feature vector and one text feature vector.

[0041] S3. Calculate the similarity between two feature vectors in each set to obtain the visual feature fusion similarity and the text feature fusion similarity. Preferably, the visual feature fusion similarity and the text feature fusion similarity are obtained through multiple similarity fusion methods. These multiple similarities include Euclidean distance similarity, cosine similarity, Jaccard similarity, and semantic feature similarity, etc., which allows for the combination of the advantages of different similarities when calculating distance.

[0042] Preferably, the visual feature fusion similarity and the text feature fusion similarity are each equal to the average of multiple similarities. The visual feature fusion similarity and the text feature fusion similarity can be calculated using the following formulas:

[0043] in, This represents the similarity of visual features between image i and image j. This represents the visual feature fusion similarity calculated by the first similarity calculation method. This represents the visual feature fusion similarity calculated using the second similarity calculation method. This represents the visual feature fusion similarity calculated using the nth similarity calculation method. n represents the total number of different similarity calculation methods. Similarly, This represents the similarity between image i and image j based on the fusion of text features. This represents the text feature fusion similarity calculated using the first similarity calculation method. This represents the text feature fusion similarity calculated using the second similarity calculation method. This represents the text feature fusion similarity calculated by the nth similarity calculation method.

[0044] S4. Determine the similarity measure of visual feature fusion and text feature fusion based on the average of the visual feature fusion similarity and the text feature fusion similarity. Preferably, the step of determining the similarity measure of visual feature fusion and text feature fusion includes the following steps: using the average of the visual feature fusion similarity and the text feature fusion similarity as the similarity measure of visual feature fusion and text feature fusion. In typical scenarios, this can be achieved using a formula. = ( + The similarity measure obtained by fusing visual and textual features is 1 / 2. In other embodiments, the similarity can be determined based on experimental results. and Set different weights.

[0045] S5. Construct an initial graph structure from the data corresponding to the original image. Steps S5-S7 are existing technologies and will not be described in detail here.

[0046] S6. Assign weights to each edge of the initial graph structure using the metric, and merge nearest neighbor pairs of nodes based on the weights to gradually simplify the initial graph structure, resulting in a simplified coarsened graph structure.

[0047] S7. Perform hierarchical segmentation on the coarsened graph structure, and backproject it onto the initial graph structure in an iterative manner based on the segmentation to obtain several initial cut edge sets.

[0048] S8. Train the Automatic Cut-Edge Optimizer (ACEO) using the training data to obtain the trained ACEO, wherein the training data is generated by the query point { } and its nearest neighbor set The automatic edge-cutting ensemble optimizer (ACEO) comprises multiple fully connected layers that minimize the KL divergence between the predicted partition distribution and the target distribution during training. ACEO takes a query vector q as input and transforms it into a probability distribution over the partitions through L fully connected layers. Each fully connected layer performs a linear transformation followed by a non-linear activation function operation, such as ReLU. This can be expressed using the formula... Where l = 1, 2, ..., L, L represents the total number of fully connected layers (i.e., hidden layers), and l represents the l-th fully connected layer. Let σ represent the output of the l-th layer, and let σ represent the activation function. This represents the weight matrix of the l-th layer. This represents the output of the (l-1)th layer. Let represent the bias vector of the l-th bit. =q, the output layer applies the softmax function to generate the probability distribution of all possible segments.

[0049] For each query vector, the ACEO model generates a probability distribution P(y|q) about the partition. Since the neighbors of a query vector may span multiple partitions, and the distribution of these neighbors may differ across partitions, soft labels are used during training to capture this distribution variation. Specifically, the soft label probability of partition y is defined as: P(y|q) ) = | ∩y| / | To train the model, minimize the predicted partition distribution P(y|q) and the target distribution P(y|q). The objective function is the KL divergence between the nearest neighbors and the target distribution, which reflects the actual distribution of neighbors within the interval. This objective function encourages the model to approximate the true distribution of neighbors, thereby improving the overall recall of nearest neighbor search.

[0050] S9. The initial edge cut set is repartitioned using the trained automatic edge cut set optimizer, so that each boundary node... The nodes are assigned to partitions corresponding to the maximum probability distribution predicted by the trained automatic edge-cutting set optimizer, resulting in the optimized index structure. The trained ACEO model can compute the value of each boundary node. The probability distribution P(y|) of being assigned to a certain partition ), will the boundary nodes Assigning nodes to the partitions corresponding to the highest probability distribution ensures a more balanced distribution of boundary nodes without affecting stable partitions.

[0051] S10. Based on the input query vector, calculate the distance between the query vector and the center node of each partition of the index structure, and take the partition with the closest distance as the retrieval result.

[0052] This invention provides an image retrieval method based on a large model. It utilizes a multimodal large model to extract textual information from the original image, vectorizes it to calculate semantic distance, and then weights and fuses it with visual feature distance to construct a retrieval graph driven by both semantic and visual features. Furthermore, a trained automatic edge-cutting set optimizer is used to repartition the initial edge-cutting set, assigning each boundary node to the partition corresponding to the maximum distribution probability predicted by the trained automatic edge-cutting set optimizer, resulting in an optimized index structure. This approach improves retrieval accuracy while maintaining retrieval efficiency.

[0053] The following experiments verify and compare the effectiveness of the proposed image retrieval method based on a large model. The experimental results show that the proposed method outperforms existing methods in both recall and queries per second (QPS).

[0054] Experimental environment:

[0055] Operating System: Ubuntu 20.04.5 LTS

[0056] CPU: Intel(R) Xeon(R) Platinum 8362 CPU @ 2.80GHz

[0057] Memory: 256G

[0058] Datasets: SIFT1M, GIST

[0059] Table 1. Recall rates for each method in SIFT128 and 1M when performing 10,000 distance calculations:

[0060] Table 2. Queries per second for each method when querying 1000 queries in SIFT128, 1M:

[0061] Table 3. Recall rates for each method in GIST960, 1M when calculating 10,000 distance calculations:

[0062] Table 4. Queries per second for each method when querying 1000 queries on GIST960, 1M:

[0063] Based on the same technical concept as the aforementioned large-model-based image retrieval method, this embodiment provides a large-model-based image retrieval system, including the following modules:

[0064] The text extraction module uses a multimodal large model to extract the text information from each original image.

[0065] The vector conversion module is used to convert each of the original images into visual feature vectors and to convert the text information corresponding to each of the original images into text feature vectors, thereby obtaining a set of visual feature vectors and a set of text feature vectors corresponding to multiple original images.

[0066] The similarity calculation module is used to calculate the similarity between two feature vectors in each set, and obtain the visual feature fusion similarity and the text feature fusion similarity.

[0067] The similarity measurement calculation module is used to determine the similarity measure of visual feature fusion and text feature fusion based on the mean of the visual feature fusion similarity and text feature fusion similarity.

[0068] An initial graph structure construction module is used to construct an initial graph structure from the data corresponding to the original image.

[0069] The coarsening graph structure construction module assigns weights to each edge of the initial graph structure using the metric, and merges nearest neighbor node pairs based on the weights to gradually simplify the initial graph structure, resulting in a simplified coarsening graph structure.

[0070] An initial cut edge set construction module is used to hierarchically segment the coarsened graph structure and iteratively backproject it onto the initial graph structure based on the segmentation to obtain several initial cut edge sets.

[0071] An automatic edge-cutting set optimizer is trained using training data to obtain a trained automatic edge-cutting set optimizer. The training data consists of query points and their nearest neighbor sets. The automatic edge-cutting set optimizer includes multiple fully connected layers and minimizes the KL divergence between the predicted partition distribution and the target distribution during training.

[0072] The optimization module uses the trained automatic edge-cutting set optimizer to repartition the initial edge-cutting set, so that each boundary node is assigned to the partition corresponding to the maximum distribution probability predicted by the trained automatic edge-cutting set optimizer, thus obtaining the optimized index structure.

[0073] The output module is used to calculate the distance between the input query vector and the center node of each partition of the index structure, and take the partition with the closest distance as the retrieval result.

[0074] This embodiment provides an image retrieval system based on a large model. It utilizes a multimodal large model to extract textual information from the original image, vectorizes it to calculate semantic distance, and then weights and fuses it with visual feature distance to construct a retrieval graph driven by both semantic and visual features. In addition, the initial edge set is repartitioned using a trained automatic edge-cutting set optimizer, so that each boundary node is assigned to the partition corresponding to the maximum distribution probability predicted by the trained automatic edge-cutting set optimizer, resulting in an optimized index structure. This improves retrieval accuracy while ensuring retrieval efficiency.

[0075] Preferably, the visual feature fusion similarity and the text feature fusion similarity are obtained through multiple similarity fusion methods, so that the advantages of different similarities can be combined when calculating distance.

[0076] Preferably, the visual feature fusion similarity and the text feature fusion similarity are each equal to the average of multiple similarities, which simplifies the similarity calculation process.

[0077] Preferably, the multiple similarities include Euclidean distance similarity, cosine similarity, Jaccard similarity, and semantic feature similarity, so that the advantages of these similarities can be combined when calculating the distance.

[0078] In summary, the image retrieval method and system based on a large model provided by this invention utilizes a multimodal large model to extract textual information from the original image, vectorizes it to calculate semantic distance, and then weights and fuses it with visual feature distance to construct a retrieval graph driven by both semantic and visual features. In addition, the initial edge set is repartitioned using a trained automatic edge-cutting set optimizer, so that each boundary node is assigned to the partition corresponding to the maximum distribution probability predicted by the trained automatic edge-cutting set optimizer, resulting in an optimized index structure. This ensures retrieval efficiency while improving retrieval accuracy.

[0079] The above description is only a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the present invention.

Claims

1. A large model-based image retrieval method, characterized by, The method comprises the following steps: extracting text information of each original picture by using a multi-modal large model; converting each original picture into a visual feature vector and converting the text information corresponding to each original picture into a text feature vector to obtain a visual feature vector set and a text feature vector set corresponding to multiple original pictures; calculating the similarity of the two feature vectors in each set to obtain a visual feature fusion similarity and a text feature fusion similarity; determining a similarity measure of the visual feature and text feature fusion according to the average of the visual feature fusion similarity and the text feature fusion similarity; constructing data corresponding to the original pictures into an initial graph structure; assigning a weight to each edge of the initial graph structure using the measure and merging the nearest neighbor node pairs based on the weight to gradually simplify the initial graph structure to obtain a simplified coarse graph structure; performing hierarchical segmentation on the coarse graph structure and iteratively projecting onto the initial graph structure based on the segmentation to obtain a plurality of initial cut edge sets; training an automatic cut edge set optimizer using training data to obtain a trained automatic cut edge set optimizer, wherein the training data consists of a query point and its nearest neighbor set, the automatic cut edge set optimizer includes multiple fully connected layers, and the KL divergence between the predicted partition distribution and the target distribution is minimized during training; re-dividing the initial cut edge set using the trained automatic cut edge set optimizer so that each boundary node is assigned to the partition corresponding to the maximum distribution probability predicted by the trained automatic cut edge set optimizer to obtain an optimized index structure; calculating the distance between the input query vector and the center node of each partition of the index structure and taking the nearest partition as the retrieval result.

2. The large model-based image retrieval method of claim 1, wherein, The visual feature fusion similarity and the text feature fusion similarity are obtained by multiple similarity fusion methods. 3.The large model-based image retrieval method of claim 2, wherein, The visual feature fusion similarity and the text feature fusion similarity are equal to the average of multiple similarities.

4. The large model-based image retrieval method of claim 3, wherein, The multiple similarities include Euclidean distance similarity, cosine similarity, Jaccard similarity, and semantic feature similarity.

5. The large model-based image retrieval method of claim 1, wherein, The step of determining the similarity measure of the visual feature and text feature fusion comprises the following steps: taking the average of the visual feature fusion similarity and the text feature fusion similarity as the similarity measure of the visual feature and text feature fusion.

6. The large model-based image retrieval method of claim 1, wherein, The text information includes keywords and brief description information corresponding to the original picture.

7. A large model-based image retrieval system, characterized by, The method comprises the following modules: a text extraction module for extracting text information of each original picture by using a multi-modal large model; a vector conversion module for converting each original picture into a visual feature vector and converting the text information corresponding to each original picture into a text feature vector to obtain a visual feature vector set and a text feature vector set corresponding to multiple original pictures; a similarity calculation module for calculating the similarity of the two feature vectors in each set to obtain a visual feature fusion similarity and a text feature fusion similarity. The similarity measurement calculation module is used to determine the similarity measure of visual feature fusion and text feature fusion based on the mean of the visual feature fusion similarity and text feature fusion similarity. An initial graph structure construction module is used to construct an initial graph structure from the data corresponding to the original image. The coarsening graph structure construction module assigns weights to each edge of the initial graph structure using the metric, and merges nearest neighbor node pairs based on the weights to gradually simplify the initial graph structure, resulting in a simplified coarsening graph structure. An initial cut edge set construction module is used to hierarchically segment the coarsened graph structure and iteratively backproject it onto the initial graph structure based on the segmentation to obtain several initial cut edge sets. An automatic edge-cutting set optimizer is trained using training data to obtain a trained automatic edge-cutting set optimizer. The training data consists of query points and their nearest neighbor sets. The automatic edge-cutting set optimizer includes multiple fully connected layers and minimizes the KL divergence between the predicted partition distribution and the target distribution during training. The optimization module uses the trained automatic edge-cutting set optimizer to repartition the initial edge-cutting set, so that each boundary node is assigned to the partition corresponding to the maximum distribution probability predicted by the trained automatic edge-cutting set optimizer, thus obtaining the optimized index structure. The output module is used to calculate the distance between the input query vector and the center node of each partition of the index structure, and take the partition with the closest distance as the retrieval result.

8. The large model-based image retrieval system of claim 7, wherein, The visual feature fusion similarity and the text feature fusion similarity are obtained through multiple similarity fusion methods.

9. The large model-based image retrieval system of claim 8, wherein, The visual feature fusion similarity and the text feature fusion similarity are respectively equal to the average of multiple similarities.

10. The large model-based image retrieval system of claim 9, wherein, The various similarity metrics include Euclidean distance similarity, cosine similarity, Jaccard similarity, and semantic feature similarity.

Images