Knowledge-guided image search recall method, system and server
Through the knowledge-guided method, the similarity threshold is dynamically calculated using Chinese-CLIP and BERT models, which solves the problems of misrecall and missed recall in traditional image retrieval methods, and improves the accuracy and recall rate of image search.
Patent Information
- Application Number
- CN202411349349.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2025-01-07
AI Technical Summary
Traditional image retrieval methods are difficult to meet users' needs for efficient and accurate retrieval, especially in cloud disk image search scenarios, fixed number or fixed threshold strategies are likely to lead to misrecall and missed recalls.
Using a knowledge-guided method, by extracting entities in the search statement, querying concept levels based on the Chinese WordNet knowledge base, calculate the dynamic similarity threshold, combine Chinese-CLIP and BERT models to extract semantic feature vectors of images and text, and perform similarity calculations. Recall recall.
It improves the recall and accuracy of image search, can adapt to the diversity and complexity of different search statements, accurately identify time information, remove meaningless semantics, and improves the accuracy of image search.
Smart Images
Figure CN119273983A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of information technology, and in particular relates to a knowledge-guided image search and recall method, system and server. Background Art
[0002] With the rapid development of the Internet and the explosive growth of information, the generation and storage of image data has also increased significantly. Faced with the shortage of local storage space, more and more users choose to upload their data to cloud storage systems such as cloud disks. However, the massive amount of image data has brought great challenges to users in retrieval and management. It has become increasingly difficult for users to efficiently and accurately retrieve the required data from the huge amount of personal image data.
[0003] Traditional retrieval methods based on image file names and image classification labels have significant shortcomings. First, the types of image classification labels are limited, and it is difficult to fully cover the user's retrieval needs. In addition, image classification labels usually only contain the main information in the image, and fail to fully explore the richer semantic information in the image, resulting in a low retrieval recall rate. Therefore, this method cannot effectively meet the user's needs for efficient and accurate retrieval.
[0004] With the development of deep learning, retrieval methods based on image-text semantic feature matching have been widely used in the task of searching images with text. This type of method is based on contrastive learning for pre-training (e.g. CLIP, Contrastive Language-Image Pre-Training), mapping images and texts to a unified vector space, in which related texts and images have similar vector representations. During the search process, the search statement (Query) is sorted based on the similarity of the representation vector (e.g., cosine similarity) between the search statement and the image to be searched. Then, using the sorting as input, two different strategies are usually used to obtain the final search recall result: 1) Fixed number: return the top N images with the highest similarity values; 2) Fixed threshold: return images with similarity values above a certain threshold.
[0005] However, in actual application scenarios such as cloud disk image search, the optimal image-text similarity threshold is highly correlated with the search statement (for example, compared to searching for "Persian cat", when a user searches for "cat", the search intent is broader, so the recall threshold should be relatively lower). Whether a fixed number or fixed threshold strategy is adopted, false recall and missed recall are prone to occur.
[0006] Therefore, there is an urgent need to construct a more generalized and flexible search method that can dynamically select the optimal similarity threshold according to the content of the search statement, thereby achieving more accurate search recall. Summary of the invention
[0007] In order to address the shortcomings of the prior art, the present application provides a knowledge-guided image search recall method, system and server. Compared with other cross-modal image search recall methods, this method adopts knowledge guidance, first extracts entities in the search statement, and based on the knowledge base, queries the conceptual hierarchy of the entities in the search statement, and calculates the similarity threshold of the image-text feature vector matching, thereby effectively improving the recall rate and accuracy of image search.
[0008] The technical effects to be achieved by this application are achieved through the following solutions:
[0009] According to a first aspect of the present application, a knowledge-guided image search and recall method is provided, comprising the following steps:
[0010] Step 1: When the image is stored in the cloud storage system, the visual encoder of the Chinese-CLIP model is used to extract the image semantic feature vector of the image, and the image semantic feature vector is stored in the database;
[0011] Step 2: When a search statement is entered, the BERT model is used to perform entity recognition on the search statement and extract all entities in the search statement;
[0012] Step 3: Match the entities with words in the Chinese WordNet knowledge base to obtain the concept hierarchy of all entities;
[0013] Step 4: According to the concept hierarchy, normalize and calculate to obtain a similarity threshold;
[0014] Step 5: extract the text semantic feature vector of the search statement, perform similarity calculation with the image semantic feature vector after deleting the time-related semantics, and obtain the recall result based on the similarity threshold.
[0015] Preferably, in step 1, when extracting the image semantic feature vector, the image is segmented into small images of 224*224 pixels, and a 768-dimensional image semantic feature vector is extracted.
[0016] Preferably, in step 2, the BERT model converts the entity recognition task of the search sentence into a sequence labeling task, uses the BERT model to perform BIO labeling on the words in the search sentence, and uses the conditional random field decoder to calculate the probability of the label sequence, and completes the entity recognition task according to the probability.
[0017] Preferably, the specific method for completing the entity recognition task according to the probability is:
[0018] Processing the search statement using a BERT model to obtain a BERT encoding representation of each word;
[0019] The BERT encoded representation is input into the linear layer and mapped to the entity label space using the following formula:
[0020] z i =W·h i +b
[0021] Where W and b are the parameters of the linear layer, h i is the BERT encoding representation of the i-th word;
[0022] Apply a normalized exponential function to the output of the linear layer to calculate the entity label probability distribution for each word:
[0023]
[0024] Among them, K is the dimension of z in BERT encoding representation, and z is the entity label space;
[0025] According to the entity tag probability distribution, the tag with the highest probability is selected as the entity tag, and combined with the corresponding entity words to generate an entity list.
[0026] Preferably, in step 3, after all entities in the search sentence are identified, the conceptual level of each entity is queried in the WordNet knowledge base based on BFS.
[0027] Preferably, if the entity does not exist in the WordNet knowledge base, it is calculated at the lowest level.
[0028] Preferably, in step 4, the similarity threshold is calculated using the following formula:
[0029]
[0030] in: τ is the experience value, C i is the concept level, n is the total number of concept levels, and α is the empirical value coefficient.
[0031] Preferably, in step 5, the text semantic feature vector of the search sentence is extracted based on the text encoder in the Chinese-CLIP model, the cosine similarity between the text semantic feature vector and the image semantic feature vector is calculated, and images above the similarity threshold are recalled as recall objects.
[0032] According to a second aspect of the present application, an image search and recall system using the above-mentioned knowledge-guided image search and recall method is provided, comprising:
[0033] An image semantic feature extraction module is used to extract image semantic feature vectors of all image information stored by the user, and store the image semantic feature vectors in a vector database;
[0034] A dynamic threshold calculation module is used to perform entity recognition on search sentences and calculate a similarity threshold;
[0035] The search recall module is used to extract the text semantic feature vector in the search sentence, calculate the similarity with the image semantic feature vector, and perform image recall according to the similarity threshold.
[0036] According to a third aspect of the present application, there is provided a server, comprising: a memory and at least one processor;
[0037] The memory stores a computer program, and the at least one processor executes the computer program stored in the memory to implement the above-mentioned knowledge-guided image search and recall method.
[0038] According to an embodiment of the present application, the beneficial effects of using the knowledge-guided image search and recall method are: it can fully extract the semantic information in the user's search sentence, and calculate different vector matching thresholds based on the concept level to obtain a similarity threshold, which can adapt to the diversity and complexity of different prompts compared to the traditional fixed threshold, greatly improving the image recall accuracy;
[0039] This method can accurately identify the time information in the user's search sentence, remove the interference of time semantics and other meaningless semantics on the understanding of text semantics, effectively use text knowledge to guide image search, and effectively improve the recall rate and accuracy of image search. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present application or the existing technical solutions, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0041] Figure 1 This is a flow chart of a knowledge-guided image search and recall method in one embodiment of the present application;
[0042] Figure 2 This is a structural block diagram of an image search and recall system in one embodiment of the present application;
[0043] Figure 3 This is a structural block diagram of a server in an embodiment of the present application. DETAILED DESCRIPTION
[0044] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0045] like Figure 1 As shown, the knowledge-guided image search and recall method in one embodiment of the present application includes the following steps:
[0046] Step 1: When the image is stored in the cloud storage system, the visual encoder (Vision Transformer model) of the Chinese-CLIP model is used to extract the image semantic feature vector of the image, and the image semantic feature vector is stored in the database;
[0047] In this step, when extracting the semantic feature vector of the image, the ViT-H-14 model is used to resize the image to an image with a size of 224*224 pixels, and extract a 768-dimensional feature vector.
[0048] During the training process, the Chinese-CLIP model extracts semantic feature vectors of image-text pairs respectively. If there is a semantic matching relationship between the image-text pairs, the similarity between the feature vectors will be increased; if there is no semantic matching relationship, the similarity between the feature vectors will be reduced, so as to achieve cross-modal matching of images and texts.
[0049] Step 2: When entering a search statement, use the BERT model to perform entity recognition on the search statement and extract all entities in the search statement.
[0050] In this step, the NER (Named Entity Recognition) task of the search sentence is converted into a sequence labeling task based on BERT, and BERT is used to perform BIO labeling on the words in the search sentence. By classifying each word or token into the form of "BX", "IX" or "O", different entities in the text can be effectively distinguished and identified.
[0051] In other embodiments, Bi-LSTM and other models can be used as needed. After the Bi-LSTM model is trained, it is easy to deploy. BERT has good generalization ability and can process text data in different fields.
[0052] Then, the CRF (Conditional Random Field) decoder is used to calculate the probability of the tag sequence. Compared with the traditional named entity recognition method, this method has higher accuracy and stronger generalization, and is more suitable for image search scenarios in cloud disks.
[0053] Specifically: First, input the search sentence T into the BERT model to obtain the BERT encoding representation of each word in the search sentence.
[0054] The encoded representation of BERT output is then fed into a linear layer and mapped to the entity label space:
[0055] z i =W·h i +b
[0056] Where: W and b are the parameters of the linear layer, h i is the BERT encoding representation of the i-th word;
[0057] Use the normalized exponential function (Softmax) on the output of the linear layer to calculate the entity label probability distribution of each word:
[0058]
[0059] Where K is the dimension of BERT encoding representation z;
[0060] Determine the entity tag for each word by selecting the tag with the highest probability:
[0061] y i = argmax P(y i |h i )
[0062] Finally, combine the corresponding entity words and generate an entity list based on the predicted labels.
[0063] In this step, the entity list refers to extracting all entities recognized in the text and putting them into a list. For example, if the search text is "Zhang San and a dog at the beach", there are three entities "Zhang San", "dog", and "beach" in the text, which are people, animals, and places respectively. The entity list is ["Zhang San", "dog", "beach"].
[0064] Step 3: Match the entities with words in the Chinese WordNet knowledge base to obtain the concept hierarchy of all entities
[0065] In this step, after identifying all entities in the search statement, the conceptual level of each entity is queried in the WordNet knowledge base based on BFS (breadth-first search method). If the entity does not exist in the WordNet knowledge base, it is calculated according to the lowest level, and finally the corresponding conceptual level of the entity list is obtained.
[0066] The specific principle of BFS is: starting from the root node, add the node to the queue, then continuously take out the node from the front of the queue and explore its adjacent nodes. For each visited node, it will be marked and added to the queue until the target node is found or the queue is empty. It is suitable for solving the shortest path.
[0067] In this step, the conceptual hierarchy of entities in the text mainly includes the analysis of entity recognition, classification, relationships and attributes, which facilitates the subsequent retention of required concepts in the text, such as retaining people, organizations, places, events, etc. classified according to their nature.
[0068] Step 4: Based on the concept hierarchy, normalize and calculate the similarity threshold
[0069] In this step, the similarity threshold is calculated using the following formula:
[0070]
[0071] Wherein: τ is an empirical value, which is 0.62 in this embodiment and can be set to different values according to different scenarios in actual applications; C i is the concept level; n is the total number of concept levels; α is the empirical value coefficient, which is 0.3 in this embodiment.
[0072] In this step, different similarity thresholds are dynamically set for different search texts, which can adapt to the diversity and complexity of different prompts compared to the fixed thresholds in the prior art.
[0073] For example, for a search term with a lower conceptual level, such as "Persian cat", the similarity between related images and the search term is concentrated in a smaller range; for a search term with a higher conceptual level, such as "cat", the similarity between related images and the search term is in a wider range. If a traditional fixed threshold is used, some images under certain prompts may be over-screened (i.e., too many irrelevant images are included) or under-screened (i.e., many relevant images are missed), thus affecting the accuracy and recall rate of the search.
[0074] Step 5: extract the text semantic feature vector of the search statement, perform similarity calculation with the image semantic feature vector after deleting the time-related semantics, and obtain the recall result based on the similarity threshold.
[0075] In this step, the text encoding RoBERTa model based on Chinese-CLIP is used to extract the text semantic feature vector of the search sentence input by the user;
[0076] The cosine similarity between the semantic feature vector of the search statement and the semantic feature vector of the image in the database is calculated, and based on the dynamic similarity threshold calculated in the dynamic threshold calculation module, images with similarity higher than the threshold are determined as search recall results.
[0077] Deleting time-related semantics means deleting the time text after the entity is recognized as a time entity. For example, if the search text is "last year at the beach", the image content itself cannot reflect the time concept of "last year". Therefore, when doing image-text matching, time-related text must be removed to ensure that no invalid information affects the recall rate.
[0078] like Figure 2 As shown, in one embodiment of the present application, an image search and recall system using the above-mentioned knowledge-guided image search and recall method includes:
[0079] Image semantic feature extraction module, which is based on the image encoding ViT-H-14 model of Chinese-CLIP, extracts the image semantic feature vectors of all image information stored by the user, and stores the vectors in the vector database (Milvus) for subsequent queries.
[0080] Dynamic threshold calculation module, which first inputs the search sentence T into the BERT language model to obtain the BERT encoding representation H = [h1,h2,…,h n ];
[0081] The encoded representation of BERT output is then fed into a linear layer and mapped to the entity label space: i =W·h i +b, where W and b are parameters of the linear layer, h i is the BERT encoding representation of the i-th word.
[0082] Use the normalized exponential function (Softmax) on the output of the linear layer to calculate the entity label probability distribution of each word: Here, K is the dimension of BERT encoding representation z.
[0083] Determine the entity tag for each word by selecting the tag with the highest probability: y i = argmax P(y i |h i ), finally, combine the corresponding entity words and generate an entity list based on the predicted labels
[0084] Based on the entity list, the concept level of the corresponding entity is searched in the WordNet knowledge base. If it cannot be found in the knowledge base, it is calculated according to the lowest concept level, and finally the corresponding concept level of the entity list is obtained. Finally, the final similarity threshold is calculated using the following formula, where τ is an empirical value. In this embodiment, the value is 0.62. In actual applications, it can be set to different values according to different scenarios:
[0085]
[0086] Search recall module, which first uses the RoBERTa model based on the text encoding of Chinese-CLIP to extract the text semantic feature vector of the search sentence input by the user;
[0087] The cosine similarity between the text semantic feature vector and the image semantic feature vector is calculated, and based on the dynamic similarity threshold calculated in the dynamic threshold calculation module, images with similarity higher than the threshold are determined as search recall results.
[0088] like Figure 3 As shown, a server in an embodiment of the present application includes: a memory 301 and at least one processor 302;
[0089] The memory 301 stores a computer program, and the at least one processor 302 executes the computer program stored in the memory 301 to implement the above-mentioned knowledge-guided image search and recall method.
[0090] This method combines the language knowledge base to perform semantic analysis on the search statement, identify the conceptual hierarchy of the core entities involved, and dynamically calculate the recall threshold based on this, thereby effectively improving the recall rate and accuracy of image search:
[0091] It can fully extract the semantic information in the user's search sentence and calculate different vector matching thresholds based on the concept hierarchy to obtain a dynamic similarity threshold. Compared with the traditional fixed threshold, it can adapt to the diversity and complexity of different prompts and greatly improve the image recall accuracy.
[0092] It can accurately identify the time information in user search statements, remove the interference of time semantics and other meaningless semantics on text semantic understanding, and effectively use text knowledge to guide image search.
[0093] It should be noted that the above detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present application belongs.
[0094] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should also be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0095] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.
[0096] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products, or apparatuses.
[0097] For ease of description, spatially relative terms, such as "above", "above", "on the upper surface of", "above", etc., may be used herein to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" may include both "above" and "below". The device may also be positioned in other different ways, such as rotated 90 degrees or in other orientations, and the spatially relative descriptions used herein are interpreted accordingly.
[0098] In the above detailed description, reference is made to the accompanying drawings, which form a part of this document. In the accompanying drawings, similar symbols typically identify similar components unless the context indicates otherwise. The illustrated embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be used, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein.
[0099] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A knowledge-guided image search and recall method, characterized in that: The steps include: Step 1: When the image is stored in the cloud storage system, the visual encoder of the contrastive language image pre-training model CLIP is used to extract the image semantic feature vector of the image, and the image semantic feature vector is stored in the database; Step 2: When a search statement is entered, the BERT model is used to perform entity recognition on the search statement and extract all entities in the search statement; Step 3: Match the entities with words in the Chinese WordNet knowledge base to obtain the concept hierarchy of all entities; Step 4: According to the concept hierarchy, normalize and calculate to obtain a similarity threshold; Step 5: extract the text semantic feature vector of the search statement, perform similarity calculation with the image semantic feature vector after deleting the time-related semantics, and obtain the recall result based on the similarity threshold.
2. The knowledge-guided image search and recall method according to claim 1, characterized in that: In step 1, when extracting the image semantic feature vector, the size of the image is adjusted to 224*224 pixels, and a 768-dimensional image semantic feature vector is extracted.
3. The knowledge-guided image search and recall method according to claim 1, characterized in that: In step 2, the BERT model converts the entity recognition task of the search sentence into a sequence labeling task, uses the BERT model to perform BIO labeling on the words in the search sentence, and uses the conditional random field decoder to calculate the probability of the label sequence, and completes the entity recognition task based on the probability.
4. The knowledge-guided image search and recall method according to claim 3, characterized in that: The specific method for completing the entity recognition task according to the probability is: Processing the search statement using a BERT model to obtain a BERT encoding representation of each word; The BERT encoded representation is input into the linear layer and mapped to the entity label space using the following formula: z i =W·h i +b Where W and b are the parameters of the linear layer, h i is the BERT encoding representation of the i-th word; Apply a normalized exponential function to the output of the linear layer to calculate the entity label probability distribution for each word: Among them, K is the dimension of z in BERT encoding representation, and z is the entity label space; According to the entity tag probability distribution, the tag with the highest probability is selected as the entity tag, and combined with the corresponding entity words to generate an entity list.
5. The knowledge-guided image search and recall method according to claim 3, characterized in that: In step 3, after all entities in the search sentence are identified, the conceptual level of each entity is queried in the WordNet knowledge base based on breadth-first search.
6. The knowledge-guided image search and recall method according to claim 5, characterized in that: If the entity does not exist in the WordNet knowledge base, it is calculated according to the lowest level.
7. The knowledge-guided image search and recall method according to claim 5, characterized in that: In step 4, the similarity threshold is calculated using the following formula: Where: τ is the empirical value, C i is the concept level, n is the total number of concept levels, and α is the empirical value coefficient.
8. The knowledge-guided image search and recall method according to claim 1, characterized in that: In step 5, the text semantic feature vector of the search sentence is extracted based on the text encoder in the Chinese-CLIP model, the cosine similarity between the text semantic feature vector and the image semantic feature vector is calculated, and images with a similarity threshold value higher than the similarity threshold are recalled as recall objects.
9. An image search and recall system using the knowledge-guided image search and recall method according to any one of claims 1 to 8, characterized in that: include: An image semantic feature extraction module is used to extract image semantic feature vectors of all image information stored by the user, and store the image semantic feature vectors in a vector database; A dynamic threshold calculation module is used to perform entity recognition on search sentences and calculate a similarity threshold; The search recall module is used to extract the text semantic feature vector in the search sentence, calculate the similarity with the image semantic feature vector, and perform image recall according to the similarity threshold.
10. A server, characterized in that: include: memory and at least one processor; The memory stores a computer program, and the at least one processor executes the computer program stored in the memory to implement the knowledge-guided image search and recall method according to any one of claims 1 to 8.