Vector type image and video retrieval method

Abstract

The invention discloses a vector type image and video retrieval method, which realizes efficient and accurate image and video retrieval through semantic analysis and field word fine tuning of a multi-modal large model CLIP and in combination with an accurate matching mechanism of a vector database. The method comprises the steps that firstly, heterogeneous data such as texts, images and videos are mapped to a unified vector space through a multi-mode semantic analysis engine; secondly, establishing a special field lexicon for a specific application scene, and performing key feature representation; and finally, realizing comparison and retrieval of the vector library in combination with nearest neighbor search and an inverted index mechanism. The system can meet the requirements of the vertical field, supports multi-modal retrieval, and has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision, natural language processing and multimedia retrieval, and in particular to a vector image and video retrieval method. Background Art

[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.

[0003] With the rapid growth of image and video data in professional fields, traditional retrieval methods based on keywords or simple visual features can no longer meet the needs of accurate retrieval.

[0004] The existing technology has the following problems: Insufficient semantic understanding: Traditional methods have difficulty understanding the deep semantic information of images in closed-source professional fields, resulting in inaccurate retrieval results. Poor domain adaptability: General models perform poorly in specialized fields and cannot effectively handle vocabulary and semantics related to closed-source professional fields. Insufficient support for multimodal retrieval: Existing systems generally only support image or text retrieval and lack support for video retrieval. Low retrieval efficiency and inability to optimize retrieval results: The retrieval efficiency of large-scale image and video data is low, making it difficult to meet real-time requirements, and it is impossible to provide secondary feedback on the retrieval results to promote system optimization. Summary of the Invention

[0005] In order to solve the above problems, the present invention proposes a vector-based image and video retrieval method, which realizes efficient and accurate image and video retrieval through semantic parsing of a multimodal large model, domain word fine-tuning and a two-stage matching mechanism.

[0006] Technical Solution: This invention proposes a vector-based image and video retrieval method. The specific steps include: Step 1: Image understanding and semantic parsing: Use a multimodal contrastive learning model to perform semantic parsing on the input image and obtain a text description of the image. Step 2: Extract domain labels. Segment the text description obtained in step 1, filter the segmentation results through a closed-source professional domain vocabulary, and obtain keywords related to the specific field as labels.

[0007] Step 3: Extract and store image and video feature vectors. Extract image feature vectors from images and key frames from videos. Select the most representative frame in each shot to extract its feature vector. Store all extracted feature vectors and their domain labels in a vector database to support subsequent similarity retrieval.

[0008] Step 4: For the first matching, extract the query vector from the input image or video, perform a similarity search on the query vector using the vector database obtained in step 3, and return the result based on the most similar feature vector found in the query.

[0009] Step 5: Second matching: convert the input image or video into a text description, extract the label through the text description, and filter and deduplicate the candidate results by calculating the similarity between the label of the candidate image or video obtained in step 4 and the user query label.

[0010] Furthermore, step 1 specifically includes: Feature extraction and mapping process: The CLIP model's Vision Transformer encoder is used. The input image is segmented into 16×16 pixel blocks through the patch embedding layer and mapped to a 768-dimensional feature space via linear projection. The text encoder uses a 12-layer Transformer architecture, with each layer containing a 768-dimensional hidden state and 12 attention heads, ultimately outputting a 768-dimensional text feature vector. Based on the attention weight matrix W∈R^(d×d) of the CLIP text encoder, a low-rank decomposition matrix is ​​injected: W' = W + BA. Here, B∈R^(d×r) and A∈R^(r×d). The rank r is set to 16, reducing the number of parameters in the original weight matrix to 1 / 48.

[0011] Loss function design: The contrast loss function calculates the similarity matrix between image feature I and text feature T:

[0012] Where τ=0.07 is the temperature parameter and sim is the cosine similarity calculation.

[0013] Parameter update strategy: freeze all parameters of the visual encoder; only update the LORA matrix parameters of the last three layers of Transformer blocks of the text encoder; use the Adam optimizer, learning rate 1e-5, and weight decay 0.01; set the batch size to 64 and train for 20 epochs; calculate the gradient of the loss function with respect to the LORA matrices A and B through the backpropagation algorithm: add professional domain vocabulary to the word embedding layer of the text encoder, expand the vocabulary to 52,000 words, and initialize the initial embedding vector of the newly added vocabulary by the weighted average of the existing vocabulary, and the weight coefficient is calculated by the TF-IDF algorithm; use the cosine annealing learning rate scheduling strategy, and decay the initial learning rate from 1e-5 to 1e-7 to ensure the optimal alignment of image features and domain text features on the 768-dimensional hypersphere.

[0014] Step 1-1: Use the multimodal model CLIP (Contrastive Language-Image Pre-training) to perform semantic analysis on the input image and video frames, generate text descriptions, and map them into a fixed 1*768 feature vector.

[0015] Step 1-2: Use the LORA algorithm to adjust the domain terms of the CLIP model. Use the LORA algorithm (Low-Rank Adaptation) to add a low-rank matrix to the attention layer of the CLIP model to adapt to the vocabulary and semantics of the closed-source professional field.

[0016] Furthermore, the domain-specific tuning of the CLIP model in steps 1-2 specifically involves training the CLIP model on closed-source, professional domain image-text data based on a fine-tuning objective to minimize the contrastive loss between image and text features. The CLIP model is fine-tuned for domain terms using LORA technology. This involves freezing the CLIP visual encoding, fine-tuning only the last three Transformer blocks of the text encoder, and injecting a domain dictionary for additional embedding.

[0017] Furthermore, step 2 specifically includes: Step 2-1: Text segmentation, based on the dictionary maximum matching algorithm MMSEG, uses the word segmentation function of the HanLP toolkit to segment the text description generated by the CLIP model.

[0018] Step 2-2: Domain word filtering: Combined with a closed-source domain vocabulary, use the TF-IDF term frequency-inverse document frequency algorithm to filter the word segmentation results from step 2-1 and extract keywords related to the specific domain as tags. For example, from the description "T100 DJI drone, coaxial twin-rotor", tags such as "T100", "DJI drone", and "coaxial twin-rotor" can be extracted.

[0019] Furthermore, step 3 specifically includes: Step 3-1: Image feature extraction: Extract image feature vectors using the EfficientNet model, optimize the model depth, width, and resolution through compound scaling, and output a 2048-dimensional feature vector.

[0020] Step 3-2: Video key frame extraction: Use a shot boundary detection-based extraction method to extract key frames from the video and extract the feature vectors of the key frames. Specifically, the shot boundaries in the video are detected through the frame difference method. In each acquired shot, the K-Means clustering algorithm is used to select the most representative key frame, and the feature vectors of the key frames are extracted using the EfficientNet model.

[0021] Step 3-3: Vector storage: Store the feature vectors and domain labels of the image and video keyframes in the Elasticsearch vector database. Use ES's dense vector search feature to support efficient similarity searches. The domain labels here are pre-defined labels, which are a subset of the keyword label set obtained in Step 2.

[0022] Furthermore, step 4 specifically includes: Step 4-1: Query vector generation. For the image or video input by the user, the query vector is extracted through the EfficientNet model.

[0023] Step 4-2: Similarity Search: Use cosine similarity in the Elasticsearch vector database to find the image or video feature vector most similar to the query vector. Using an approximate nearest neighbor search (ANN), results are quickly returned from the large-scale stored data. The data size can range from tens of thousands to hundreds of thousands. Results are returned for a set number of the most similar results, which can be 50 or 100, or for all results with a similarity greater than a set threshold, such as 0.65.

[0024] Furthermore, step 5 specifically includes: Step 5-1: Convert the input image or video into text description through the fine-tuned CLIP model.

[0025] Step 5-2: Use Hanlp to extract labels for domain words in text expressions.

[0026] Step 5-3: Label matching algorithm uses Jaccard similarity to calculate the similarity between the domain label of the candidate image or video and the user query label. For example, if the user query label is "plant protection drone" and the candidate labels are "T100", "DJI drone", and "coaxial twin-rotor", then the Jaccard similarity is: Jaccard = |{DJI drone}∩{T100, plant protection drone, coaxial twin-rotor}|||{plant protection drone}∪{T100,DJI drone, coaxial twin-rotor}| Step 5-4: Screening and sorting, screen the candidate results according to label similarity, retain images or videos with similarity higher than the threshold, and sort the results.

[0027] Furthermore, the sorting of the screening results in step 5-4 specifically includes comprehensively calculating the visual feature similarity and label similarity, and calculating the comprehensive score of the screening results by the following formula: Comprehensive score = α·visual similarity + (1-α)·label similarity comprehensive score; where α is the weight parameter.

[0028] Furthermore, the value of the weight parameter α is preferably 0.6.

[0029] Furthermore, it includes optimization steps, specifically collecting user feedback, including collecting user behavior data such as clicks, collections, and ratings on search results. Based on user feedback data, the CLIP model is optimized through online learning methods; the feature vectors and domain labels in the Elasticsearch vector database are regularly updated to ensure the timeliness of the system.

[0030] Furthermore, the method also includes step 6, system optimization, which specifically includes collecting user feedback, optimizing the CLIP model through an online learning method based on the user feedback data; and regularly updating the feature vectors and domain labels in the Elasticsearch vector database.

[0031] Compared with the prior art, the present invention has the following significant advantages: More accurate semantic understanding: Through the CLIP model and domain word fine-tuning, the system can accurately understand the semantic information of closed-source professional images. Before domain word fine-tuning, there was an issue with being unable to extract certain closed-source tags, but fine-tuning has resolved this issue.

[0032] Efficient retrieval performance: A two-stage matching mechanism based on the Elasticsearch vector database significantly improves retrieval efficiency and accuracy. Testing shows that compared to the basic CLIP model, the LORA fine-tuned CLIP solution achieves: improved accuracy: Top-1 and Top-5 accuracy increased by over 17% and 8%, respectively; enhanced domain adaptability: semantic matching accuracy on private domain datasets: the general CLIP model achieved 62% accuracy; after domain dictionary injection, the accuracy reached 81% (an increase of over 19.1%).

[0033] Multimodal retrieval support: supports multimodal retrieval of images, text, and videos to meet diverse user needs.

[0034] Domain adaptability: Through LORA fine-tuning and closed-source professional domain vocabulary, the system can adapt to the needs of various fields and provide personalized retrieval services. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0036] Figure 1 This is a system architecture diagram of the present invention.

[0037] Figure 2 It is the workflow diagram of the present invention.

[0038] Figure 3 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0039] like Figure 1-3 As shown, the present invention proposes a vector image and video retrieval method. The specific steps include: Step 1: Image understanding and semantic parsing: Use a multimodal contrastive learning model to perform semantic parsing on the input image and obtain a text description of the image. Step 2: Extract domain labels. Segment the text description obtained in step 1, filter the segmentation results through a closed-source professional domain vocabulary, and obtain keywords related to the specific field as labels.

[0040] Step 3: Extract and store image and video feature vectors. Extract image feature vectors from images and key frames from videos. Select the most representative frame in each shot to extract its feature vector. Store all extracted feature vectors and their domain labels in a vector database to support subsequent similarity retrieval.

[0041] Step 4: For the first matching, extract the query vector from the input image or video, perform a similarity search on the query vector using the vector database obtained in step 3, and return the result based on the most similar feature vector found in the query.

[0042] Step 5: Second matching: convert the input image or video into a text description, extract the label through the text description, and filter and remove duplicate candidate results by calculating the similarity between the label of the candidate image or video obtained in step 4 and the user query label.

[0043] Example 1: Special Equipment Image Retrieval

[0044] (1) Image understanding and semantic parsing: Input a picture of a plant protection drone, and the fine-tuned CLIP model generates a description: "T100, DJI drone, coaxial twin-rotor."

[0045] (2) Domain label extraction: Use the HanLP word segmenter to extract labels: "T100", "UAV", "twin rotor".

[0046] (3) Image feature vector extraction: Use EfficientNet to extract the feature vector of the drone and prepare it for comparison with the feature vector of the Elasticsearch vector database.

[0047] (4) First matching: The system finds images with similar visual features in the ES vector database.

[0048] (5) Second matching: Filter out images with labels including “twin rotor” and “drone”.

[0049] (6) Result display: Display drone images that highly match the query semantics and visual features.

[0050] Example 2: Special Equipment Video Retrieval

[0051] (1) Video keyframe extraction and semantic parsing: Input a video of a group of engineering vehicles transporting goods, extract the keyframes, and use the fine-tuned CLIP model to generate a description: "Engineering vehicle group moving in formation, desert terrain".

[0052] (2) Domain label extraction: Use the HanLP word segmenter to extract labels: "engineering vehicle group" and "desert terrain".

[0053] (3) Video feature vector extraction: Use EfficientNet to extract the feature vectors of the video key frames and prepare them for comparison with the feature vectors in the Elasticsearch vector database.

[0054] (4) First matching: The system finds videos with similar visual features in the ES vector database.

[0055] (5) Second matching: Filter out videos with the tags “engineering vehicle” and “desert terrain”.

[0056] (6) Result display: Display engineering operation videos that highly match the query semantics and visual features.

[0057] The present invention provides a concept and method for vector-based image and video retrieval. There are numerous methods and approaches for implementing this technical solution. The above is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A vector image and video retrieval method, characterized in that: The specific steps include: Step 1: Image understanding and semantic parsing: semantically parse the input image using a multimodal contrastive learning model to obtain a textual description of the image. Step 2: Domain label extraction: segment the text description obtained in step 1, filter the segmentation results through a closed-source professional domain vocabulary, and obtain keywords related to the specific domain as labels; Step 3: Extract and store image and video feature vectors. Extract image feature vectors from images and key frames from videos. Select the most representative frame in each shot to extract its feature vector. Store all extracted feature vectors and their domain labels in a vector database to support subsequent similarity retrieval. Step 4: First matching: extract the query vector from the input image or video, perform a similarity search on the query vector using the vector database obtained in step 3, and return the result based on the most similar feature vector found in the query. Step 5: Second matching: convert the input image or video into a text description, extract the label through the text description, and filter and deduplicate the candidate results by calculating the similarity between the label of the candidate image or video obtained in step 4 and the user query label.

2. The vector image and video retrieval method according to claim 1, wherein: Step 1 specifically includes: Step 1-1: Use the multimodal model CLIP to perform semantic analysis on the input image and video frames, generate text descriptions, and map them into a fixed 1*768 feature vector; Step 1-2: Use the LORA algorithm to adjust the domain terms of the CLIP model. Use the LORA algorithm to add a low-rank matrix to the attention layer of the CLIP model to adapt to the vocabulary and semantics of the closed-source professional field.

3. The vector image and video retrieval method according to claim 2, wherein: The domain-specific tuning of the CLIP model in steps 1-2 involves training the CLIP model on closed-source domain image-text pairs based on the fine-tuning objective of minimizing the contrast loss between image and text features. The CLIP model is fine-tuned for domain terms using LORA technology. This involves freezing the CLIP visual encoding, fine-tuning only the last three Transformer blocks of the text encoder, and injecting a domain dictionary for additional embedding.

4. The vector image and video retrieval method according to claim 3, wherein: Step 2 specifically includes: Step 2-1: Text segmentation: Based on the dictionary-based maximum matching algorithm MMSEG, the text description generated by the CLIP model is segmented using the word segmentation function of the HanLP toolkit; Step 2-2: Domain word filtering. Combined with the closed-source professional domain vocabulary, use the TF-IDF term frequency-inverse document frequency algorithm to filter the word segmentation results obtained in step 2-1 and extract keywords related to specific fields as tags.

5. The vector image and video retrieval method according to claim 4, wherein: Step 3 specifically includes: Step 3-1: Image feature extraction: extract image feature vectors through the EfficientNet model, optimize the model depth, width, and resolution through compound scaling, and output a 2048-dimensional feature vector; Step 3-2: Video keyframe extraction: Use a shot boundary detection-based extraction method to extract keyframes from the video and extract the keyframe feature vectors. Specifically, the shot boundaries in the video are detected using the inter-frame difference method. For each acquired shot, the most representative keyframe is selected using the K-Means clustering algorithm. The feature vectors of the keyframes are then extracted using the EfficientNet model. Step 3-3: Vector storage: store the feature vectors of the image and video key frames and their domain labels in the Elasticsearch vector database, where the domain labels are a subset of the keyword label set.

6. The vector image and video retrieval method according to claim 5, characterized in that: Step 4 specifically includes: Step 4-1: Query vector generation: For the image or video input by the user, the query vector is extracted through the EfficientNet model; Step 4-2: Similarity search: Use cosine similarity to perform a similarity search in the Elasticsearch vector database to obtain the image or video feature vector that is most similar to the query vector. Results are returned through an approximate nearest neighbor search. The returned results range is a certain number of the most similar results, or all results with a similarity greater than a set threshold.

7. The vector image and video retrieval method according to claim 6, wherein: Step 5 specifically includes: Step 5-1: Convert the input image or video into text description through the fine-tuned CLIP model; Step 5-2: Use Hanlp to extract labels for domain words in text expressions; Step 5-3: Label matching algorithm, using Jaccard similarity to calculate the similarity between the domain label of the candidate image or video and the user query label; Step 5-4: Screening and sorting, screen the candidate results according to label similarity, retain images or videos with similarity higher than the threshold, and sort the results.

8. The vector image and video retrieval method according to claim 7, wherein: The sorting of the screening results in step 5-4 specifically includes combining the visual feature similarity and label similarity to calculate the comprehensive score of the screening results using the following formula: Comprehensive score = α·visual similarity + (1-α)·label similarity comprehensive score; where α is a weight parameter. The visual similarity is the cosine similarity obtained in step 4-2.

9. The vector image and video retrieval method according to claim 8, wherein: The value of the weight parameter α is 0.

6.

10. The vector image and video retrieval method according to claim 9, wherein: This includes step 6, system optimization, which specifically involves collecting user feedback, optimizing the CLIP model through online learning methods based on the user feedback data, and regularly updating the feature vectors and domain labels in the Elasticsearch vector database.

Images