Vertical domain image retrieval method based on text and semantic fusion

Abstract

This invention belongs to the field of information retrieval technology, and specifically discloses a vertical domain image retrieval method based on text and semantic fusion. This solution uses the image filename, storage path, and visual description associated with each image as multi-source text information, and constructs an inverted index corresponding to the images. This effectively utilizes the implicit classification and semantics within folder hierarchies, improving resource utilization. Furthermore, in a hybrid mode of text retrieval and deep semantic retrieval, it significantly improves retrieval accuracy and efficiency, especially for images with non-standard naming, resolving the fundamental contradiction of existing retrieval methods that struggle to balance efficiency and accuracy. In addition, by assigning weight coefficients to different sources within the multi-source text information, this solution consistently ensures that images matching the query terms and image filenames receive the highest scores and are ranked at the top, significantly improving query accuracy and efficiency while preserving the user's original query habits.

Description

Technical Field

[0001] This invention belongs to the field of information retrieval technology, and in particular relates to a vertical domain image retrieval method based on text and semantic fusion. Background Technology

[0002] In highly specialized vertical fields such as construction engineering, document management, and industrial quality inspection, the management of massive image assets has long faced the fundamental contradiction of balancing retrieval efficiency and accuracy. How to quickly and accurately locate target image information from massive image assets has become a technical problem that continues to be solved.

[0003] Currently, traditional image retrieval technologies mainly fall into two categories: one relies on text annotation and keyword matching (represented by BM25 retrieval), and the other is semantic vector retrieval based on deep learning (represented by large-scale vision-language pre-trained models such as CLIP). The first retrieval mode is efficient when searching for items with precise literal meaning, such as specific project numbers or contract numbers; that is, its retrieval capability is entirely limited by the quality and standardization of text annotation. However, in real-world production environments, many images become "dark matter" due to arbitrary naming (e.g., "IMG_001.jpg"), brief descriptions, or inconsistent terminology, making them unretrievable. This is primarily because the first method cannot perform semantic recognition. To address this, a second retrieval mode has emerged. This mode maps images and text to a unified high-dimensional vector space and achieves "text-based image search" by calculating the cosine or Euclidean distance between vectors. This method demonstrates powerful semantic understanding capabilities in general scenarios. However, when directly applied to vertical fields such as construction engineering, record management, and industrial quality inspection, it still has certain shortcomings. First, this field is filled with a large number of unique professional terms, symbol systems, and standard specifications (such as HRB400 steel bar type and 16G101-1 drawing set number). General pre-trained models lack accurate understanding of these, which easily leads to semantic drift. For example, when a user precisely searches for "electrical schematic diagram with project number P-2023-028", the model may be more inclined to return several drawings with similar visual styles and element layouts but completely different numbers, which seriously damages the precision. Second, pure vector retrieval schemes rely on a huge vector database. Image input requires time-consuming neural network forward inference to generate vectors, and data update and maintenance costs are high. Furthermore, its retrieval process lacks a mechanism to identify and protect the user's clear and precise search intent (such as precise number and model), often using vague semantic associations to cover up precise search needs.

[0004] In summary, existing technologies present a polarized pattern: on one end is precise but rigid text keyword retrieval, which cannot reach visual content; on the other end is flexible but vague vector semantic retrieval, which is difficult to control domain precision. Therefore, there is an urgent need for a new generation of intelligent image retrieval methods that can accurately match and understand semantics and can gain insight into domain characteristics in order to unlock the true value of massive image assets in vertical domains. Summary of the Invention

[0005] The purpose of this invention is to provide a vertical domain image retrieval method based on text and semantic fusion. This method effectively utilizes multi-source text information associated with images and integrates text retrieval with deep semantic retrieval, realizing hybrid retrieval based on file-level information. This solves the problem that a single retrieval mode cannot simultaneously achieve retrieval efficiency and accuracy in vertical domains.

[0006] To address the aforementioned issues, this application provides a vertical domain image retrieval method based on text and semantic fusion, comprising the following steps: Step S1: Based on the multi-source text information associated with each image, construct an inverted index corresponding to that image. The multi-source text information includes at least the image file name, storage path, and visual description content corresponding to each image. Step S2: In response to the query text, weight coefficients are assigned to different sources in the multi-source text information based on the relevance between the query text and the multi-source text information. Based on the weight coefficients, the relevance score of each image to the query text is calculated, and then a set of candidate images related to the query text is retrieved from the inverted index. Step S3: Extract the visual feature vector and the text feature vector of the query text corresponding to each image in the candidate image set, and determine the visual semantic score of each image and the query text based on the visual feature vector and the text feature vector; Step S4: After normalizing the relevance score and visual semantic score corresponding to each image, a weighted fusion algorithm is used to obtain the final score, and a simplified image set is output from the candidate image set based on the final score.

[0007] As a preferred embodiment of this application: In step S1, the specific steps for constructing the inverted index corresponding to each image based on the multi-source text information associated with that image include: Step S11: Obtain the image file name, storage path, and visual description content corresponding to each image; Step S12: Combine the image file name, storage path and visual description content corresponding to each image to form a virtual document, and then perform word segmentation to create an inverted index. Alternatively, treat the image file name, storage path and visual description content corresponding to each image as independent fields and create an inverted index.

[0008] As a preferred embodiment of this application: In step S2, assigning weight coefficients to different sources in the multi-source text information based on the relevance of the query text to the multi-source text information specifically includes: The query terms are obtained by segmenting the query text into words: When a query term matches an image file name in multi-source text information, the image file name is assigned the highest weight; When a query term matches a storage path in multi-source text information, the storage path is assigned the second-highest weight. When a query term matches visual descriptions in multi-source text information, maintain the basic weight.

[0009] As a preferred embodiment of this application: In step S3, the specific methods for calculating the relevance score between each image and the query text based on the weighting coefficient include the following two: Method 1: When the image file name, storage path and visual description content are treated as independent fields, the score of each query term in each field is calculated separately, then multiplied by the weight coefficient of the corresponding field, and finally the scores of each query term are summed to calculate the relevance score. Method 2: When the image file name, storage path and visual description content are combined to form a virtual document, the scores of each query term in the virtual document are calculated and summed. Then, the query terms are traversed to determine the highest weight of the hit. Finally, the scores are weighted and calculated with the highest weight coefficient to obtain the relevance score.

[0010] As a preferred embodiment of this application: In step S3, CLIP retrieval is used to extract the visual feature vector corresponding to each image in the candidate image set and the text feature vector of the query text. Before extraction, each image in the candidate image set is preprocessed, including size adjustment and center cropping.

[0011] As a preferred embodiment of this application: In step S3, the specific steps for determining the visual semantic score of each image and the query text based on the visual feature vector and the text feature vector include: Step S31: Map the visual feature vector and the text feature vector to the same vector space; Step S32: Calculate the similarity between the visual feature vector and the text feature vector using cosine similarity to obtain a visual semantic score.

[0012] As a preferred embodiment of this application: In step S4, after obtaining the final score and before outputting the simplified image set, it is determined whether the relevance score of each image exceeds a preset value. If so, a reward score is added to the final score based on the reward mechanism, and the simplified image set is output from the candidate image set based on the final score after adding the reward score.

[0013] As a preferred embodiment of this application: In step S4, a weighted fusion algorithm is used to obtain the final score based on a preset weighting strategy for the relevance score and the visual semantic score.

[0014] As a preferred embodiment of this application: In step S2, BM25 text retrieval is used to retrieve the candidate image set from the inverted index.

[0015] Compared with existing technologies, the advantages of this invention's vertical domain image retrieval method based on text and semantic fusion are as follows: (1) This solution uses the image file name, storage path and visual description associated with each image as multi-source text information, and uses this multi-source text information to construct an inverted index corresponding to the image. As is well known, in engineering and archive management practice, images do not exist in isolation, but are nested in folder hierarchy with clear business logic (e.g., " / Shanghai-Suzhou-Huzhou High-speed Railway Project / Section 2 / Bridge Engineering / Pier Construction / Quality Inspection Photos / "). This folder hierarchy itself contains rich, manually organized contextual semantics, which is a valuable prior knowledge. However, existing technologies only regard it as a simple storage location and fail to deeply analyze and utilize it in the retrieval model, resulting in a waste of information resources. It can be seen that this solution effectively utilizes the classification and semantics implicit in the folder hierarchy and improves the effective utilization of resources.

[0016] (2) This scheme adopts a hybrid retrieval mode of text retrieval and deep semantic retrieval. The inverted index based on multi-source text information is used for preliminary coarse retrieval by text retrieval, and then a candidate image set related to the query text and including both standardized and non-standardized naming is obtained. The deep semantic retrieval uses the candidate image set as the retrieval data, which can effectively reduce the power consumption of deep semantic retrieval and improve the response speed. Deep semantic detection redefines the relevance between each image in the candidate image set and the query text through semantic analysis. Finally, a weighted fusion method is used to output a simplified image set from the candidate image set, realizing high-precision image retrieval. That is, this scheme, based on the inverted index based on multi-source text information, adopts a hybrid retrieval mode of text retrieval and deep semantic retrieval to effectively solve the fundamental contradiction that existing retrieval efficiency and accuracy are difficult to achieve simultaneously.

[0017] (3) This solution is particularly suitable for industries that rely on file directory management, such as construction, manufacturing and archives management. It can accurately understand the classification information implied by the folder hierarchy and effectively release the real value of massive image assets in the vertical field.

[0018] (4) The hybrid retrieval mode of text retrieval and deep semantic retrieval is adopted, which solves the shortcomings of the existing single retrieval mode. Attached Figure Description

[0019] Figure 1 A flowchart of a vertical domain image retrieval method based on text and semantic fusion provided in an embodiment of the present invention.

[0020] Figure 2 A schematic diagram illustrating the specific principle of the vertical domain image retrieval method based on text and semantic fusion provided in this embodiment of the invention. Detailed Implementation

[0021] The present invention will be further described in detail below with reference to specific embodiments and accompanying drawings. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0022] Example 1:

[0023] This embodiment provides a vertical domain image retrieval method based on text and semantic fusion. See [link to relevant documentation]. Figure 1 The method specifically includes the following steps: Step S1: Based on the multi-source text information associated with each image, construct an inverted index corresponding to that image. The multi-source text information includes at least the image file name, storage path, and visual description content corresponding to each image.

[0024] In this step, in addition to the image file name, the multi-source text information also includes the storage path representing the folder hierarchy. Furthermore, visual descriptions are added. As is well known, in engineering and record management practice, images do not exist in isolation but are nested within folder hierarchies with clear business logic (e.g., " / Shanghai-Suzhou-Huzhou High-Speed ​​Railway Project / Section 2 / Bridge Engineering / Pier Construction / Quality Evaluation Photos / "). This folder hierarchy itself contains rich, manually organized contextual semantics, representing highly valuable prior knowledge. However, existing technologies only treat it as a simple storage location, failing to deeply analyze and utilize it in the retrieval model, resulting in a waste of information resources. Therefore, this embodiment, based on existing technologies, adds a folder hierarchy structure—the storage path—to the multi-source text information associated with each image, using this storage path as one of the basic retrieval data.

[0025] In this step, the specific steps for constructing the inverted index corresponding to each image based on the multi-source text information associated with that image include: Step S11: Obtain the image file name (e.g., quality inspection.jpg), storage path (e.g., / Shanghai-Suzhou-Huzhou High-speed Railway Project / Section 2 / Bridge Engineering / Pier Construction / Quality Inspection Photos / ) and visual description content (e.g., the pier that has just been completed) for each image. Step S12: Combine the image file name, storage path, and visual description content corresponding to each image to form a virtual document. That is, the three fields are combined into one virtual document. The fields can be segmented using word segmentation symbols such as "-" or " / ". For example, Quality Inspection.jpg / Shanghai-Suzhou-Huzhou High-speed Railway Project / Second Section / Bridge Engineering / Pier Construction / Quality Inspection Photo / Just Completed Pier. After word segmentation, an inverted index is established. Alternatively, the image file name, storage path, and visual description content corresponding to each image can be used as independent fields, and an inverted index can be established. In this embodiment, the method of establishing an inverted index for each image based on virtual text or independent fields is a conventional existing technology, which will not be described in detail here.

[0026] Step S2: In response to the query text, weight coefficients are assigned to different sources in the multi-source text information based on the relevance of the query text to the multi-source text information. Based on these weight coefficients, the relevance score of each image to the query text is calculated, and then a set of candidate images related to the query text is retrieved from the inverted index.

[0027] In this step, the query text is entered by the user. The keywords in the query text can include item names, file names, paths, attributes, times, actions, and other query terms. Understandably, before the search, the query text needs to be segmented and cleaned to ensure the validity of the query terms.

[0028] In this step, assigning weight coefficients to different sources in the multi-source text information based on the relevance of the query text to the multi-source text information specifically includes: The query terms are obtained by segmenting the query text into words: When the query term matches an image file name in multi-source text information, the image file name is assigned the highest weight. In this embodiment, the preferred weight coefficient is 30.0. When a query term matches a storage path in multi-source text information, the storage path is assigned the second-highest weight. In this embodiment, the preferred weight coefficient is 20.0. When a query term matches visual descriptions in multi-source text information, the base weight is maintained, and the preferred base weight coefficient is 1.00.

[0029] In other words, this step assigns a weight coefficient to the source of multi-source text information and adds this weight coefficient to the result of the correlation analysis between each image and the query term. The purpose of this step is to simulate the user's operation habit of searching by file name and ensure that files with matching file names are ranked first.

[0030] In this step, the specific methods for calculating the relevance score between each image and the query text based on the weight coefficient include the following two methods. In this embodiment, the ranking function corresponding to the BM25 retrieval model is preferred for calculation.

[0031] Method 1: When treating the image filename, storage path, and visual description of each image as independent fields, calculate the score for each query term in each field, then multiply by the corresponding field's weight coefficient, and finally sum the scores for each query term to calculate the relevance score. Specifically: Field 1: Image file name (image_name), weight w_Image=30.0; Field 2: Storage path (Path Parts), weight w_path=20.0; Field 3: Visual description content (description), weight w_desc=1.00, base weight; For each query term t i Calculate its BM25 score in each of the three fields: score(t i , filename), score(t i , filepath), score(t i (description); The score for a single query term is: p i =[score(t i , filename) * 30.0] + [score(t i ,filepath) * 20.0] + [score(t i [, description) * 1.00]; The relevance score P is obtained by summing the scores of multiple query terms, where P = p1 + p2 + ... + P. n .

[0032] Method 2 involves combining the image filename, storage path, and visual description of each image to form a virtual document. The scores of each query term within the virtual document are calculated and summed. Then, the query terms are iterated through to determine the highest-weighted term. Finally, the sum of the scores is weighted by the highest-weighted coefficient to calculate the relevance score. Specifically: Tokenization of the virtual document D: The content of the virtual document is image_name / Path Parts / description, for example, project_proposal.docx / work / important / urgent / This document outlines the initial proposal for the upcoming AI project; Search terms such as "proposal" and "important" are acceptable. Calculate each query term t i Score in virtual document D: IDF(t i )=log((Nn(t i )+0.5) / (n(t i )+0.5)+1) (1) In the formula, N is the total number of documents in the document set, and n(t) i ) is a word containing t i The number of documents; Based on the above, the total number of documents N=3. The number of documents containing "proposal" is n("proposal") = 2; The number of files containing "important" is n("important") = 1; Substituting the above quantities into equation (1) will yield the scores for the query terms "proposal" and "important" respectively, and the sum of the two scores will yield SCORE_base; Then, the query terms are traversed to determine the highest level of the matched field. As mentioned above, the query term "proposal" matches the image file name, so it obtains the highest level weight with a weight coefficient of W=30.0. Finally, the relevance score is obtained by weighting and is SCORE_base * w.

[0033] In this embodiment, after calculating the relevance scores of all images using either of the two methods described above, the relevance scores of multiple images are sorted according to their scores to obtain a relevance score list, and the top N images are output as the candidate image set Top-N. It can be understood that since this embodiment adds the weight of multi-source text information in the score calculation, the image whose query term matches the image file name has the highest relevance score and is ranked first.

[0034] In this embodiment, the former method is preferred because the word frequency ratio in virtual documents is low, and even with the weighting coefficient of multi-source text information, there is no significant change, which is not conducive to highlighting the advantages of image file name matching.

[0035] As can be seen, in this step, a text retrieval mode (preferably BM25) is used to retrieve the Top-N candidate image set related to the query term from the inverted index, achieving a preliminary coarse retrieval. This coarse retrieval can obtain images related to the query term as comprehensively as possible, achieving effective acquisition of target images on the one hand, and reducing the power consumption of subsequent fine retrieval and improving response speed on the other. In addition, in this embodiment, the inverted index includes image storage paths as basic retrieval data. Therefore, as long as the query term contains a path, the corresponding image stored under that path can be retrieved, even if the image name is not standardized, ensuring the comprehensiveness of the coarse retrieval.

[0036] S3: Extract the visual feature vector corresponding to each image in the candidate image set and the text feature vector of the query text, and determine the visual semantic score of each image and the query text based on the visual feature vector and the text feature vector.

[0037] In this step, the candidate image set Top-N is used as the retrieval object. Deep semantic retrieval (CLIP retrieval) is used to extract the visual feature vector corresponding to each image in the candidate image set Top-N and the text feature vector of the query text. In this embodiment, for computational efficiency, it is preferable to first extract the text feature vector T from the query text using the text encoder of CLIP retrieval, and this text feature vector T is extracted only once. Then, the visual feature vectors [I1, I2, ..., I] corresponding to N images in the candidate image set are extracted using the image encoder of CLIP retrieval. N ].

[0038] This step, which determines the visual semantic score of each image and the query text based on visual and textual feature vectors, includes the following specific steps: Step S31: Map the visual feature vector and the text feature vector to the same vector space; Step S32: Calculate the similarity between the visual feature vector and the text feature vector using cosine similarity to obtain a visual semantic score. In this embodiment, the visual semantic score is the similarity score calculated using cosine similarity, which is usually in the range of [-1,1]. Alternatively, the visual semantic score is the normalized (limited to the range of [0,1]) similarity score.

[0039] Specifically, the text feature vector T and the visual feature vectors [I1, I2, ..., I] corresponding to the N images are used to... N Mapping to the same high-dimensional vector space, in which semantically similar images and texts have vectors that are also close to each other, and finally using cosine similarity to calculate the text feature vector T, which is then compared with N visual feature vectors [I1, I2, ..., I...]. NThe similarity score between the images is used as the visual semantic score. The calculated N visual semantic scores are sorted from high to low to obtain a list of visual semantic scores. That is, based on text retrieval, this implementation uses deep semantic retrieval to redefine the relevance between each image in the candidate image set and the query text. This deep semantic retrieval can especially improve the similarity score of images with non-standard names.

[0040] In this embodiment, before extracting feature vectors, each image in the candidate image set needs to undergo preprocessing, including resizing and center cropping, to meet or adapt to CLIP retrieval.

[0041] Step S4: After normalizing the relevance score and visual semantic score corresponding to each image, a weighted fusion algorithm is used to obtain the final score. Based on the final score, a simplified image set Top-K is output from the candidate image set Top-N. It can be understood that K is less than N.

[0042] In this step, a weighted fusion algorithm is used to obtain the final score based on a pre-defined weight strategy for relevance score and visual semantic score. Preferably, the weight coefficient assigned to the relevance score in the pre-defined weight strategy is less than or equal to the weight coefficient for visual semantics; in this embodiment, both are preferably 0.5. The purpose of this step is to assign equal weights to file retrieval and deep semantic retrieval, ultimately ensuring that images matching the query term with the image filename have the highest score and are ranked at the top. Simultaneously, images with non-standard naming but semantically matching are ranked higher to avoid missed detections and significantly improve retrieval efficiency. Of course, when necessary, the weight coefficient assigned to the relevance score can be greater than the weight coefficient for visual semantics. It is understood that the pre-defined weight coefficients in this weight strategy can be flexibly selected according to actual retrieval needs to meet the desired retrieval results.

[0043] In this embodiment, preferably, after obtaining the final score and before outputting the simplified image set, it is determined whether the relevance score of each image exceeds a preset value. This preset value is set according to the actual situation. If so, a reward score is added to the final score based on the reward mechanism. Based on the final score after adding the reward score, the simplified image set Top-K is output from the candidate image set Top-N. This can force the locking of the retrieval results that exactly match the query term and the image file name, prevent the fuzzy semantic scoring of the deep semantic model from interfering with the retrieval results, and further ensure that the image with the query term and the image file name match the highest score.

[0044] In summary, in this embodiment, the image file name, storage path, and visual description content associated with each image are used as multi-source text information, and an inverted index corresponding to the image is constructed using this multi-source text information. This not only effectively utilizes the classification and semantics implicit in the folder hierarchy, improving the effective utilization of resources, but also significantly improves the retrieval accuracy and efficiency in the hybrid mode of text retrieval and deep semantic retrieval. In particular, it can more accurately retrieve images with non-standard names, preventing them from becoming "data dark matter" and being unable to be effectively retrieved, thus solving the fundamental contradiction that it is difficult to achieve both existing retrieval efficiency and accuracy. Additionally, in this embodiment, the retrieval method can always ensure that the image matching the query term and the image file name has the highest score and is ranked at the forefront by assigning weight coefficients to different sources in the multi-source text information and combining a reward mechanism, significantly improving the query accuracy and efficiency while retaining the user's original query habits.

[0045] Embodiment 2:

[0046] The following is a preferred specific embodiment of this solution. For the specific process, see Figure 2 as shown: Implementation environment: Programming language: Python 3.x Core algorithm libraries: rank_bm25 (text retrieval), jieba (Chinese word segmentation), OpenAI CLIP (visual model), PyTorch (deep learning framework).

[0047] Hardware: A GPU server supporting CUDA acceleration (for real-time feature extraction of CLIP).

[0048] Specific implementation steps

[0049] Step S1: Index initialization Traverse all image files in the image library (such as all_image_result.jsonl); Extract the multi-source text information of each image. The multi-source text information at least includes the image file name (image_name) corresponding to each image, the storage path image_path (such as download_image / earthwork / excavator_01.jpg), and the visual description content (description); Construct an inverted index; Initialize the BM25Okapi model.

[0050] Step S2: Coarse ranking retrieval (Recall) The user enters the query term: "excavator"; Enhanced logic: Iterate through the results. If "excavator" appears in image_name, Score *= 30; if it appears in path_parts, Score *= 20. BM25 returns the Top-20 candidate images and their relevance scores (BM25 Score). (It should be noted that due to the introduction of multi-source text information weighting coefficients (enhancement logic), the value of this BM25 Score may be greater than 10.)

[0051] Step S3: Re-rank Load the images from the Top-20 candidate image set and preprocess them into (224, 224) tensors; Input the CLIP model to obtain 512-dimensional image feature vectors (Image Embeddings); Input the visual description: excavator into CLIP Text Encoder to obtain the text feature vectors Text Embeddings; Calculate the cosine similarity to obtain the visual semantic score CLIP Score; Step S4: Output Results Normalize the BM25 Score and CLIP Score to obtain BM25_Norm and CLIP_Norm; The formula used is: Final_Score = 0.5 * BM25_Norm + 0.5 * CLIP_Norm; Decision: If the original BM25 Score > 50, then the Final_Score is increased by 10; Sort the images in descending order of Final_Score and output the Top-K images and their Base64 encodings. Implementation effect

[0052] Scenario testing: Scenario A (Precise Search): Input a specific filename. Because this embodiment introduces "maximum weighting of image filename matching" and "reward mechanism", the target image will be firmly ranked in Top-1, while traditional CLIP search may drown it out among similar images.

[0053] Scenario B (Semantic Search): Input "workers composting". Since the multi-source text information in this embodiment includes the storage path, even if the file name is a meaningless number, text retrieval can recall the corresponding image in the candidate image set. Then, CLIP retrieval can accurately identify the content of the image and recall it to the simplified image set.

[0054] Performance: Thanks to the hybrid retrieval architecture that combines text retrieval and deep semantic retrieval, the end-to-end retrieval time in a million-level image library is controlled within 1 second.

[0055] The above descriptions are merely embodiments of the present invention, and common knowledge regarding specific structures and characteristics in the solutions is not described in detail here. It should be noted that those skilled in the art can make several improvements without departing from the present invention, and these should also be considered within the scope of protection of the present invention. These improvements will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of the claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A vertical domain image retrieval method based on text and semantic fusion, characterized in that, Includes the following steps: Step S1: Based on the multi-source text information associated with each image, construct an inverted index corresponding to that image. The multi-source text information includes at least the image file name, storage path, and visual description content corresponding to each image. Step S2: In response to the query text, weight coefficients are assigned to different sources in the multi-source text information based on the relevance between the query text and the multi-source text information. Based on the weight coefficients, the relevance score of each image to the query text is calculated, and then a set of candidate images related to the query text is retrieved from the inverted index. Step S3: Extract the visual feature vector and the text feature vector of the query text corresponding to each image in the candidate image set, and determine the visual semantic score of each image and the query text based on the visual feature vector and the text feature vector; Step S4: After normalizing the relevance score and visual semantic score corresponding to each image, a weighted fusion algorithm is used to obtain the final score, and a simplified image set is output from the candidate image set based on the final score.

2. The vertical domain image retrieval method based on text and semantic fusion according to claim 1, characterized in that: In step S1, the specific steps for constructing the inverted index corresponding to each image based on the multi-source text information associated with that image include: Step S11: Obtain the image file name, storage path, and visual description content corresponding to each image; Step S12: Combine the image file name, storage path and visual description content corresponding to each image to form a virtual document, and then perform word segmentation to create an inverted index. Alternatively, treat the image file name, storage path and visual description content corresponding to each image as independent fields and create an inverted index.

3. The vertical domain image retrieval method based on text and semantic fusion according to claim 2, characterized in that: In step S2, assigning weight coefficients to different sources in the multi-source text information based on the relevance of the query text to the multi-source text information specifically includes: The query terms are obtained by segmenting the query text into words: When a query term matches an image file name in multi-source text information, the image file name is assigned the highest weight; When a query term matches a storage path in multi-source text information, the storage path is assigned the second-highest weight. When a query term matches visual descriptions in multi-source text information, maintain the basic weight.

4. The vertical domain image retrieval method based on text and semantic fusion according to claim 3, characterized in that: In step S3, the specific methods for calculating the relevance score between each image and the query text based on the weighting coefficient include the following two: Method 1: When the image file name, storage path and visual description content are treated as independent fields, the score of each query term in each field is calculated separately, then multiplied by the weight coefficient of the corresponding field, and finally the scores of each query term are summed to calculate the relevance score. Method 2: When the image file name, storage path and visual description content are combined to form a virtual document, the scores of each query term in the virtual document are calculated and summed. Then, the query terms are traversed to determine the highest weight of the hit. Finally, the scores are weighted and calculated with the highest weight coefficient to obtain the relevance score.

5. The vertical domain image retrieval method based on text and semantic fusion according to claim 1, characterized in that: In step S3, CLIP retrieval is used to extract the visual feature vector corresponding to each image in the candidate image set and the text feature vector of the query text. Before extraction, each image in the candidate image set is preprocessed, including size adjustment and center cropping.

6. The vertical domain image retrieval method based on text and semantic fusion according to claim 1, characterized in that: In step S3, the specific steps for determining the visual semantic score of each image and the query text based on the visual feature vector and the text feature vector include: Step S31: Map the visual feature vector and the text feature vector to the same vector space; Step S32: Calculate the similarity between the visual feature vector and the text feature vector using cosine similarity to obtain a visual semantic score.

7. The vertical domain image retrieval method based on text and semantic fusion according to claim 1, characterized in that: In step S4, after obtaining the final score and before outputting the simplified image set, it is determined whether the relevance score of each image exceeds a preset value. If so, a bonus score is added to the final score based on the reward mechanism, and the simplified image set is output from the candidate image set based on the final score after adding the bonus score.

8. The vertical domain image retrieval method based on text and semantic fusion according to claim 1, characterized in that: In step S4, a weighted fusion algorithm is used to obtain the final score based on a preset weighting strategy for the relevance score and the visual semantic score.

9. The vertical domain image retrieval method based on text and semantic fusion according to claim 1, characterized in that: In step S2, BM25 text retrieval is used to retrieve the candidate image set from the inverted index.

Images