A cross-modal image-text retrieval method for a professional field

Abstract

The application provides a cross-modal image-text retrieval method for professional fields, which comprises the following steps: constructing an adapter according to an efficient parameter fine-tuning technology, and training the adapter through a contrastive learning training mode; extracting and expanding features of image-text data of different professional fields uploaded by a user according to the trained adapter, and constructing an index structure based on a hierarchical graph by using feature embedding vectors of the expanded text and image; selecting a corresponding professional field retrieval interface according to query content of the user, and extracting features of the query content by using a trained adapter of the corresponding field; and performing approximate nearest neighbor search on the extracted features of the query content in the constructed index structure, obtaining the most relevant query result, and returning the result to the user. Based on the scheme provided by the application, accurate and efficient cross-modal retrieval services for different professional field data can be provided for the user, and the training cost is low.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of content retrieval, and in particular to a cross-modal image-text retrieval method for professional fields. BACKGROUND

[0002] With the rapid development of the Internet and big data technology, multimedia data mainly in the form of images and texts has shown an explosive growth. How to provide accurate and efficient cross-modal retrieval services for users in the massive data has become an urgent problem in the field of content retrieval. In addition, the rapid iteration of various industries has also made the data more and more professional.

[0003] However, the traditional cross-modal retrieval scheme basically focuses on solving the data retrieval problem in the general field, and the feature extraction capability for knowledge-intensive image-text data in the professional field is poor, and the retrieval accuracy is not high. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the related art to some extent.

[0005] To this end, the present application aims to provide a method to realize accurate and efficient cross-modal retrieval of users in different professional field data, which comprises:

[0006] constructing an adapter according to the efficient parameter fine-tuning technology, and training the adapter through a contrastive learning training method;

[0007] extracting features from image-text data uploaded by users in different professional fields according to the trained adapter and expanding the features, and constructing an index structure based on a hierarchical graph using the feature embedding vectors of the expanded texts and images;

[0008] selecting a retrieval interface of a corresponding professional field according to the query content of the user, and extracting features from the query content using the trained adapter of the corresponding field;

[0009] performing approximate nearest neighbor search on the extracted features of the query content in the constructed index structure, obtaining the most relevant query results, and returning the results to the user.

[0010] Optionally, the constructing an adapter according to the efficient parameter fine-tuning technology comprises:

[0011] The adapters are constructed by adding learnable scaling vectors to the outputs of the Key, Value sub-layers and the feed-forward network layer in the multi-head attention layers of the Transformer networks of the text and visual encoders of the multi-modal pre-trained model with all parameters frozen, wherein the added learnable scaling vectors are collectively referred to as SpecializedAdapter modules.

[0012] Optionally, the adapters are trained in a training manner of contrastive learning, including:

[0013] The Specialized Adapter model is trained in the training manner of contrastive learning using Chinese text-image data sets of different professional fields.

[0014] Optionally, the adapters are trained in a training manner of contrastive learning, including:

[0015] The text-image data uploaded by the user is classified according to different professional fields, and the multi-modal pre-trained model with the Specialized Adapter module of the corresponding field is used for feature extraction according to different categories;

[0016] The feature embedding vectors of the text and image obtained after feature extraction are extended by one dimension to store the file number of the modal data corresponding to the feature embedding vectors of the text and image;

[0017] The feature embedding vectors of the text and image are extended, and the HNSW algorithm is combined to construct the index structure based on the hierarchical graph.

[0018] Optionally, the feature embedding vectors of the text and image are extended, and the HNSW algorithm is combined to construct the index structure based on the hierarchical graph, including:

[0019] An empty graph structure is created, each data feature embedding is regarded as a vector node, and the nodes are inserted in turn, each new node is randomly assigned a level, the higher the level, the fewer the nodes;

[0020] When inserting a new node, start from the highest level, find the nearest neighbor of the current node belonging to the level through greedy search, and repeat this process in lower levels to determine more accurate neighbor nodes;

[0021] After finding the nearest neighbor nodes, the inserted new node is connected with these neighbor nodes, and the graph structure is updated.

[0022] Optionally, the selecting the corresponding professional field retrieval interface according to the query content of the user, the trained adapter of the corresponding field is selected to extract features of the query content, including:

[0023] According to the selection of the professional field retrieval interface by the user, the multi-modal pre-training model with the SpecializedAdapter module of the corresponding field is selected to extract features of the query content of the user, and a dimension of the feature embedding vector of the extracted query content is expanded to store 0 values.

[0024] Optionally, the approximate nearest neighbor search is performed on the extracted features of the query content in the built index structure, the most relevant query result is obtained and returned to the user, including:

[0025] The feature embedding vector of the extracted query content is inserted into the built index structure as a vector node;

[0026] In the built index structure, starting from the highest level entry point, search layer by layer downward; wherein, in each layer, the node closest to the query vector node is found by greedy search, and the node is used as the starting point for further search until a closer node is found; if a closer node cannot be found in the current layer, search is performed in the next layer, and the process is repeated until the bottom layer is reached;

[0027] In the bottom layer, the node closest to the query vector node is searched, and the corresponding file is returned through the file number identified by the last dimension of the vector node.

[0028] Optionally, it further includes:

[0029] The cosine distance is used as the distance between the vector nodes on the index structure, and the calculation expression is:

[0030]

[0031] Wherein, the Cosine Distance is the cosine distance, e is the feature embedding vector of the user query, e is the feature embedding vector of the data file, and the reciprocal of the cosine distance represents the cosine similarity between the corresponding data.

[0032] The technical scheme provided by the embodiment of the application at least brings the following beneficial effects:

[0033] By adding a learnable vector for dynamic scaling at a specific position of the multi-head attention layer and the feedforward network layer in the Transformer network on the visual and text side of the model, and training in the manner of contrastive learning, the model can flexibly and accurately extract features for data in different professional fields, and the amount of parameters required for training is only about 0.017% of the original model; by using the HNSW vector approximate nearest neighbor search algorithm, the efficiency of searching for user query content in a large amount of high-dimensional vectors is improved. The present application can be used for cross-modal retrieval of professional field image-text data, and provides efficient and accurate cross-modal retrieval service for different professional field image-text data for users while having low training overhead.

[0034] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following and / or can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0035] The above and / or additional aspects and advantages of the application will become apparent and be readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0036] Figure 1 is a flowchart of a cross-modal image-text retrieval method for a professional field according to an exemplary embodiment;

[0037] Figure 2 is a schematic diagram of a parameter fine-tuning structure of an adapter according to an exemplary embodiment;

[0038] Figure 3 is a correct rate schematic diagram of different models performing a zero-shot classification task on pictures in a professional field data set according to an exemplary embodiment;

[0039] Figure 4 is a recall rate schematic diagram of the model of the present application performing a picture-to-text retrieval task in a professional field data set according to an exemplary embodiment;

[0040] Figure 5 is a recall rate schematic diagram of the model of the present application performing a text-to-picture retrieval task in a professional field data set according to an exemplary embodiment;

[0041] Figure 6 is a comparison schematic diagram of the time consumption of an index construction process varying with the number of files according to an exemplary embodiment;

[0042] Figure 7 is a comparison schematic diagram of the time consumption of a search process varying with the number of files according to an exemplary embodiment. DETAILED DESCRIPTION

[0043] Embodiments of the present application are described below in detail, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0044] With the rapid development of the Internet and big data technology, multimedia data mainly in the form of pictures and texts has shown an explosive growth. How to provide accurate and efficient cross-modal retrieval services for users in the massive data has become an urgent problem to be solved in the field of content retrieval. In addition, the rapid iteration of technologies in various industries has also made the specialization of data increasingly high. However, the traditional cross-modal retrieval scheme basically focuses on solving the data retrieval problem in the general field, and the feature extraction capability for knowledge-intensive picture-text data in the professional field is poor, and the retrieval accuracy is not high. Therefore, it is necessary to design a scheme that can provide efficient and accurate cross-modal data retrieval services for users in various professional fields.

[0045] In view of the problem that users have difficulty in model training and efficient and accurate cross-modal retrieval for data covering multiple professional fields, the present application proposes a cross-modal picture-text retrieval method for professional field picture-text data.

[0046] Figure 1 is a flowchart of a cross-modal picture-text retrieval method for professional field picture-text data according to an embodiment of the present application, as shown in Figure 1 , the method comprises the following steps:

[0047] Step 101, constructing an adapter according to an efficient parameter fine-tuning technology, and training the adapter by a contrastive learning training mode.

[0048] This step involves the construction and training phase of the adapter.

[0049] The efficient parameter fine-tuning technology is a kind of efficient parameter fine-tuning technology for pre-training large models. Since the parameter quantity of the existing pre-training large models is very large, using the traditional full amount fine-tuning method to retrain the model is no longer a suitable choice. The efficient parameter fine-tuning technology usually freezes most or all parameters of the original model, and adds corresponding structures for fine-tuning according to different specific tasks. Only the weight parameters in the new structure need to be updated during model training, which greatly reduces the computing power requirement during fine-tuning training. The present application uses the efficient parameter fine-tuning technology to improve the feature extraction capability of the original model for specific professional field picture-text data by training only 0.017% of the parameters of the original model.

[0050] Specifically, the present application is based on a multi-modal pre-training model with all parameters frozen, and a learnable scaling vector is added to the output of the Key, Value sublayer and feedforward network layer in the multi-head attention layer of the Transformer network of the text and visual end encoders to construct an adapter. The added learnable scaling vector is collectively referred to as a Specialized Adapter module.

[0051] In one possible embodiment, a multi-modal pre-training model CN-CLIP ViT-H / 14 is used as the basis, the number of layers of the multi-head attention module of the text end feature extractor is 24, and the number of heads is also 24; the number of layers of the multi-head attention module of the visual end feature extractor is 32, and the number of heads is also 16. The feature embedding dimensions of the text and picture data extracted by them are both 1024.

[0052] In one embodiment of the present application, as shown in Figure 2 , three groups of learnable vectors l k , l v and l ff are injected into the outputs of the Key, Value sublayer and feedforward network layer in the multi-head attention layer of the Transformer network of the text and visual end encoders, respectively, to finely amplify or suppress the output results of the original positions. They can be collectively referred to as Specialized Adapter. If the output of the Key sublayer needs to be amplified, a positive learnable vector is added; otherwise, if it needs to be suppressed, a negative learnable vector is added. The degree of suppression or amplification is determined by the specific value of l k .

[0053] After designing the structure of the adapter, the Specialized Adapter model is trained using different professional field Chinese text and picture data sets through a contrastive learning training method, thereby having the learning and representation ability of professional field knowledge.

[0054] It should be noted that through this training method, only a small number of parameters in the Specialized Adapter module need to be trained, which accounts for about 0.017% of the total number of parameters of the original model. After training is completed, the model with the corresponding field Specialized Adapter module can be used for feature extraction according to the user's retrieval needs of professional field text and picture data.

[0055] Step 102, according to the trained adapter, the features of the user uploaded different professional field of data are extracted and expanded, and the feature embedding vectors of the expanded text and image are used to construct the index structure based on hierarchical graph.

[0056] This step involves the construction process of the index structure.

[0057] Firstly, the user uploaded different professional field of data is classified, and the multi-modal pre-training model with the corresponding field of Specialized Adapter module is used for feature extraction according to different categories; secondly, the feature embedding vectors of the text and image obtained after feature extraction are expanded by one dimension, which is used to store the file number of the modal data corresponding to the feature embedding vectors of the text and image; finally, according to the expanded feature embedding vectors of the text and image, the index structure based on hierarchical graph is constructed by combining HNSW algorithm.

[0058] HNSW (Hierarchical Navigable Small World) algorithm is a high efficient vector approximate nearest neighbor search algorithm, mainly used for searching the nearest neighbor point in high dimensional space, especially on large scale data set. The algorithm is based on the theory of navigable small world algorithm, which quickly finds the nearest neighbor nodes of the query nodes by constructing and searching the hierarchical graph structure. The application uses this algorithm to realize efficient search in massive high dimensional vectors.

[0059] As a possible implementation, the construction process of the index structure based on hierarchical graph combined with HNSW algorithm includes:

[0060] Firstly, an empty graph structure is created, each data feature embedding is regarded as a vector node, and the nodes are inserted in turn, each new node is randomly assigned a level, the higher the level, the fewer the nodes; when inserting a new node, starting from the highest level, the nearest neighbor of the current node is found through greedy search, and the process is repeated in lower level to determine more accurate neighbor nodes; after finding the nearest neighbor nodes, the inserted new node is connected with these neighbor nodes, and the graph structure is updated. Finally, by increasing the connection and adjusting the edge, the navigability of the hierarchical graph is further optimized.

[0061] Step 103, according to the user's query content, the corresponding professional field of retrieval interface is selected, and the trained adapter of the corresponding field is used to extract the features of the query content.

[0062] This step involves the feature extraction process of the query content.

[0063] In the embodiments of the present application, according to the user's selection of the professional field retrieval interface, a multi-modal pre-training model with a corresponding field Specialized Adapter module is selected to extract the features of the user's query content, and a dimension of 0 is expanded for the extracted feature embedding vector of the query content. Finally, the dimension of the feature embedding obtained is 1025.

[0064] Step 104: In the built index structure, an approximate nearest neighbor search is performed on the extracted features of the query content to obtain the most relevant query result and return it to the user.

[0065] This step involves the retrieval process of the query content.

[0066] In the embodiments of the present application, the feature embedding vector of the query content extracted in step 103 is inserted into the built index structure as a vector node. In the built index structure, the search process starts from the highest level entry point and searches layer by layer downward. In each layer, the node closest to the query vector node is found through greedy search, and the node is used as the starting point for further search until a closer node is found. If no closer node is found in the current layer, the search is transferred to the next layer, and the process is repeated until the bottom layer is reached. In the bottom layer, the node closest to the query vector node is searched, and the corresponding file is returned through the file number identified by the last dimension of the vector node.

[0067] As a possible implementation, the cosine distance is taken as the distance between the vector nodes on the index structure, and the calculation expression is:

[0068]

[0069] where Cosine Distance is the cosine distance, e is the feature embedding vector of the user query, e is the feature embedding vector of the data file, and the reciprocal of the cosine distance represents the cosine similarity between the corresponding data.

[0070] In order to verify the effectiveness of the present application, in one embodiment of the present application, the feature extraction capability of the present application in the professional field data and the time consumption in the index construction and retrieval process are tested.

[0071] The hardware environment of the experiment is: GPU: two NVIDIA GeForce RTX 4090D; CPU: 30vCPU Intel(R) Xeon(R) Platinum 8474C. The base model of the application uses CN-CLIP, the text end feature extractor uses RoBERTa-wwm-Large, the visual end feature extractor uses ViT-H / 14, and the dimension of the generated feature embedding vector is 1025. The total number of parameters of the base model is 958M. Figure 1 The specific structure of the Specialized Adapter module of the application and the position added to the original model.

[0072] Figure 3 The correct rate comparison (which can be regarded as a picture search text task with a recall number of 1) when using the model equipped with the Specialized Adapter module trained in different professional field data to perform the picture zero-shot classification task, the fine-grained classification label of the picture in each field is used as the text, and the comparison model is the base model CN-CLIP. The data set is the Chinese version of the professional field data set oxford-pets, DTD, EuroSAT, oxford-flower-102, food-101, which belong to pet, object texture, satellite geographic image, flower, and food categories respectively.

[0073] The experimental results of Figure 3 It can be seen from the experimental results that the application has strong learning and representation ability for knowledge in different professional fields, and the overall correct rate is improved by about 14.4% compared with the previous SOTA method.

[0074] Figure 4 And Figure 5 The recall rate of the application when performing the picture search text task and the text search picture task on the professional field data set respectively. Figure 6 The time consumption of the retrieval model of the application when using the feature embedding vector of the file to construct the index structure, Figure 7 The time consumption of the search process of the retrieval model on the built index structure (a single text or picture can be regarded as a file), the number of approximate neighbors of the HNSW algorithm is set to 5, and the maximum number of connections between each node and the remaining nodes is 16. The comparison model selects the ciphertext retrieval scheme SSSW2 and SSRB2 based on the text semantics, and the dimension of the generated text feature embedding vector is 768.

[0075] The experimental results of Figure 7The experimental results show that the time consumption of the search model in the approximate nearest neighbor search process of the vector on the built index structure is very stable with the increase of the file quantity.

[0076] It should be understood that the steps can be reordered, added, or deleted using the various forms of flow shown above. For example, the steps described in the present application can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which is not limited herein.

[0077] The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for cross-modal document image retrieval for a professional field, characterized in that, The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure.

2. The method of claim 1, wherein, The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure.

3. The method of claim 2, wherein, The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal retrieval system based on a hierarchical graph index structure. The application relates to a method for constructing a multi-modal According to the user's selection of the professional field retrieval interface, a multi-modal pre-training model with a corresponding field Specialized Adapter module is selected to extract the features of the user's query content, and a dimension of 0 is expanded for the extracted query content feature embedding vector.

4. The method of claim 3, wherein, The approximate nearest neighbor search is performed on the built index structure for the extracted query content features to obtain the most relevant query results and return them to the user, including: The extracted query content feature embedding vector is inserted into the built index structure as a vector node. In the built index structure, starting from the highest level entry point, search layer by layer downward; wherein in each layer, the nearest node to the query vector node is found through greedy search, and the node is used as the starting point for further search until a closer node is found; if no closer node is found in the current layer, search in the next layer, and repeat the process until the bottom layer is reached. In the bottom layer, the nearest node to the query vector node is searched, and the corresponding file is returned through the file number identified by the last dimension of the vector node.

5. The method of claim 4, wherein, Also including: The cosine distance is used as the distance between the vector nodes on the index structure, and the expression is: wherein, is the cosine distance, is the feature embedding vector of the query, is the feature embedding vector of the data file, and the inverse of the cosine distance represents the cosine similarity between the corresponding data.

Images