Cross-modal image-text retrieval method for professional field

Abstract

The invention provides a cross-modal image-text retrieval method for the professional field, and the method comprises the steps: constructing an adapter according to an efficient parameter fine tuning technology, and training the adapter through a contrast learning training mode; according to the trained adapter, feature extraction and extension are carried out on image-text data, uploaded by a user, in different professional fields, and feature embedding vectors of expanded texts and images are utilized to construct an index structure based on a hierarchical graph; selecting a retrieval interface of a corresponding professional field according to query content of a user, and selecting a trained adapter of the corresponding field to perform feature extraction on the query content; and carrying out approximate nearest neighbor search on the extracted characteristics of the query content in the established index structure to obtain a most relevant query result, and returning the most relevant query result to the user. Based on the scheme provided by the invention, while accurate and efficient cross-modal retrieval service aiming at data in different professional fields is provided for the user, the training overhead is relatively low.

Description

Technical Field

[0001] The present application relates to the field of content retrieval, and in particular to a cross-modal image and text retrieval method for a professional field. Background Art

[0002] With the rapid development of the Internet and big data technology, multimedia data, mainly pictures and texts, has experienced explosive growth. How to provide users with accurate and efficient cross-modal retrieval services in massive amounts of data has become an increasingly urgent problem in the field of content retrieval. In addition, the rapid iteration of technology in various industries has also led to an increasing degree of specialization in data.

[0003] However, traditional cross-modal retrieval solutions are basically focused on solving data retrieval problems in general fields, while for knowledge-intensive graphic and text data in professional fields, their feature extraction capabilities are poor and the retrieval accuracy is not high. Summary of the invention

[0004] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0005] To this end, the purpose of this application is to propose a method to enable users to perform accurate and efficient cross-modal retrieval in different professional field data, the method comprising:

[0006] Building an adapter based on an efficient parameter fine-tuning technique, and training the adapter through a contrastive learning training method;

[0007] The trained adapter is used to extract and expand the features of the graphic and text data in different professional fields uploaded by users, and the feature embedding vectors of the expanded text and images are used to construct an index structure based on a hierarchical graph;

[0008] Select a search interface in a corresponding professional field according to the user's query content, and use a trained adapter in the corresponding field to extract features from the query content;

[0009] An approximate nearest neighbor search is performed for the features of the extracted query content in the established index structure to obtain the most relevant query results and return them to the user.

[0010] Optionally, constructing an adapter according to the efficient parameter fine-tuning technology includes:

[0011] Based on a multimodal pre-trained model with all parameters frozen, the adapter is constructed by adding a learnable scaling vector to the outputs of the Key, Value sublayers and feedforward network layers of the multi-head attention layer in the Transformer network of its text-end and vision-end encoders, respectively, wherein the added learnable scaling vectors are collectively referred to as SpecializedAdapter modules.

[0012] Optionally, the training of the adapter by contrastive learning includes:

[0013] The Specialized Adapter model is trained using Chinese image and text datasets from different professional fields through a comparative learning training method.

[0014] Optionally, the feature extraction and expansion of the graphic and text data in different professional fields uploaded by the user according to the trained adapter, and the construction of the hierarchical graph-based index structure using the feature embedding vectors of the expanded text and image, include:

[0015] Classify the graphic and text data in different professional fields uploaded by users, and use the multimodal pre-trained model with the Specialized Adapter module of the corresponding field to extract features for different categories;

[0016] The feature embedding vectors of the text and image obtained after feature extraction are expanded by one dimension to store the file numbers of the modal data corresponding to the feature embedding vectors of the text and image;

[0017] According to the expanded feature embedding vectors of text and image, the HNSW algorithm is combined to construct a hierarchical graph-based index structure.

[0018] Optionally, the step of constructing a hierarchical graph-based index structure based on the expanded feature embedding vectors of the text and image in combination with the HNSW algorithm includes:

[0019] Create an empty graph structure, treat each data feature embedding as a vector node, and insert nodes one by one. Each new node will be randomly assigned a level. The higher the level, the fewer nodes there are.

[0020] When inserting a new node, start from the highest level and use greedy search to find the nearest neighbor of the level to which the current node belongs, and repeat this process at lower levels to determine more precise neighbor nodes;

[0021] After finding the nearest neighbor nodes, the inserted new node is connected to these neighbor nodes and the graph structure is updated.

[0022] Optionally, selecting a search interface corresponding to a professional field according to the user's query content, and selecting a trained adapter in the corresponding field to perform feature extraction on the query content includes:

[0023] According to the user's choice of professional field retrieval interface, a multimodal pre-trained model with a SpecializedAdapter module of the corresponding field is selected to extract features of the user's query content, and a dimension is expanded for the feature embedding vector of the extracted query content to store 0 values.

[0024] Optionally, performing an approximate nearest neighbor search for the extracted query content features in the established index structure to obtain the most relevant query results and return them to the user includes:

[0025] Insert the feature embedding vector of the extracted query content into the established index structure as a vector node;

[0026] In the established index structure, we start from the entry point of the highest layer and search downward layer by layer. In each layer, we use greedy search to find the node closest to the query vector node, and use this node as the starting point to continue searching until a closer node is found. If no closer node is found in the current layer, we go to the next layer to search, and repeat this process until we reach the bottom layer.

[0027] At the bottom layer, the node closest to the queried vector node is searched, and the corresponding file is returned by the file number identified by the last dimension of the vector node.

[0028] Optionally, also include:

[0029] Taking the cosine distance as the distance between vector nodes on the index structure, the calculation expression is:

[0030]

[0031] Wherein, Cosine Distance is the cosine distance, e is the feature embedding vector of the user query, e is the feature embedding vector of the data file, and the reciprocal of the cosine distance represents the cosine similarity between the corresponding data.

[0032] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0033] By adding learnable vectors for dynamic scaling at specific positions of the multi-head attention layer and feedforward network layer in the Transformer network at the visual and text ends of the model, and combining training with contrastive learning, the model can flexibly and accurately extract features from data in different professional fields, and the number of parameters that need to be trained only accounts for about 0.017% of the original model; by using the HNSW vector approximate nearest neighbor search algorithm, the efficiency of searching for user query content in massive high-dimensional vectors is improved. This application can be used for cross-modal retrieval of graphic and text data in professional fields, providing users with efficient and accurate cross-modal retrieval services for graphic and text data in different professional fields, while having low training overhead.

[0034] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0036] Figure 1 is a flow chart showing a cross-modal image and text retrieval method for a professional field according to an exemplary embodiment;

[0037] Figure 2 is a schematic diagram of a parameter fine-tuning structure of an adapter according to an exemplary embodiment;

[0038] Figure 3 is a schematic diagram showing the accuracy of different models when performing a zero-sample classification task of images in a professional field dataset according to an exemplary embodiment;

[0039] Figure 4 It is a schematic diagram of the recall rate of the model of the present application when performing the image-based text search task in a professional field data set according to an exemplary embodiment;

[0040] Figure 5 is a schematic diagram of the recall rate of the model of the present application when performing a text-to-image search task in a professional field data set according to an exemplary embodiment;

[0041] Figure 6 is a schematic diagram showing a comparison of the time consumption of an index building process with the change of the number of files according to an exemplary embodiment;

[0042] Figure 7 The figure is a schematic diagram showing a comparison of the time consumption of a search process with the change of the number of files according to an exemplary embodiment. DETAILED DESCRIPTION

[0043] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0044] With the rapid development of the Internet and big data technology, multimedia data with images and text as the main types has shown explosive growth. How to provide users with accurate and efficient cross-modal retrieval services in massive data has become an increasingly urgent problem in the field of content retrieval. In addition, the rapid iteration of technologies in various industries has also made data more and more specialized. However, traditional cross-modal retrieval solutions are basically focused on solving data retrieval problems in general fields, while for knowledge-intensive graphic and text data in professional fields, their feature extraction capabilities are poor and the retrieval accuracy is not high. Therefore, it is very necessary to design a solution that can provide users with efficient and accurate cross-modal data retrieval services in various professional fields.

[0045] In response to the problem that users find it difficult to train models and perform efficient and accurate cross-modal retrieval for data covering multiple professional fields, this application proposes a cross-modal image and text retrieval method for image and text data in professional fields.

[0046] Figure 1 is a flow chart of a cross-modal image and text retrieval method for professional field image and text data according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0047] Step 101 , construct an adapter according to the efficient parameter fine-tuning technology, and train the adapter by contrastive learning.

[0048] This step involves the adapter construction and training phase.

[0049] Efficient parameter fine-tuning technology is an efficient parameter fine-tuning technology for pre-trained large models. Since the parameters of existing pre-trained large models are very large, using the traditional full-scale fine-tuning method to retrain the model is no longer a suitable option. Efficient parameter fine-tuning technology usually freezes most or all of the parameters of the original model, and adds corresponding structures for fine-tuning according to different specific tasks. When training the model, only the weight parameters in the newly added structure need to be updated, which greatly reduces the computing power required for fine-tuning training. The present invention utilizes efficient parameter fine-tuning technology to improve the original model's ability to extract features for graphic data in specific professional fields while only training parameters equivalent to 0.017% of the original model.

[0050] Specifically, the present application is based on a multimodal pre-trained model with all parameters frozen, and constructs an adapter by adding a learnable scaling vector to the output of the Key, Value sublayer and feedforward network layer of the multi-head attention layer in the Transformer network of its text-end and vision-end encoders, respectively, wherein the added learnable scaling vectors are collectively referred to as Specialized Adapter modules.

[0051] In one possible embodiment, a multimodal pre-trained model CN-CLIP with all parameters frozen is used. ViT-H / 14 Based on , the multi-head attention module of the text-side feature extractor has 24 layers and 24 heads; the multi-head attention module of the visual-side feature extractor has 32 layers and 16 heads. The feature embedding dimensions of the text and image data they extract are both 1024.

[0052] In one embodiment of the present application, Figure 2 As shown in the figure, three sets of learnable vectors l are injected into the output of the Key, Value sublayer and feedforward network layer of the multi-head attention layer in the Transformer network of the text-side and visual-side encoders, respectively. k , l v and l ff , which is used to perform fine-grained amplification or suppression of the output results of the original position. They can be collectively referred to as Specialized Adapter. If the output of the Key sublayer needs to be amplified, a positive learnable vector will be added; conversely, if it needs to be suppressed, a negative learnable vector will be added. The degree of suppression or amplification is determined by l k The specific value of is determined by .

[0053] After designing the structure of the adapter, we used Chinese text and image datasets from different professional fields and trained the Specialized Adapter model through comparative learning, thus enabling it to learn and represent knowledge in various professional fields.

[0054] It should be noted that through this training method, only a small number of parameters in the Specialized Adapter module need to be trained, which account for about 0.017% of the total parameters of the original model. After the training is completed, the model equipped with the Specialized Adapter module of the corresponding field can be used for feature extraction according to the user's retrieval needs for professional field graphic data.

[0055] Step 102, extracting and expanding the features of the graphic and text data in different professional fields uploaded by the user according to the trained adapter, and constructing an index structure based on a hierarchical graph using the feature embedding vectors of the expanded text and image.

[0056] This step involves the construction process of the index structure.

[0057] Firstly, the graphic and text data in different professional fields uploaded by users are classified, and for different categories, feature extraction is performed using a multimodal pre-trained model equipped with Specialized Adapter modules in the corresponding fields. Secondly, the feature embedding vectors of text and images obtained after feature extraction are expanded by one dimension to store the file numbers of the modal data corresponding to the feature embedding vectors of text and image. Finally, according to the expanded feature embedding vectors of text and image, the HNSW algorithm is combined to construct an index structure based on a hierarchical graph.

[0058] The HNSW (Hierarchical Navigable Small World) algorithm is an efficient vector approximate nearest neighbor search algorithm, which is mainly used to search for the nearest neighbor point in high-dimensional space, especially on large-scale data sets. The algorithm is based on the theory of navigable small world algorithm, and quickly finds the nodes that are the nearest neighbors of the query node by constructing and searching a hierarchical graph structure. This application uses this algorithm to achieve efficient search in massive high-dimensional vectors.

[0059] As a possible implementation method, the process of constructing an index structure based on a hierarchical graph in combination with the HNSW algorithm includes:

[0060] First, an empty graph structure is created. Each data feature is embedded as a vector node, and nodes are inserted in sequence. Each new node is randomly assigned a level. The higher the level, the fewer nodes there are. When inserting a new node, starting from the highest level, a greedy search is used to find the nearest neighbor of the level to which the current node belongs, and this process is repeated at lower levels to determine more accurate neighbor nodes. After finding the nearest neighbor nodes, the inserted new node is connected to these neighbor nodes and the graph structure is updated. Finally, the navigation of the hierarchical graph is further optimized by adding connections and adjusting edges.

[0061] Step 103, select a search interface corresponding to a professional field according to the user's query content, and select a trained adapter in the corresponding field to perform feature extraction on the query content.

[0062] This step involves the feature extraction process of the query content.

[0063] In an embodiment of the present application, based on the user's selection of a professional field retrieval interface, a multimodal pre-trained model equipped with a Specialized Adapter module for the corresponding field is selected to perform feature extraction on the user's query content, and the feature embedding vector of the extracted query content is expanded by one dimension to store a value of 0, and the final feature embedding dimension is 1025.

[0064] Step 104 , performing an approximate nearest neighbor search for the extracted query content features in the established index structure, obtaining the most relevant query results and returning them to the user.

[0065] This step involves the retrieval process of the query content.

[0066] In an embodiment of the present application, the feature embedding vector of the query content extracted in step 103 is inserted into the established index structure as a vector node; in the established index structure, the search process starts from the entry point of the highest layer and searches downward layer by layer; in each layer, a node closest to the queried vector node is found through greedy search, and the search is continued using the node as the starting point until a closer node is found; if a closer node cannot be found in the current layer, the search is transferred to the next layer, and this process is repeated until the bottom layer is reached; at the bottom layer, the node closest to the queried vector node is searched, and the corresponding file is returned through the file number identified by the last dimension of the vector node.

[0067] As a possible implementation method, the cosine distance is used as the distance between vector nodes on the index structure, and the calculation expression is:

[0068]

[0069] Among them, Cosine Distance is the cosine distance, e is the feature embedding vector of the user query, e is the feature embedding vector of the data file, and the reciprocal of the cosine distance represents the cosine similarity between the corresponding data.

[0070] In order to verify the effectiveness of the present application, in one embodiment of the present application, the present application tests the feature extraction capability of the present application in professional field data and the time consumption in the index construction and retrieval process.

[0071] The hardware environment of the experiment is: GPU: two NVIDIA GeForce RTX 4090D; CPU: 30vCPU Intel(R)Xeon(R)Platinum 8474C. The basic model of this application uses CN-CLIP, the text-side feature extractor uses RoBERTa-wwm-Large, and the visual-side feature extractor uses ViT-H / 14. The dimension of the generated feature embedding vector is 1025, and the total number of parameters of the basic model is 958M. Figure 1 The specific structure of the Specialized Adapter module of the present invention and the position where it is added to the original model.

[0072] Figure 3 In order to compare the accuracy of zero-shot image classification tasks using models equipped with Specialized Adapter modules trained on data from different professional fields (which can be regarded as an image-to-text search task with a recall of 1), the fine-grained classification labels of images in each field are used as text. The comparison model is the basic model CN-CLIP, and the datasets are the Chinese versions of the professional field datasets oxford-pets, DTD, EuroSAT, oxford-flower-102, and food-101, which belong to the categories of pets, object textures, satellite geographic images, flowers, and food, respectively.

[0073] Depend on Figure 3 The experimental results show that the present invention has a strong ability to learn and represent knowledge in different professional fields, and the overall accuracy is improved by about 14.4% compared with the previous SOTA method.

[0074] Figure 4 and Figure 5 They are respectively the recall rates of the present invention when performing image-to-text search tasks and text-to-image search tasks on professional field datasets. Figure 6 The time consumption of the retrieval model of the present invention when using the feature embedding vector of the file to build the index varies with the number of files. Figure 7 In order to measure the variation of the time consumption of the search process of the retrieval model on the established index structure with the number of files (a single text or image can be regarded as a file), the number of approximate neighbors of the HNSW algorithm is set to 5, and the maximum number of connections between each node and the remaining nodes is 16. The comparison model selects the text semantics-based ciphertext retrieval schemes SSSW2 and SSRB2, which generate text feature embedding vectors with a dimension of 768.

[0075] Depend on Figure 7The experimental results show that the time consumption of the retrieval model of the present invention for performing the approximate nearest neighbor search of vectors on the established index structure varies very stably with the increase in the number of files. Compared with the other two solutions, the more files there are, the more obvious the speed advantage of the present invention in executing the search process.

[0076] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of this application can be achieved, and this document is not limited here.

[0077] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A cross-modal image and text retrieval method for professional fields, It is characterized in that include: Building an adapter based on an efficient parameter fine-tuning technique, and training the adapter through a contrastive learning training method; The trained adapter is used to extract and expand the features of the graphic and text data in different professional fields uploaded by users, and the feature embedding vectors of the expanded text and images are used to construct an index structure based on a hierarchical graph; Select a search interface in a corresponding professional field according to the user's query content, and use a trained adapter in the corresponding field to extract features from the query content; An approximate nearest neighbor search is performed for the features of the extracted query content in the established index structure to obtain the most relevant query results and return them to the user.

2. The method according to claim 1, It is characterized in that The adapter is constructed according to the efficient parameter fine-tuning technology, including: Based on a multimodal pre-trained model with all parameters frozen, the adapter is constructed by adding a learnable scaling vector to the outputs of the Key, Value sublayers and feedforward network layers of the multi-head attention layer in the Transformer network of its text-end and vision-end encoders, respectively, wherein the added learnable scaling vectors are collectively referred to as SpecializedAdapter modules.

3. The method according to claim 2, It is characterized in that The step of training the adapter by contrastive learning includes: The SpecializedAdapter model is trained using Chinese text and image datasets from different professional fields through a comparative learning training method.

4. The method according to claim 3, It is characterized in that The method extracts and expands the features of the graphic and text data in different professional fields uploaded by the user according to the trained adapter, and constructs the index structure based on the hierarchical graph by using the feature embedding vectors of the expanded text and image, including: Classify the graphic and text data in different professional fields uploaded by users, and use the multimodal pre-trained model with the Specialized Adapter module of the corresponding field to extract features for different categories; The feature embedding vectors of the text and image obtained after feature extraction are expanded by one dimension to store the file numbers of the modal data corresponding to the feature embedding vectors of the text and image; According to the expanded feature embedding vectors of text and image, the HNSW algorithm is combined to construct a hierarchical graph-based index structure.

5. The method according to claim 4, It is characterized in that The construction of the hierarchical graph-based index structure based on the expanded text and image feature embedding vectors in combination with the HNSW algorithm includes: Create an empty graph structure, treat each data feature embedding as a vector node, and insert nodes one by one. Each new node will be randomly assigned a level. The higher the level, the fewer nodes there are. When inserting a new node, start from the highest level and use greedy search to find the nearest neighbor of the level to which the current node belongs, and repeat this process at lower levels to determine more precise neighbor nodes; After finding the nearest neighbor nodes, the inserted new node is connected to these neighbor nodes and the graph structure is updated.

6. The method according to claim 5, It is characterized in that The step of selecting a search interface corresponding to a professional field according to the query content of the user and selecting a trained adapter in the corresponding field to perform feature extraction on the query content includes: According to the user's choice of professional field retrieval interface, a multimodal pre-trained model with a Specialized Adapter module of the corresponding field is selected to extract features of the user's query content, and a dimension is expanded for the feature embedding vector of the extracted query content to store 0 values.

7. The method according to claim 5, It is characterized in that The method of performing an approximate nearest neighbor search for the extracted query content features in the established index structure to obtain the most relevant query results and return them to the user includes: Insert the feature embedding vector of the extracted query content into the established index structure as a vector node; In the established index structure, we start from the entry point of the highest layer and search downward layer by layer. In each layer, we use greedy search to find the node closest to the query vector node, and use this node as the starting point to continue searching until a closer node is found. If no closer node is found in the current layer, we go to the next layer to search, and repeat this process until we reach the bottom layer. At the bottom layer, the node closest to the queried vector node is searched, and the corresponding file is returned by the file number identified by the last dimension of the vector node.

8. The method according to claim 7, It is characterized in that Also includes: Taking the cosine distance as the distance between vector nodes on the index structure, the calculation expression is: Wherein, Cosine Distance is the cosine distance, e is the feature embedding vector of the user query, e is the feature embedding vector of the data file, and the reciprocal of the cosine distance represents the cosine similarity between the corresponding data.

Images