Image retrieval method and device based on multi-modal mixed index, equipment and storage medium

Abstract

The embodiment of the invention provides an image retrieval method based on a multi-modal mixed index, which comprises the following steps: according to an image description inverted index and an image tag inverted index, recall by utilizing an expanded search term to obtain an image description recall result and an image tag recall result; according to the image content feature index, recall is carried out through the expanded search word multi-modal features, and a multi-modal recall result is obtained; according to the image description feature index, carrying out recall by utilizing the expanded search word text features to obtain a text recall result; extracting reordering features of the search terms to obtain search term reordering features; reordering features of candidate images in the multiple recall results are extracted, and recall result reordering features are obtained; according to the search word reordering features and the recall result reordering features, the multiple recall results are preliminarily ordered to obtain a reordering result, the multiple recall results and the reordering result are combined and ordered again, and an ordered search result is obtained.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image retrieval technology and related technical fields, and in particular, to an image retrieval method, apparatus, device and storage medium based on multimodal hybrid indexing. Background Art

[0002] Image understanding and analysis technology has been extensively explored. With the help of technologies such as target detection and subject recognition, we can deeply understand the connotation of images and accurately analyze their content. However, the current image retrieval technology based on these analysis results still has room for improvement and urgently needs innovation and improvement.

[0003] However, in the existing technology, most solutions focus on multimodal feature alignment, image description and vector retrieval mechanisms, which extract rich multimodal information from images through these strategies, and then support keyword-based image retrieval. This retrieval approach not only greatly alleviates the difficulty of image retrieval, but also inspires the emergence of many innovative applications. However, with the increasing popularity and deepening of these applications, the requirements for the accuracy and quality of image retrieval are also getting higher and higher.

[0004] Therefore, there is an urgent need for an image retrieval method based on multimodal hybrid indexing to support the quality of image retrieval. Summary of the invention

[0005] The embodiments described in this article provide an image retrieval method, apparatus, device and storage medium based on multimodal hybrid indexing to solve the problems existing in the prior art.

[0006] According to a first aspect of the present disclosure, there is provided an image retrieval method based on a multimodal hybrid index, including online retrieval, comprising:

[0007] Classifying and expanding the received search terms to obtain expanded search terms;

[0008] According to the image description inverted index and the image label inverted index, the expanded search term is used to perform recall to obtain an image description recall result and an image label recall result;

[0009] Performing multimodal feature extraction on the expanded search term to obtain multimodal features of the expanded search term;

[0010] Performing text encoding on the expanded search term to obtain text features of the expanded search term;

[0011] According to the image content feature index, the multimodal features of the expanded search term are used for recall to obtain a multimodal recall result;

[0012] According to the image description feature index, the expanded search term text feature is used to perform recall to obtain a text recall result;

[0013] Extracting re-ranking features from the search words to obtain search word re-ranking features;

[0014] Extracting re-ranking features from the candidate images in the multi-channel recall results to obtain recall result re-ranking features, wherein the multi-channel recall results include: the image description recall result, the image label recall result, the text recall result and the multi-modal recall result;

[0015] According to the search term reordering feature and the recall result reordering feature, the multiple recall results are preliminarily sorted to obtain a reordered result, and then the multiple recall results and the reordered result are combined and sorted again to obtain a sorted retrieval result.

[0016] In some embodiments of the present disclosure, before the step of classifying and expanding the received search terms, offline processing is also included, and the offline processing includes:

[0017] Extracting image content features from the image, and writing the image content features into the image content feature index;

[0018] In some embodiments of the present disclosure, before the step of classifying and expanding the received search terms, offline processing is also included, and the offline processing includes:

[0019] generating an image description for the image, and writing the image description into the image description inverse index;

[0020] In some embodiments of the present disclosure, before the step of classifying and expanding the received search terms, offline processing is also included, and the offline processing includes:

[0021] Performing text encoding on the image description to obtain image description features, and writing the image description features into the image description feature index;

[0022] In some embodiments of the present disclosure, before the step of classifying and expanding the received search terms, offline processing is also included, and the offline processing includes:

[0023] Based on the pre-constructed label tree, the image is classified to obtain the image label, and the image label is written into the image description inverted index.

[0024] In some embodiments of the present disclosure, the step of performing recall based on the image description inverted index and the image tag inverted index using the expanded search term to obtain the image description recall result and the image tag recall result includes:

[0025] Searching for a corresponding document list of the expanded search term in the image description inverted index, and sorting or merging the results according to the relevance of the expanded search term, and obtaining the image description inverted index related to the expanded search term as the image description recall result;

[0026] The corresponding document list of the expanded search term in the image tag inverted index is searched, and the results are sorted or merged according to the relevance of the expanded search term, and the image tag inverted index related to the expanded search term is obtained as the image tag recall result.

[0027] In some embodiments of the present disclosure, the step of performing recall based on the image content feature index and utilizing the multimodal features of the expanded search term to obtain a multimodal recall result includes:

[0028] The expanded search term multimodal feature is used as a feature input into a recall module, and similarity matching is performed with the features in the image content feature index to obtain image index information that matches the expanded search term multimodal feature. The image index information that matches the expanded search term multimodal feature is the multimodal recall result.

[0029] In some embodiments of the present disclosure, the step of performing recall based on the image description feature index and utilizing the expanded search term text feature to obtain a text recall result includes:

[0030] The expanded search term text feature is input as a feature into the recall module. The recall module compares the received expanded search term text feature with the features in each image description feature index for matching, and obtains the image description feature index that matches the expanded search term text feature. The image description feature index that matches the expanded search term text feature is the text recall result.

[0031] In some embodiments of the present disclosure, the multimodal features include: text features and image features;

[0032] The text features and the image features after feature extraction are aligned in space.

[0033] According to a second aspect of the present disclosure, there is provided an image retrieval device based on a multimodal hybrid index, comprising an online retrieval module, wherein the online retrieval module comprises:

[0034] A search term processing module, used for classifying and expanding the received search terms to obtain expanded search terms;

[0035] A first recall module is used to perform recall using the expanded search term according to the image description inverted index and the image label inverted index to obtain an image description recall result and an image label recall result;

[0036] A first extraction module is used to extract multimodal features of the expanded search terms to obtain multimodal features of the expanded search terms;

[0037] An encoding module, used for performing text encoding on the expanded search term to obtain text features of the expanded search term;

[0038] A second recall module is used to perform recall based on the image content feature index and the multimodal features of the expanded search term to obtain a multimodal recall result;

[0039] A third recall module is used to perform recall based on the image description feature index and the expanded search term text feature to obtain a text recall result;

[0040] A second extraction module is used to extract re-ranking features from the search words to obtain search word re-ranking features;

[0041] A third extraction module extracts reordering features from the candidate images in the multi-channel recall results to obtain reordering features of the recall results, wherein the multi-channel recall results include: the image description recall results, the image label recall results, the text recall results and the multimodal recall results;

[0042] The sorting module is used to preliminarily sort the multiple recall results according to the search word re-sorting feature and the recall result re-sorting feature to obtain a re-sorting result, and then merge the multiple recall results with the re-sorting result and sort them again to obtain a sorted retrieval result.

[0043] In some embodiments of the present disclosure, it further includes:

[0044] A processing module, used for classifying and expanding the received search terms to obtain expanded search terms;

[0045] In some embodiments of the present disclosure, an offline processing module is further included, and the offline processing module includes:

[0046] A first processing module, configured to extract image content features from an image and write the image content features into the image content feature index;

[0047] A second processing module is used to generate an image description for the image, and write the image description into the image description inverse sorting index;

[0048] A third processing module is used to perform text encoding on the image description to obtain image description features, and write the image description features into the image description feature index;

[0049] The fourth processing module is used to classify the image based on the pre-constructed label tree, obtain the image label, and write the image label into the image description inverse index.

[0050] According to a third aspect of the present disclosure, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the method in any one of the above embodiments are implemented.

[0051] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in any one of the above embodiments are implemented.

[0052] The present disclosure provides an image retrieval method, apparatus, device and storage medium based on multimodal hybrid indexing. The method includes: online retrieval, wherein the online retrieval includes: classifying and expanding the received search terms to obtain expanded search terms; recalling the expanded search terms according to the image description inverted index and the image label inverted index to obtain image description recall results and image label recall results; extracting multimodal features from the expanded search terms to obtain multimodal features of the expanded search terms; text encoding the expanded search terms to obtain text features of the expanded search terms; and extracting the multimodal features from the expanded search terms according to the image content feature index. The multimodal features of the expanded search terms are used for recall to obtain multimodal recall results; according to the image description feature index, the text features of the expanded search terms are used for recall to obtain text recall results; the reordering features of the search terms are extracted to obtain the search term reordering features; the reordering features of the candidate images in the multi-way recall results are extracted to obtain the recall result reordering features; according to the search term reordering features and the recall result reordering features, the multi-way recall results are preliminarily sorted to obtain the reordering results, and then the multi-way recall results and the reordering results are merged and sorted again to obtain the sorted retrieval results. This application optimizes the indexing strategy, efficiently integrates multimodal information, significantly improves the image retrieval quality, and meets high standards. Construct a hybrid index, integrate multi-dimensional features, flexibly support various retrieval methods and keyword input, and adapt to different user scenarios. Adopt multi-stage index generation, instant construction to improve short-term retrieval speed, and continuous optimization to ensure long-term effect and stability. Support intelligent sorting, efficiently process multi-dimensional data, significantly improve retrieval efficiency, and solve the retrieval speed bottleneck under multi-dimensional features.

[0053] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the embodiment of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly described below. It should be noted that the drawings described below only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure, wherein:

[0055] Figure 1 It is a flowchart of an image retrieval method based on multimodal hybrid indexing provided by an embodiment of the present disclosure;

[0056] Figure 2 It is a schematic diagram of the overall process of an image retrieval method based on multimodal hybrid indexing provided by an embodiment of the present disclosure;

[0057] Figure 3 is a structural schematic diagram of an image retrieval device based on multimodal hybrid indexing provided by an embodiment of the present disclosure;

[0058] Figure 4 It is a structural diagram of a computer device provided in an embodiment of the present disclosure.

[0059] In the drawings, reference numerals with the same last two digits correspond to the same elements. It should be noted that the elements in the drawings are schematic and not drawn to scale. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work also fall within the scope of protection of the present disclosure.

[0061] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase "embodiments" in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0062] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists, A and B exist at the same time, and B exists. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0063] Furthermore, in all embodiments of the present disclosure, terms such as “first” and “second” are used only to distinguish one component (or a part of a component) from another component (or another part of a component).

[0064] In the description of the present application, unless otherwise specified, "plurality" means more than two (including two), and similarly, "multiple groups" means more than two groups (including two).

[0065] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0066] Based on the problems existing in the existing technology, Figure 1 is a flowchart of an image retrieval method based on multimodal hybrid indexing provided by an embodiment of the present disclosure, such as Figure 1 As shown, the image retrieval method based on multimodal hybrid indexing includes an online retrieval process, which is a process of converting the search terms input by the user into ranked candidate images. The online retrieval process includes the following steps:

[0067] S110: Classify and expand the received search terms to obtain expanded search terms.

[0068] In a specific implementation process, the received search terms are segmented using a search term module, stop words are removed, and word meanings are expanded to obtain expanded search terms;

[0069] Specifically, the search term processing module is used to classify and expand the meaning of search terms.

[0070] Specifically, the search words are keywords or phrases for which the user wishes to obtain information, and they may be single words or sentences or phrases composed of multiple words.

[0071] As a specific implementable manner, the received search terms are classified using word segmentation technology so as to split them into individual words. Word segmentation technology is a process of recombining a continuous sequence of characters into a sequence of words according to certain specifications. Stop words are words that frequently appear in the text but contribute little to the meaning of the text and may even interfere with text processing, such as "de" (的), "le" (了), etc., and are usually not considered during the search and analysis process. Removing stop words can improve the efficiency and accuracy of subsequent processing steps. A predetermined stop word list can be used to compare with the word segmentation results and remove the matching stop words from the text. According to the specific content and context of the text, words that contribute little to the meaning of the text are dynamically identified and removed.

[0072] As a specific implementable manner, some word sense expansion processing techniques are used, such as using synonyms, related words, or word roots to expand the search terms to obtain a wider range of search results. Word sense expansion processing is a process of expanding the search scope by adding synonyms, near synonyms, related words, or phrases, etc. while retaining the core meaning of the original search terms, which helps to improve the accuracy and coverage of the search. A synonym dictionary or database can be utilized to replace the words in the search terms with synonyms or near synonyms. Based on semantic analysis technology, the deep meaning of the search terms is understood, and accordingly, related words or phrases are generated. Considering the context environment of the search terms, related background information or context words are generated. Word embeddings are used to map words to vectors in a high-dimensional space, and by calculating the similarity between vectors, words or phrases similar to the search terms are found. Thus, an expanded search term list or set is obtained as the expanded search terms. This list contains the search terms and their synonyms, near synonyms, related words, etc., which can greatly improve the breadth and depth of the search and help users find the required information faster.

[0073] In some embodiments, before step S110, step S100 is further included, which is an offline processing process. Offline processing is a process of converting an image into a multi-modal hybrid index.

[0074] Step S100 includes:

[0075] Step S1001: A multi-modal feature extractor extracts image content features from the image;

[0076] The image content features are written into an image content feature index.

[0077] As a specific implementation method, first, a multimodal feature extractor is used to extract multimodal features from images, which includes using computer vision technology to extract visual features, as well as possible other modal features, such as text descriptions, label information, etc., to obtain comprehensive image content features. Then, based on the image content features, an image feature index is constructed, which will help to efficiently organize and manage large-scale image data, and support applications such as image retrieval and similarity matching.

[0078] Specifically, the multimodal feature extractor is a neural network for extracting features, which can extract features of text and images at the same time and ensure that features of different modules are aligned in space. The multimodal features extracted from the image are aligned in space with the multimodal features of the expanded search terms.

[0079] Specifically, the image content feature is the output result of the image input to the multimodal feature extractor.

[0080] Step S100 includes:

[0081] Step S1002: The image describer generates an image description for the image;

[0082] The image description is written into the image description inverse index.

[0083] As a specific implementation method, first, for extracting image descriptions from images, an image descriptor can be used to extract text description information from the image and then convert it into description features. Then, based on the description features, an image description inverse index can be constructed. Such an index structure can quickly locate and retrieve images matching a specific description based on the description features.

[0084] Specifically, the image descriptor is a neural network for image description, which can convert an image into a text describing the image content. Image description refers to extracting meaningful description information from an image using the image descriptor and converting it into an understandable and operable format.

[0085] Specifically, the image description is the output of the image describer, that is, a text describing the image content.

[0086] Specifically, the index is a data structure for quickly retrieving a large amount of data.

[0087] Step S100 includes:

[0088] Step S1003, performing text encoding on the image description obtained in step S1002 through a text encoder to obtain image description features;

[0089] The image description feature is written into the image description feature index.

[0090] As a specific implementation method, firstly, for the description features, text encoding technology can be applied, such as using word embedding or other text representation methods to encode the image description, so as to obtain image description features. Then, based on the image description features, an image description feature index can be constructed. This can improve the efficiency of processing and utilizing image semantic information.

[0091] Specifically, the image description feature is the output result of the image description input to the text encoder.

[0092] Specifically, a text encoder is a feature extraction neural network used to extract text features. Text encoding is the process of converting natural language text into a format that can be directly processed or stored by a computer. In the context of image description features, it refers to converting the image description text into a form suitable for subsequent processing and indexing. For the image description text, a suitable text encoding method can be selected to convert it into an image description feature in the form of a vector or matrix.

[0093] Specifically, constructing an image description feature index means using a text encoder to organize the image description features according to certain rules so as to quickly retrieve images related to a given query. Specifically, the extracted image description features are standardized, a suitable index structure is selected to store the image description features, and the image description features are organized into index files according to the requirements of the index structure. For the inverted index, an inverted list is constructed for each feature item to record the identifiers of all images containing the feature.

[0094] Step S100 includes:

[0095] Step S1004: The image classifier classifies the image based on the pre-built label tree to obtain an image label;

[0096] Write the image tag into the image description inverse index.

[0097] As a specific implementation method, first use an image classifier to automatically classify the input image, extract useful features from the image, and divide the image into different categories based on these features. Then map the image recognition results to a pre-built label tree, and for each identified image category or attribute, find the corresponding node in the label tree and use it as the label of the image. If the image has multiple categories or attributes, multiple labels can be assigned to it, and the relevant nodes of these labels can be found in the label tree.

[0098] Specifically, the label tree is a hierarchical data structure used to represent and organize the hierarchical relationship between different labels. In the context of image classification, the label tree can describe and classify images according to the category, attributes, and features of the image. The label tree is an established image label system that includes a first-level classification, each of which contains several second-level classifications. Each classification name is a simple noun that describes the content or style of the image. There is only one label tree, and the image classifier matches the image with the labels in the label tree. The image description inverted index is which images a label in the label tree corresponds to.

[0099] Specifically, the image classifier is a neural network for image classification, and can classify images according to a given classification name.

[0100] Specifically, the inverted index is a commonly used data structure in information retrieval systems for quickly querying images containing image features. In the context of image description, the inverted index can regard images as documents, image tags as terms, and build an inverted index based on these tags. The image tags are standardized, all unique image tags are collected, and an index dictionary is built, and each tag is assigned a unique identifier. The inverted index uses a dictionary to record the list of documents corresponding to each word. In the current system, an image and related description information can be understood as a document. For each tag in the index dictionary, an inverted list is generated, which contains the identifiers of all images containing the tag, as well as the number of times or weights of the tag appear in the image. It stores a mapping table of all words in the document, indicating in what document each word appears.

[0101] Specifically, the image label is the output of the image classifier, that is, the label in the label tree corresponding to the image.

[0102] Step S120: According to the image description inverted index and the image label inverted index, the expanded search term is used to perform recall to obtain image description recall results and image label recall results.

[0103] As a specific implementation method, the expanded search terms are retrieved through the image description inverted index through the recall module. Specifically, the corresponding document list of each expanded search term in the image description inverted index is searched, and the results are sorted or merged according to the relevance of the expanded search terms. The image description inverted index related to the expanded search terms is obtained as the image description recall result. Since the expanded search terms may match different keywords in the document, relevant documents can be recalled more comprehensively.

[0104] As a specific implementation method, the expanded search terms are retrieved through the image tag inverted index through the recall module. Specifically, the corresponding document list of each expanded search term in the image tag inverted index is searched, and the results are sorted or merged according to the relevance of the expanded search terms. The image tag inverted index related to the expanded search terms is obtained as the image tag recall result. Since the expanded search terms may match different keywords in the document, relevant documents can be recalled more comprehensively.

[0105] Specifically, the recall module is used to find the most relevant data in the index based on the input features and return the result.

[0106] Step S130: extract multimodal features of the expanded search terms to obtain multimodal features of the expanded search terms.

[0107] Multimodal feature extraction is performed on the expanded search term. The feature extraction is a process of comprehensively capturing and parsing the search term in different dimensions. This process spans multiple modalities, including text and images. The multimodal features include text features and image features, and the extracted text features and image features are aligned in space.

[0108] Specifically, for expanding search terms, that is, expanding the search term list, text analysis is first performed, including but not limited to lexical analysis (such as vocabulary tagging and stem extraction), syntactic analysis (understanding the structural relationship between words), and semantic analysis (understanding the meaning and context behind the words). Not only can text information be used, but also information from other modalities, such as images, can be combined to enrich the features of the expanded search terms. Through multimodal feature extraction, a more comprehensive and richer search term feature representation can be obtained, which helps to improve the accuracy and relevance of search results.

[0109] Step S140: Perform text encoding on the expanded search term to obtain text features of the expanded search term.

[0110] As a specific feasible implementation method, firstly, for the expanded search terms, text encoding technology can be used, such as using word embedding or other text representation methods to encode the expanded search terms, thereby obtaining the expanded search term text features. Such text features can help better characterize the semantic information of the expanded search terms in multimodal information retrieval. By using the expanded search term text features, text and image information can be better combined in search and relevance matching tasks, thereby improving the accuracy and diversity of search results.

[0111] Specifically, the expanded search term is converted into a text feature, which can be achieved through text encoding technology, including but not limited to a bag-of-words model, TF-IDF, word embedding, and a deep learning model (such as a convolutional neural network CNN, a recurrent neural network RNN, or a Transformer or other deep learning model to encode text). The text of the expanded search term is encoded to obtain the expanded search term text feature.

[0112] Step S150: According to the image content feature index, the multimodal features of the expanded search term are used to perform recall to obtain a multimodal recall result.

[0113] As a specific implementation method, the expanded search term multimodal feature is used as a feature input into a recall module, and similarity matching is performed with the features in the image content feature index to obtain image index information that matches the expanded search term multimodal feature. The image index information that matches the expanded search term multimodal feature is the multimodal recall result.

[0114] Specifically, recall refers to retrieving data items or information that match specific conditions or standards from a data set. Specifically, the goal of recall is to find as many data items as possible that meet the query conditions. In information retrieval, recall is usually performed by using indexes, inverted indexes or other related technologies to quickly retrieve large-scale document collections and return documents or data items that match the query conditions. Once a recall is performed, the returned data can be sorted and filtered by ranking or other methods so that it can be finally presented to the user or used for further analysis and processing.

[0115] In a specific implementation, the purpose of recall is to retrieve relevant images that match the query and the multimodal features of the expanded search term from the image content feature index library.

[0116] Step S160: According to the image description feature index, the expanded search term text feature is used to perform recall to obtain a text recall result.

[0117] As a specific feasible implementation method, the expanded search term text feature is input as a feature into a recall module. The recall module compares the received expanded search term text feature with the features in each image description feature index for matching degree to obtain an image description feature index that matches the expanded search term text feature. The image description feature index that matches the expanded search term text feature is the text recall result.

[0118] Specifically, the expanded search term text features are used as feature inputs to the recall module, which can improve the accuracy of image search by expanding the current search term text features. The text features of the search terms can be increased, such as synonyms, related words, and words related to the image description, thereby expanding the coverage of the search terms and improving the recall rate of the search results.

[0119] Specifically, the recall module finds the most relevant data in the index based on the input features and returns the results.

[0120] By expanding the text features of search terms and utilizing image description feature indexes, image search can be made more accurate and efficient.

[0121] Step S170, extracting re-ranking features of the search words to obtain search word re-ranking features;

[0122] As a specific feasible implementation method, firstly, for the search terms, re-ranking features can be extracted through different methods, such as: relevance score of the search terms, frequency of occurrence of the search terms, semantic similarity of the search terms, etc. These features can be used to evaluate the matching degree and relevance between the search terms and the recall results. By extracting the re-ranking features of the search terms, the search process can be better modeled, thereby improving the ranking quality and relevance of the search results.

[0123] By extracting the re-ranking features of search terms, we can perform more detailed analysis and understanding of the search terms during the search result ranking stage, providing users with a ranking effect that better meets their expectations.

[0124] Step S180, extracting re-ranking features from the candidate images in the multi-channel recall results to obtain recall result re-ranking features;

[0125] Wherein, the multi-channel recall result includes: the image description recall result, the image label recall result, the text recall result and the multimodal recall result;

[0126] As a specific feasible implementation method, firstly, the image description recall results, image label recall results, text recall results and multimodal recall results can be sorted using a re-ranking feature extractor. Re-ranking features are first extracted for the image description recall results, image label recall results, text recall results and multimodal recall results. These re-ranking features can cover aspects such as the similarity between images and texts and the degree of association of multimodal information, which are helpful for comprehensive evaluation and sorting of recall results. By extracting the recall result re-ranking features, the sorting quality of search results can be improved, thereby better meeting the information needs of users.

[0127] Specifically, the re-ranking feature extractor is a neural network for extracting features. Compared with the above-mentioned neural network structure, it is more complex and cannot process large amounts of data. It is used to extract fine features of small batches of data, and sorting can be performed based on the similarity between the features.

[0128] By extracting the re-ranking features of the recall results, the search results can be more finely ranked in combination with the ranking algorithm to improve the relevance of the search results.

[0129] Step S190: preliminarily sorting the multiple recall results according to the search word re-ranking feature and the recall result re-ranking feature to obtain a re-ranking result, and then merging the multiple recall results with the re-ranking result and sorting them again to obtain a sorted retrieval result;

[0130] The various recall results include: the image description recall result, the image tag recall result, the text recall result and the multimodal recall result.

[0131] As a specific feasible implementation method, the sorting module first preliminarily sorts the image description recall results, the image label recall results, the text recall results and the multimodal recall results according to the recall result re-ranking features and the search term re-ranking features to obtain the re-ranking results; then the image description recall results, the image label recall results, the text recall results and the multimodal recall results are merged with the re-ranking results and sorted again, and a weighted RRF algorithm can be used for multi-way re-ranking to obtain the sorted retrieval results.

[0132] The sorted search results are returned as search results.

[0133] Among them, the weighted RRF algorithm process is as follows:

[0134] For an element d in an ordered list (each recall result is an ordered list) L, its score RRF(d) is defined as:

[0135]

[0136] Where N is the total length of list L, rank i (d) is the order of the elements in the list, k L is the sorting weight of the ordered list L, and i represents the i-th ordered list.

[0137] Calculate the score of each element in each recall result, and finally sort them by score to get the sorting result.

[0138] Specifically, the sorting module is used to perform multi-way re-sorting on a plurality of unrelated, ordered recall results. In particular, if a re-sorting feature is input, the final re-sorting will be performed mainly based on the re-sorting feature.

[0139] The image retrieval method based on multimodal hybrid indexing provided by the embodiment of the present disclosure includes online retrieval, which includes: classifying and expanding the received search terms to obtain expanded search terms; recalling the expanded search terms according to the image description inverted index and the image label inverted index to obtain image description recall results and image label recall results; extracting multimodal features of the expanded search terms to obtain multimodal features of the expanded search terms; text encoding the expanded search terms to obtain text features of the expanded search terms; and extracting multimodal features of the expanded search terms according to the image content feature index using the expanded search terms. The multi-modal features are used for recall to obtain multi-modal recall results; according to the image description feature index, the expanded search term text features are used for recall to obtain text recall results; the search term is extracted for re-ranking features to obtain the search term re-ranking features; the re-ranking features are extracted for the candidate images in the multi-way recall results to obtain the recall result re-ranking features; according to the search term re-ranking features and the recall result re-ranking features, the multi-way recall results are preliminarily sorted to obtain the re-ranking results, and then the multi-way recall results are merged with the re-ranking results and sorted again to obtain the sorted retrieval results. This application optimizes the indexing strategy, efficiently integrates multi-modal information, significantly improves the image retrieval quality, and meets high standards. Construct a hybrid index, integrate multi-dimensional features, flexibly support various retrieval methods and keyword input, and adapt to different user scenarios. Adopt multi-stage index generation, instant construction to improve short-term retrieval speed, and continuous optimization to ensure long-term effect and stability. Support intelligent sorting, efficiently process multi-dimensional data, significantly improve retrieval efficiency, and solve the retrieval speed bottleneck under multi-dimensional features.

[0140] The overall process of the multimodal mixed index image detection method provided in the embodiment of the present disclosure can be referred to Figure 2 shown.

[0141] Based on the above embodiments, the present disclosure also provides an image retrieval device based on multi-modal hybrid indexing. Figure 3 As shown, the image retrieval device based on multimodal hybrid indexing includes an online retrieval module 300, and the online retrieval module 300 includes:

[0142] A search term processing module 301 is used to classify and expand the received search terms to obtain expanded search terms;

[0143] A first recall module 302 is used to perform recall based on the image description inverted index and the image label inverted index using the expanded search term to obtain an image description recall result and an image label recall result;

[0144] A first extraction module 303 is used to extract multimodal features of the expanded search term to obtain multimodal features of the expanded search term;

[0145] An encoding module 304 is used to perform text encoding on the expanded search term to obtain text features of the expanded search term;

[0146] The second recall module 305 is used to perform recall based on the image content feature index and the multimodal feature of the expanded search term to obtain a multimodal recall result;

[0147] A third recall module 306 is used to perform recall based on the image description feature index and the expanded search term text feature to obtain a text recall result;

[0148] The second extraction module 307 is used to extract the re-ranking feature of the search word to obtain the search word re-ranking feature;

[0149] The third extraction module 308 extracts reordering features from the candidate images in the multi-channel recall results to obtain reordering features of the recall results, wherein the multi-channel recall results include: the image description recall results, the image label recall results, the text recall results and the multimodal recall results;

[0150] The sorting module 309 is used to preliminarily sort the multiple recall results according to the search word re-sorting feature and the recall result re-sorting feature to obtain a re-sorting result, and then merge the multiple recall results with the re-sorting result and sort them again to obtain a sorted retrieval result.

[0151] In a specific implementation, an offline processing module 500 is also included, and the offline processing module 500 specifically includes:

[0152] The first processing module 501 is used to extract image content features from an image and write the image content features into the image content feature index;

[0153] A second processing module 502 is used to generate an image description for the image, and write the image description into the image description reverse order index;

[0154] The third processing module 503 is used to perform text encoding on the image description to obtain image description features, and write the image description features into the image description feature index;

[0155] The fourth processing module 504 is used to classify the image based on the pre-constructed label tree, obtain the image label, and write the image label into the image description inverse index.

[0156] In a specific implementation, the first recall module 302 is specifically used to:

[0157] Searching for a corresponding document list of the expanded search term in the image description inverted index, and sorting or merging the results according to the relevance of the expanded search term, and obtaining the image description inverted index related to the expanded search term as the image description recall result;

[0158] The corresponding document list of the expanded search term in the image tag inverted index is searched, and the results are sorted or merged according to the relevance of the expanded search term, and the image tag inverted index related to the expanded search term is obtained as the image tag recall result.

[0159] In a specific implementation, the second recall module 305 is specifically used to:

[0160] The expanded search term multimodal feature is used as a feature input into a recall module, and similarity matching is performed with the features in the image content feature index to obtain image index information that matches the expanded search term multimodal feature. The image index information that matches the expanded search term multimodal feature is the multimodal recall result.

[0161] In a specific implementation, the third recall module 306 is used to

[0162] The expanded search term text feature is input as a feature into the recall module. The recall module compares the received expanded search term text feature with the features in each image description feature index for matching, and obtains the image description feature index that matches the expanded search term text feature. The image description feature index that matches the expanded search term text feature is the text recall result.

[0163] In a specific implementation, the multimodal features include: text features and image features;

[0164] The text features and the image features after feature extraction are aligned in space.

[0165] The image retrieval device based on multimodal hybrid indexing provided by the disclosed embodiment optimizes the indexing strategy and efficiently integrates multimodal information, significantly improving the image retrieval quality and meeting high standards. Build a hybrid index, integrate multi-dimensional features, flexibly support various retrieval methods and keyword input, and adapt to different user scenarios. Adopt multi-stage index generation, instant construction to improve short-term retrieval speed, and continuous optimization to ensure long-term effect and stability. Support intelligent sorting, efficiently process multi-dimensional data, significantly improve retrieval efficiency, and solve the retrieval speed bottleneck under multi-dimensional features.

[0166] The present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0167] The computer device includes a memory 410 and a processor 420 that are connected to each other through a system bus. It should be noted that the figure only shows a computer device with components 410-420, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0168] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0169] The memory 410 includes at least one type of readable storage medium, and the readable storage medium includes a non-volatile memory or a volatile memory, for example, a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc., and the RAM may include a static RAM or a dynamic RAM. In some embodiments, the memory 410 may be an internal storage unit of a computer device, for example, a hard disk or a memory of the computer device. In other embodiments, the memory 310 may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card or a flash card (FlashCard) equipped on the computer device. Of course, the memory 410 may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory 410 is generally used to store the operating system and various application software installed on the computer device, such as the program code of the above method. In addition, the memory 410 may also be used to temporarily store various data that have been output or are to be output.

[0170] The processor 420 is generally used to perform the overall operation of the computer device. In this embodiment, the memory 410 is used to store program codes or instructions, the program code includes computer operation instructions, and the processor 420 is used to execute the program codes or instructions stored in the memory 410 or process data, such as running the program code of the above method.

[0171] In this article, the bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus system can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0172] Another embodiment of the present application also provides a computer-readable medium, which may be a computer-readable signal medium or a computer-readable medium. A processor in a computer reads a computer-readable program code stored in the computer-readable medium, so that the processor can execute the functional actions specified in each step or a combination of steps in the above method; and generate a device for implementing the functional actions specified in each block or a combination of blocks in the block diagram.

[0173] Computer-readable media include but are not limited to electronic, magnetic, optical, electromagnetic, infrared memory or semiconductor systems, devices or apparatuses, or any appropriate combination of the foregoing, the memory is used to store program codes or instructions, the program codes include computer operating instructions, and the processor is used to execute the program codes or instructions of the above methods stored in the memory.

[0174] For the definitions of memory and processor, please refer to the description of the aforementioned computer device embodiment and will not be repeated here.

[0175] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0176] Each functional unit or module in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0177] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

[0178] Unless the context clearly indicates otherwise, the singular form of the words used herein and in the appended claims includes the plural and vice versa. Thus, when referring to the singular, the plural form of the corresponding term is generally included. Similarly, the words "comprise" and "include" are to be interpreted as inclusive rather than exclusive. Likewise, the terms "include" and "or" should be interpreted as inclusive unless such interpretation is expressly prohibited herein. Where the term "example" is used herein, particularly when it is located after a group of terms, the "example" is merely exemplary and illustrative and should not be considered exclusive or comprehensive.

[0179] Further aspects and scopes of adaptability become apparent from the description provided herein. It should be understood that various aspects of the present application can be implemented individually or in combination with one or more other aspects. It should also be understood that the description and specific embodiments herein are intended for purposes of illustration only and are not intended to limit the scope of the present application.

[0180] Several embodiments of the present disclosure are described in detail above, but it is obvious that those skilled in the art can make various modifications and variations to the embodiments of the present disclosure without departing from the spirit and scope of the present disclosure. The protection scope of the present disclosure is defined by the attached claims.

Claims

1. An image retrieval method based on multimodal hybrid indexing, characterized in that: Includes online search, including: Classifying and expanding the received search terms to obtain expanded search terms; According to the image description inverted index and the image label inverted index, the expanded search term is used to perform recall to obtain an image description recall result and an image label recall result; Performing multimodal feature extraction on the expanded search term to obtain multimodal features of the expanded search term; Performing text encoding on the expanded search term to obtain text features of the expanded search term; According to the image content feature index, the multimodal features of the expanded search term are used for recall to obtain a multimodal recall result; According to the image description feature index, the expanded search term text feature is used to perform recall to obtain a text recall result; Extracting re-ranking features from the search words to obtain search word re-ranking features; Extracting re-ranking features from the candidate images in the multi-channel recall results to obtain recall result re-ranking features, wherein the multi-channel recall results include: the image description recall result, the image label recall result, the text recall result and the multi-modal recall result; According to the search term reordering feature and the recall result reordering feature, the multiple recall results are preliminarily sorted to obtain a reordered result, and then the multiple recall results and the reordered result are combined and sorted again to obtain a sorted retrieval result.

2. The method according to claim 1, characterized in that Before the step of classifying and expanding the received search terms, offline processing is also included, and the offline processing includes: Extracting image content features from the image, and writing the image content features into the image content feature index; generating an image description for the image, and writing the image description into the image description inverse index; Performing text encoding on the image description to obtain image description features, and writing the image description features into the image description feature index; Based on the pre-constructed label tree, the image is classified to obtain the image label, and the image label is written into the image description inverted index.

3. The method according to claim 1, characterized in that: The step of performing recall based on the image description inverted index and the image label inverted index using the expanded search term to obtain the image description recall result and the image label recall result comprises: Searching for a corresponding document list of the expanded search term in the image description inverted index, and sorting or merging the results according to the relevance of the expanded search term, and obtaining the image description inverted index related to the expanded search term as the image description recall result; The corresponding document list of the expanded search term in the image tag inverted index is searched, and the results are sorted or merged according to the relevance of the expanded search term, and the image tag inverted index related to the expanded search term is obtained as the image tag recall result.

4. The method according to claim 1, characterized in that: The step of performing recall based on the image content feature index and utilizing the multimodal features of the expanded search term to obtain a multimodal recall result comprises: The expanded search term multimodal feature is used as a feature input into a recall module, and similarity matching is performed with the features in the image content feature index to obtain image index information that matches the expanded search term multimodal feature. The image index information that matches the expanded search term multimodal feature is the multimodal recall result.

5. The method according to claim 1, characterized in that The step of performing recall based on the image description feature index and utilizing the expanded search term text feature to obtain a text recall result comprises: The expanded search term text feature is input as a feature into the recall module. The recall module compares the received expanded search term text feature with the features in each image description feature index for matching, and obtains the image description feature index that matches the expanded search term text feature. The image description feature index that matches the expanded search term text feature is the text recall result.

6. The method according to claim 1, characterized in that The multimodal features include: text features and image features; The text features and the image features after feature extraction are aligned in space.

7. An image retrieval device based on multimodal hybrid indexing, characterized in that: An online search module is included, and the online search module includes: A search term processing module, used for classifying and expanding the received search terms to obtain expanded search terms; A first recall module is used to perform recall using the expanded search term according to the image description inverted index and the image label inverted index to obtain an image description recall result and an image label recall result; A first extraction module, configured to extract multimodal features of the expanded search term to obtain multimodal features of the expanded search term; An encoding module, used for performing text encoding on the expanded search term to obtain text features of the expanded search term; A second recall module is used to perform recall based on the image content feature index and the multimodal features of the expanded search term to obtain a multimodal recall result; A third recall module is used to perform recall based on the image description feature index and the expanded search term text feature to obtain a text recall result; A second extraction module is used to extract re-ranking features from the search words to obtain search word re-ranking features; A third extraction module extracts reordering features from the candidate images in the multi-channel recall results to obtain reordering features of the recall results, wherein the multi-channel recall results include: the image description recall results, the image label recall results, the text recall results and the multimodal recall results; The sorting module is used to preliminarily sort the multiple recall results according to the search word re-sorting feature and the recall result re-sorting feature to obtain a re-sorting result, and then merge the multiple recall results with the re-sorting result and sort them again to obtain a sorted retrieval result.

8. The device according to claim 7, characterized in that It also includes an offline processing module, which includes: A first processing module, configured to extract image content features from an image and write the image content features into the image content feature index; A second processing module is used to generate an image description for the image, and write the image description into the image description inverse sorting index; A third processing module is used to perform text encoding on the image description to obtain image description features, and write the image description features into the image description feature index; The fourth processing module is used to classify the image based on the pre-constructed label tree, obtain the image label, and write the image label into the image description inverse index.

9. A computer device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images