Image retrieval method and device, equipment and storage medium

Abstract

The invention discloses an image retrieval method and device, equipment and a storage medium. The method comprises the following steps: receiving image text description information for image retrieval; respectively determining a description feature vector of the image text description information, a candidate image feature vector of a candidate image in an image retrieval library and a character content feature vector of character extraction content corresponding to the candidate image; and according to the description feature vector, the candidate image feature vector and the text content feature vector, determining an image retrieval result meeting a retrieval condition from an image retrieval library. By utilizing the method, the image retrieval result meeting the retrieval condition is determined from the image retrieval library according to the description feature vector, the candidate image feature vector and the text content feature vector, multi-modal retrieval is realized, and the content of the retrieved image is wider by adding the text description information and the text extraction content, so that the retrieval efficiency is improved. The image retrieval results are more diversified, and the correlation and accuracy of image retrieval are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image retrieval method, device, equipment and storage medium. Background Art

[0002] Content-based image retrieval has been widely used in daily life. For example, you can retrieve pictures containing specific objects by inputting content description information, such as a red car in the picture.

[0003] Currently, content-based image retrieval methods only use the query image as a retrieval query condition to retrieve the images most similar to the query image from the massive image library.

[0004] This image retrieval method can only retrieve images that are visually similar to the query image, resulting in serious missed detections and limited application scope. Summary of the invention

[0005] The present invention provides an image retrieval method, device, equipment and storage medium to solve the limitation of using only a query image as a carrier to retrieve a required image.

[0006] In a first aspect, an embodiment of the present invention provides an image retrieval method, the method comprising:

[0007] receiving image text description information for image retrieval;

[0008] Determine the description feature vector of the image text description information, and respectively determine the candidate image feature vectors of the candidate images in the image retrieval library and the text content feature vectors of the text extraction content corresponding to the candidate images;

[0009] According to the description feature vector, the candidate image feature vector and the text content feature vector, an image retrieval result satisfying the retrieval condition is determined from the image retrieval library.

[0010] In a second aspect, an embodiment of the present invention provides an image retrieval device, the device comprising:

[0011] An information receiving module, used for receiving image text description information for image retrieval;

[0012] A feature determination module, used to determine the description feature vector of the image text description information, and respectively determine the candidate image feature vector of the candidate image in the image retrieval library and determine the text content feature vector of the text extraction content corresponding to the candidate image;

[0013] The result retrieval module is used to determine the image retrieval results that meet the retrieval conditions from the image retrieval library according to the description feature vector, the candidate image feature vector and the text content feature vector.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0015] at least one processor;

[0016] and a memory communicatively coupled to the at least one processor;

[0017] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image retrieval method described in any embodiment of the present invention.

[0018] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image retrieval method described in any embodiment of the present invention when executed.

[0019] The technical solution of the embodiment of the present invention receives image text description information for image retrieval; determines the description feature vector of the image text description information, and respectively determines the candidate image feature vectors of the candidate images in the image retrieval library and determines the text content feature vectors of the text extraction content corresponding to the candidate images; and determines the image retrieval results that meet the retrieval conditions from the image retrieval library according to the description feature vector, the candidate image feature vector and the text content feature vector. The technical solution of the present invention implements multimodal retrieval by determining the image retrieval results that meet the retrieval conditions from the image retrieval library according to the description feature vector, the candidate image feature vector and the text content feature vector, and by adding text description information and text extraction content, the retrieved image content is more extensive, the image retrieval results are more diverse, and the relevance and accuracy of the image retrieval are improved.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flowchart of an image retrieval method provided by an embodiment of the present invention;

[0023] Figure 2 A logical implementation diagram of image retrieval applicable to an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of the structure of an image retrieval device provided by an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of the structure of an electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] It is understandable that before using the technical solutions disclosed in the embodiments of the present invention, the type, scope of use, usage scenarios, etc. of the personal information involved in the present invention should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0029] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present invention according to the prompt message.

[0030] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0031] It is understandable that the above notification and user authorization process is only illustrative and does not constitute a limitation on the implementation of the present invention. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present invention.

[0032] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0033] It should be noted that only using the query image as the retrieval query condition to search for the image most similar to the query image in the massive image library can only retrieve images similar to the query image, resulting in serious missed detection and limited application scope.

[0034] Based on this, an embodiment of the present invention provides an image retrieval method. Figure 1 A flowchart of an image retrieval method provided in an embodiment of the present invention. The embodiment of the present invention is applicable to the case of retrieving images corresponding to text descriptions. The method can be executed by an image retrieval device, which can be implemented in the form of software and / or hardware, and optionally, by an electronic device, which is preferably a mobile terminal, a desktop computer, a laptop computer, a server, etc.

[0035] like Figure 1 As shown, the image retrieval method provided by the embodiment of the present invention may specifically include:

[0036] S101: Receive image text description information for image retrieval.

[0037] The image text description information can be considered as text information describing the image content, which can be a simple description or a detailed description. The image text description information can include information such as objects, scenes, and actions appearing in the image, and can also include the relationship between objects, scenes, and actions, and can also include description information such as appearance, color, and function, and the present invention is not limited here.

[0038] For example, the image text description information may be "The Oriental Pearl Tower in Shanghai stands on the bank of the Huangpu River in the picture and is a landmark building in Shanghai. It was designed and built by Shanghai Radio and Television Station and was completed and put into operation in 1994. The entire tower is 468 meters high and has an observation hall and a revolving restaurant. Visitors can take the high-speed elevator to the top of the tower and overlook the entire Shanghai cityscape. During the Spring Festival, the Oriental Pearl Tower will also light up gorgeous fireworks, which is very beautiful."

[0039] Exemplarily, the image text description information may be input through a client, or may be generated by automatically analyzing the image using computer vision technology.

[0040] In this embodiment, image text description information about the image content is received, and matching images can be found in a large number of images according to the image text description information.

[0041] S102, determining description feature vectors of image text description information, and respectively determining candidate image feature vectors of candidate images in an image retrieval library and text content feature vectors of text extraction content corresponding to the candidate images.

[0042] Among them, the description feature vector can be considered as a vector composed of the features of the image text description information; the image retrieval library can be considered as a database storing a large amount of image data to be retrieved; the candidate image can be understood as an image in the image retrieval library to be retrieved to see if it is similar to the image text description information; the candidate image feature vector can be considered as a vector composed of the features of the candidate image; the text extraction content can be understood as the text information extracted from the candidate image, which can be extracted using text extraction tools such as PaddleOCR, UmiOCR, Catocr, etc.; the text content feature vector can be understood as a vector composed of the features of the text extraction content.

[0043] In this embodiment, text in the candidate image is extracted by text extraction technology to obtain text extraction content, and then the description feature vector in the received image text description information, the candidate image feature vector of the candidate image in the image retrieval library, and the text content feature vector of the text extraction content can be determined respectively through large language models, deep learning architectures, and machine learning methods.

[0044] S103: Determine image retrieval results that meet the retrieval conditions from the image retrieval library according to the description feature vector, the candidate image feature vector and the text content feature vector.

[0045] The image retrieval result can be considered as an image that meets the retrieval conditions, which can be one image or multiple images.

[0046] In this embodiment, the similarity between the description feature vector and the candidate image feature vector, as well as the similarity between the description feature vector and the text content feature vector can be calculated, and the two calculated similarity values ​​can be fused, such as adding or multiplying, to determine the image retrieval results that meet the retrieval conditions in the image retrieval library, such as the ten images with the highest fusion similarity values.

[0047] Exemplarily, the similarity calculation method may adopt cosine similarity, Euclidean distance, Manhattan distance, etc.

[0048] An image retrieval method provided by an embodiment of the present invention receives image text description information for image retrieval; determines a description feature vector of the image text description information, and respectively determines a candidate image feature vector of a candidate image in an image retrieval library and determines a text content feature vector of text extraction content corresponding to the candidate image; and determines image retrieval results that meet retrieval conditions from the image retrieval library based on the description feature vector, the candidate image feature vector, and the text content feature vector. The technical solution of the embodiment of the present invention determines image retrieval results that meet retrieval conditions from the image retrieval library based on the description feature vector, the candidate image feature vector, and the text content feature vector, thereby realizing multimodal retrieval, and by adding text description information and text extraction content, the retrieved image content is more extensive, the image retrieval results are more diverse, and the relevance and accuracy of image retrieval are improved.

[0049] As a first optional embodiment of the embodiment of the present invention, on the basis of the above embodiment, the step of determining the description feature vector of the image text description information, and respectively determining the candidate image feature vectors of the candidate images in the image retrieval library and determining the text content feature vectors of the text extraction content corresponding to the candidate images can be further specifically the following steps:

[0050] a1) Determine candidate images in the image retrieval library and text extraction content of the candidate images.

[0051] In this embodiment, the candidate images in the image retrieval library are first determined, and the text content contained in each candidate image is extracted. When the candidate image contains text, the obtained text extraction content is the text form of the text in the image. When the candidate image does not contain text, the text extraction content is empty.

[0052] b1) inputting the image text description information, candidate images and text extraction content into a preset feature analysis model as input information respectively, wherein the feature analysis model is obtained in advance by training an initial feature analysis model.

[0053] Among them, the feature analysis model can be understood as a pre-trained model with feature analysis functions. The feature analysis functions may include feature engineering, feature extraction, feature selection, and feature enhancement; the initial feature analysis model can be considered as a model that needs to be trained with a large amount of data in order to learn how to perform feature analysis on images, texts, audio, and other content.

[0054] In this embodiment, the initial feature analysis model is trained using a large amount of image and text data to obtain a feature analysis model; then the image text description information, candidate images and text extraction content are respectively input as input information into the preset feature analysis model.

[0055] As one implementation, the feature analysis model may include a feature analysis module and a feature enhancement module. Based on the above optional embodiment, the feature analysis model may process the input information, which may be further embodied as follows:

[0056] b11) performing feature analysis on the input information through the feature analysis module to obtain an initial information feature vector of the input information.

[0057] The feature analysis module can be considered as a module that can perform feature analysis on images and texts, and can be a large visual-language model such as the BLIP-2 model. The initial information feature vector can be considered as a feature vector obtained after the input information is input into the feature analysis module.

[0058] In this embodiment, the image text description information, candidate images and text extraction content are respectively input as input information into the feature analysis module of the feature analysis model, and feature analysis processing is performed on the three types of input information: the image text description information and text extraction content are respectively input into the text encoder in the feature analysis module, and the candidate image is input into the image encoder in the feature analysis module. Then, a feature extraction method such as a neural network architecture can be used to extract features from the image text description information, candidate images and text extraction content that have passed the encoder, so as to obtain an initial information feature vector of the input information.

[0059] Exemplarily, the feature analysis module adopts the BLIP-2 model. First, the image text description information and the text extraction content are respectively input into the text encoder in the feature analysis module, and the candidate image is input into the image encoder in the feature analysis module. Then, the image text description information, the candidate image and the text extraction content passed through the encoder are respectively input into the Q-former structure of the BLIP-2 model to obtain three types of initial information feature vectors output by the Q-former structure.

[0060] b12) The initial information feature vector is subjected to inference analysis and correction through the feature enhancement module to obtain an information correction feature vector, where the information correction feature vector is the description feature vector, the candidate image feature vector or the text content feature vector.

[0061] Among them, the feature enhancement module can be considered as a module that optimizes features according to different samples, which is used to ensure that the feature analysis model can better capture and utilize useful information in the data, thereby improving the accuracy and efficiency of prediction. The feature enhancement module can be a mixture of experts (MoE) model, a generative adversarial imitation learning model or a reinforcement learning model. The information correction feature vector can be understood as the vector obtained by the initial information feature vector through the feature enhancement module.

[0062] In this embodiment, the initial information feature vectors are respectively input into the feature enhancement modules, and more sophisticated reasoning analysis and correction processing are performed on the initial information feature vectors to obtain information correction feature vectors.

[0063] Exemplarily, after adopting the BLIP-2 model as the feature analysis model, the feature enhancement model adopts the MoE model, which is composed of a gating network and a group of expert networks, each of which is good at processing different aspects of a specific task. The initial information feature vectors output by the BLIP-2 model are respectively input into the MoE model, and each feature map is split into multiple sub-network feature maps using the model. Through the gating network, the MoE model can assign the task to the most suitable expert network for processing according to the characteristics of the input features, thereby correcting the initial information feature vector and obtaining the information correction feature vector.

[0064] The above technical scheme of this embodiment realizes the feature extraction of image text description information, candidate images and text extraction content by inputting input information into the feature analysis module, and improves the image retrieval accuracy and efficiency of the feature analysis model by inputting the input information into the feature enhancement module, so that the feature analysis model can better adapt to and handle complex tasks.

[0065] As another implementation, based on the above optional embodiment, the training steps of the feature analysis model can be further concretized into the following steps:

[0066] b13) Determine a sample information set, wherein the sample information set includes at least one sample information group, and each sample information group includes: sample text description information, a sample image, and sample text extraction content corresponding to the sample image.

[0067] Among them, the sample image can be considered as an image to be retrieved for training. The sample text description information can be considered as the text description information of the sample image, which can be simple or detailed. The text description information can include objects, scenes, actions, etc. in the image. The sample text extraction content can be understood as the text information extracted from the sample image. The sample information group can be considered as a set including sample text description information, sample images, and sample text extraction content corresponding to the sample images; the sample information set can be understood as a set including one or more sample information groups.

[0068] In this embodiment, an existing image and text dataset is selected, including a multimodal image and text dataset, such as LAION-5B, LAION-400M, etc., a Chinese image-text pair dataset, such as Wukong dataset, a bilingual visual instruction dataset, such as CogVLM-SFT-311K dataset, DALL-E3 model, etc. These datasets contain multiple sample images and sample text description information corresponding to the sample images; then, a text extraction tool is used to extract the text content contained in each sample image to obtain the sample text extraction content corresponding to the sample image; then, each sample image and its corresponding sample text description information, as well as the sample text extraction content, can be correspondingly combined through IP addresses, etc. to form one or more sample information groups, which constitute a sample information set.

[0069] b14) taking the sample information group in the sample information set as input information and inputting it into the initial feature analysis model to obtain a sample description feature vector, a sample image feature vector and a sample text content feature vector.

[0070] Among them, the sample description feature vector can be understood as a vector composed of the features of the sample text description information; the sample image feature vector can be considered as a vector composed of the features of the sample image; and the sample text content feature vector can be understood as a vector composed of the features of the sample text extracted content.

[0071] In this embodiment, the sample information group in the sample information set is used as input information, and the sample text description information, sample image, and sample text extraction content corresponding to the sample image in each sample information group are respectively input into the initial feature analysis model, and the input sample information group is processed by the initial feature analysis model, such as feature extraction, feature enhancement, and feature selection, to obtain the sample description feature vector, sample image feature vector, and sample text content feature vector output by the initial feature analysis model.

[0072] b15) Determine the loss function value based on the sample description feature vector, the sample image feature vector and the sample text content feature vector in combination with a given loss function formula.

[0073] Among them, the loss function can be understood as a joint calculation formula that combines different types of feature vectors. The loss function can use cross entropy loss, logarithmic loss or mean absolute error, etc.

[0074] In this embodiment, the sample description feature vector, the sample image feature vector and the sample text content feature vector are substituted into a given loss function formula to determine the loss function value. The smaller the loss function value, the closer the retrieved sample image is to the sample text description information, and the better the performance of the initial feature analysis model.

[0075] Optionally, based on the above optional embodiment, the loss function value can be determined according to the sample description feature vector, the sample image feature vector and the sample text content feature vector in combination with a given loss function formula, which can be further concretized into the following steps:

[0076] b151) Using a given cross entropy function to determine the first cross entropy value between the sample description feature vector and the sample image feature vector.

[0077] In this embodiment, the sample description feature vector and the sample image feature vector output by the initial feature analysis model are substituted into a given cross entropy function to calculate and obtain a first cross entropy value.

[0078] b152) Using the cross entropy function to determine a second cross entropy value between the sample description feature vector and the sample text content feature vector.

[0079] In this embodiment, the sample description feature vector and the sample text content feature vector output by the initial feature analysis model are substituted into a given cross entropy function to calculate and obtain a second cross entropy value.

[0080] b153) Add the first cross entropy value and the corresponding first weighting coefficient, and the second cross entropy value and the corresponding second weighting coefficient, and determine the sum as the loss function value.

[0081] Among them, the first weighting coefficient can be considered as the weight coefficient corresponding to the first cross entropy value; the second weighting coefficient can be understood as the weight coefficient corresponding to the second cross entropy value.

[0082] In this embodiment, a value obtained by multiplying the first cross entropy value by the first weighting coefficient is added to a value obtained by multiplying the second cross entropy value by the second weighting coefficient to obtain a sum result, and the sum result is determined as the loss function value.

[0083] The above technical scheme of this embodiment, by respectively calculating the first cross entropy value and the second cross entropy value and performing weighted summation on the first cross entropy value and the second cross entropy value, achieves the goal of taking into account both the correlation between the sample image and the sample text description information and the correlation between the sample text extraction content and the sample text description information, thereby improving the effect of image retrieval.

[0084] b16) adjusting the model parameters in the initial feature analysis model according to the loss function value, and using the updated model parameters to update the initial feature analysis model, returning to re-perform feature analysis processing until the training end condition is met, and determining the initial feature analysis model corresponding to the end of training as the feature analysis model.

[0085] In this embodiment, s1) before the training starts, the parameters of the initial feature analysis model (such as weights and biases) are randomly initialized; s2) the initial feature model uses the initialization parameters to process the sample information group to obtain the sample description feature vector, the sample image feature vector and the sample text content feature vector; s3) the loss function value is then calculated by a given loss function; s4) according to the gradient information of the loss function, an optimization algorithm (such as gradient descent, Adam, etc.) is used to adjust the model parameters in the initial feature analysis model; s5) the initial feature analysis model is updated with the updated model parameters; s6) the feature analysis process is re-performed using the updated model. Repeat steps s3) to s6) until a certain training end condition is met, such as reaching a predetermined number of iterations, the loss function value is lower than a certain threshold, or the performance on the validation set is no longer improved. When the training end condition is met, the initial feature analysis model of the model parameters updated for the last time is determined as the feature analysis model.

[0086] The above technical scheme of this embodiment provides specific training steps for the feature analysis model. By constructing a sample information set containing sample text extraction content, the loss function value is determined according to the sample description feature vector, the sample image feature vector and the sample text content feature vector, combined with a given loss function formula. Through continuous iterative optimization, a feature analysis model that meets image retrieval requirements is obtained.

[0087] c1) Obtaining the description feature vector, the candidate image feature vector and the text content feature vector outputted by the feature analysis model respectively relative to the image text description information, the candidate image and the text extraction content.

[0088] In this embodiment, the image text description information, candidate images and text extraction content are respectively input into the feature analysis model, and the feature analysis model processes the input information to output a description feature vector, a candidate image feature vector and a text content feature vector respectively.

[0089] The above technical solution of this embodiment obtains a description feature vector, a candidate image feature vector and a text content feature vector by inputting the image text description information, the candidate image and the text extraction content of the candidate image as input information into a preset feature analysis model respectively, for subsequent feature matching.

[0090] As a second optional embodiment of this embodiment, based on the above optimization, the image retrieval results satisfying the retrieval conditions can be determined from the image retrieval library according to the description feature vector, the candidate image feature vector and the text content feature vector, which can be specifically implemented as the following steps:

[0091] a2) determining a first cosine distance value between the descriptive feature vector and the candidate image feature vector.

[0092] Among them, the first cosine distance value can be understood as the cosine value of the angle between the description feature vector and the candidate image feature vector, which is used to determine the similarity between the description feature vector and the candidate image feature vector, which is equivalent to determining the similarity between the image text description information and the candidate image. The larger the cosine distance value, the better the retrieval effect.

[0093] In this embodiment, the first cosine distance value between the candidate image feature vector and the description feature vector of each candidate image in the image retrieval library is determined one by one.

[0094] For example, for a candidate image and a text description of an image received by the feature analysis model, the first cosine distance Cos1 The calculation method can be:

[0095]

[0096] in, F1 is the description feature vector of the image text description information, F2 is the candidate image feature vector of the candidate image.

[0097] b2) Determine a second cosine distance value between the description feature vector and the text content feature vector.

[0098] Among them, the second cosine distance value can be understood as the cosine value of the angle between the description feature vector and the text content feature vector, which is used to determine the similarity between the description feature vector and the text content feature vector, which is equivalent to determining the similarity between the image text description information and the text extraction content of the candidate image.

[0099] In this embodiment, the second cosine distance value between the text content feature vector and the description feature vector of the text extraction content corresponding to the corresponding candidate image in the image retrieval library is determined one by one.

[0100] Continue with a2) for example, for example, the second cosine distance Cos2 The calculation method can be:

[0101]

[0102] in, F3 is the text content feature vector of the text extraction content corresponding to the candidate image.

[0103] c2) determining image retrieval results satisfying retrieval conditions from the image retrieval library according to a result of a summation of the first cosine distance value and the second cosine distance value.

[0104] In this embodiment, the first cosine distance value and the second cosine distance value are added and calculated, and an image retrieval result satisfying the retrieval condition is determined from the image retrieval library according to the sum calculation result.

[0105] For example, the summation result Cos The calculation method can be:

[0106] C os＝ C os 1+C os 2 ;

[0107] The above technical scheme of this embodiment calculates the first cosine distance value between the description feature vector and the candidate image feature vector, and the second cosine distance value between the description feature vector and the text content feature vector, and determines the image retrieval results that meet the retrieval conditions based on the sum of the first cosine distance value and the second cosine distance value, and utilizes the correlation between the text content feature vector and the description feature vector, thereby improving the accuracy and diversity of the image retrieval results.

[0108] As one implementation method, the second optional embodiment may further implement the following steps to determine the image retrieval result satisfying the retrieval condition from the image retrieval library based on the sum of the first cosine distance value and the second cosine distance value:

[0109] c21) For each candidate image in the image retrieval library, obtain a cosine sum result after adding the corresponding first cosine distance value and the second cosine distance value.

[0110] In this embodiment, for each candidate image in the image retrieval library, the corresponding text extraction content is extracted using a text extraction tool. Then, the image text description information, the candidate image and the text extraction content are input into the feature analysis model to determine the description feature vector of the image text description information, and obtain the candidate image feature vector of each candidate image and the text content feature vector of the text extraction content corresponding to the candidate image, respectively calculate the first cosine distance value between the description feature vector and each candidate image feature vector, and the second cosine distance value between the description feature vector and the text content feature vector corresponding to each candidate image, and add the first cosine distance value and the second cosine distance value to obtain the cosine summation result of each candidate image.

[0111] c22) According to the sum of the cosine sum results, the corresponding candidate images are arranged from large to small, and the candidate image in the previously set ranking is determined as the image retrieval result corresponding to the image text description information.

[0112] In this embodiment, the sum of the cosine sum results of each candidate image is arranged from large to small, and the candidate image in the pre-set ranking is determined as the image retrieval result corresponding to the image text description information. The number of pre-set rankings can be set according to needs.

[0113] For example, Figure 2 FIG. 4 is a logical implementation diagram of image retrieval applicable to the embodiment of the present invention. Figure 2 As shown, the candidate images in the image retrieval library can be in the form of a list. For each candidate image in the candidate image list, a text extraction tool is used to extract the corresponding text extraction content to obtain a text extraction content list corresponding to the candidate image list; then the input image text description information is input into the feature analysis model to obtain a description feature vector, and each candidate image and its corresponding text extraction content are respectively input into the feature analysis model to obtain a candidate image feature vector and a text content feature vector. When all candidate images and the corresponding text extraction contents are input into the feature analysis model, multiple candidate image feature vectors and multiple text content feature vectors can be obtained; the first cosine distance value between the candidate image feature vector and the description feature vector of each candidate image, and the second cosine distance value between the text content feature vector and the description feature vector of each candidate image are respectively determined, and then the cosine sum result of each candidate image is obtained; the cosine sum results of all the candidate images obtained are sorted from large to small, and the candidate images whose cosine sum results meet the retrieval conditions in the image retrieval library are output as the final image retrieval results.

[0114] The above technical scheme of this embodiment determines the candidate image corresponding to the pre-set ranking as the image retrieval result by arranging the sum of the cosine sum results of each candidate image from large to small, fully considers and utilizes the correlation between the image text description information and the text extraction content, expands the scope of image retrieval, and improves the accuracy of image retrieval.

[0115] Figure 3 A schematic diagram of the structure of an image retrieval device provided by an embodiment of the present invention is shown in FIG. Figure 3 As shown, the device may include: an information receiving module 31, a feature determination module 32 and a result retrieval module 33, wherein:

[0116] An information receiving module 31 is used to receive image text description information for image retrieval;

[0117] A feature determination module 32, used to determine the description feature vector of the image text description information, and respectively determine the candidate image feature vector of the candidate image in the image retrieval library and determine the text content feature vector of the text extraction content corresponding to the candidate image;

[0118] The result retrieval module 33 is used to determine the image retrieval results satisfying the retrieval conditions from the image retrieval library according to the description feature vector, the candidate image feature vector and the text content feature vector.

[0119] An image retrieval device provided by an embodiment of the present invention receives image text description information for image retrieval; determines a description feature vector of the image text description information, and respectively determines a candidate image feature vector of a candidate image in an image retrieval library and determines a text content feature vector of text extraction content corresponding to the candidate image; and determines image retrieval results that meet retrieval conditions from the image retrieval library based on the description feature vector, the candidate image feature vector, and the text content feature vector. The technical solution of the embodiment of the present invention determines image retrieval results that meet retrieval conditions from the image retrieval library based on the description feature vector, the candidate image feature vector, and the text content feature vector, thereby realizing multimodal retrieval, and by adding text description information and text extraction content, the retrieved image content is more extensive, the image retrieval results are more diverse, and the relevance and accuracy of image retrieval are improved.

[0120] Furthermore, the feature determination module 32 may be specifically used for:

[0121] Determining candidate images in the image retrieval library and text extraction content of the candidate images;

[0122] The image text description information, the candidate image and the text extraction content are respectively input as input information to a preset feature analysis model, wherein the feature analysis model is obtained in advance by training an initial feature analysis model;

[0123] Obtain the description feature vector, the candidate image feature vector and the text content feature vector outputted by the feature analysis model respectively relative to the image text description information, the candidate image and the text extraction content.

[0124] Furthermore, the feature analysis model may include a feature analysis module and a feature enhancement module;

[0125] The feature analysis model is specifically used for:

[0126] Performing feature analysis on the input information through the feature analysis module to obtain an initial information feature vector of the input information;

[0127] The feature enhancement module is used to perform reasoning analysis and correction on the initial information feature vector to obtain an information correction feature vector, where the information correction feature vector is the description feature vector, the candidate image feature vector or the text content feature vector.

[0128] Furthermore, the image retrieval device also includes a feature analysis model training module:

[0129] A sample information determination unit, used to determine a sample information set, wherein the sample information set includes at least one sample information group, and each sample information group includes: sample text description information, a sample image, and sample text extraction content corresponding to the sample image;

[0130] A sample information input unit, used to input the sample information group in the sample information set as input information into the initial feature analysis model to obtain a sample description feature vector, a sample image feature vector and a sample text content feature vector;

[0131] A loss function value determination unit, used to determine a loss function value according to the sample description feature vector, the sample image feature vector and the sample text content feature vector in combination with a given loss function formula;

[0132] A model training unit is used to adjust the model parameters in the initial feature analysis model according to the loss function value, and use the updated model parameters to update the initial feature analysis model, return to re-perform feature analysis processing until the training end condition is met, and determine the initial feature analysis model corresponding to the end of training as the feature analysis model.

[0133] Furthermore, the loss function value determination unit can be specifically used for:

[0134] Determine a first cross entropy between the sample description feature vector and the sample image feature vector using a given cross entropy function;

[0135] Determine a second cross entropy value between the sample description feature vector and the sample text content feature vector using the cross entropy function;

[0136] The first cross entropy value and the corresponding first weighting coefficient are added to the second cross entropy value and the corresponding second weighting coefficient, and the sum is determined as the loss function value.

[0137] Furthermore, the result retrieval module 33 may specifically include:

[0138] A first cosine value determining unit, used to determine a first cosine distance value between the description feature vector and the candidate image feature vector;

[0139] A second cosine value determination unit, used to determine a second cosine distance value between the description feature vector and the text content feature vector;

[0140] The search result determination unit is used to determine the image search result that meets the search condition from the image search library according to the sum calculation result of the first cosine distance value and the second cosine distance value.

[0141] Furthermore, the search result determination unit may be specifically used for:

[0142] For each candidate image in the image retrieval library, obtaining a cosine sum result after summing the corresponding first cosine distance value and the second cosine distance value;

[0143] According to the sum of the cosine sum results, the corresponding candidate images are arranged from large to small, and the candidate image at the previously set ranking is determined as the image retrieval result corresponding to the image text description information.

[0144] The image retrieval device provided in the embodiment of the present invention can execute the image retrieval method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0145] Figure 4A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0146] like Figure 4 As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0147] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0148] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 41 performs the various methods and processes described above, such as an image retrieval method.

[0149] In some embodiments, the image retrieval method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the image retrieval method described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the image retrieval method in any other appropriate manner (e.g., by means of firmware).

[0150] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0151] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0152] The embodiment of the present invention provides a computer storage medium. In the context of the present invention, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. The computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. Alternatively, the computer readable storage medium may be a machine readable signal medium. A more specific example of a machine readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0153] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0154] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0155] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0156] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0157] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An image retrieval method, characterized in that: include: receiving image text description information for image retrieval; Determine the description feature vector of the image text description information, and respectively determine the candidate image feature vectors of the candidate images in the image retrieval library and the text content feature vectors of the text extraction content corresponding to the candidate images; According to the description feature vector, the candidate image feature vector and the text content feature vector, an image retrieval result satisfying the retrieval condition is determined from the image retrieval library.

2. The method according to claim 1, characterized in that: The step of determining the description feature vector of the image text description information, and respectively determining the candidate image feature vectors of the candidate images in the image retrieval library and determining the text content feature vectors of the text extraction content corresponding to the candidate images, comprises: Determining candidate images in the image retrieval library and text extraction content of the candidate images; The image text description information, the candidate image and the text extraction content are respectively input as input information to a preset feature analysis model, wherein the feature analysis model is obtained in advance by training an initial feature analysis model; Obtain the description feature vector, the candidate image feature vector and the text content feature vector outputted by the feature analysis model respectively relative to the image text description information, the candidate image and the text extraction content.

3. The method according to claim 2, characterized in that The feature analysis model includes a feature analysis module and a feature enhancement module; The step of processing the input information by the feature analysis model comprises: Performing feature analysis on the input information through the feature analysis module to obtain an initial information feature vector of the input information; The feature enhancement module is used to perform reasoning analysis and correction on the initial information feature vector to obtain an information correction feature vector, where the information correction feature vector is the description feature vector, the candidate image feature vector or the text content feature vector.

4. The method according to claim 2, characterized in that: The training steps of the feature analysis model include: Determine a sample information set, wherein the sample information set includes at least one sample information group, and each sample information group includes: sample text description information, a sample image, and sample text extraction content corresponding to the sample image; Input the sample information group in the sample information set as input information to the initial feature analysis model to obtain a sample description feature vector, a sample image feature vector, and a sample text content feature vector; Determine a loss function value according to the sample description feature vector, the sample image feature vector and the sample text content feature vector in combination with a given loss function formula; The model parameters in the initial feature analysis model are adjusted according to the loss function value, and the initial feature analysis model is updated with the updated model parameters, and the feature analysis process is returned to be performed again until the training end condition is met, and the initial feature analysis model corresponding to the end of the training is determined as the feature analysis model.

5. The method according to claim 4, characterized in that According to the sample description feature vector, the sample image feature vector and the sample text content feature vector, combined with a given loss function formula, a loss function value is determined, including: Determine a first cross entropy value between the sample description feature vector and the sample image feature vector using a given cross entropy function; Determine a second cross entropy value between the sample description feature vector and the sample text content feature vector using the cross entropy function; The first cross entropy value and the corresponding first weighting coefficient are added to the second cross entropy value and the corresponding second weighting coefficient, and the sum is determined as the loss function value.

6. The method according to claim 1, characterized in that The step of determining image retrieval results satisfying retrieval conditions from the image retrieval library according to the description feature vector, the candidate image feature vector, and the text content feature vector comprises: Determining a first cosine distance value between the descriptive feature vector and the candidate image feature vector; Determine a second cosine distance value between the description feature vector and the text content feature vector; According to the sum of the first cosine distance value and the second cosine distance value, an image retrieval result satisfying a retrieval condition is determined from the image retrieval library.

7. The method according to claim 6, characterized in that The step of determining, from the image retrieval library, an image retrieval result satisfying a retrieval condition according to a summation calculation result of the first cosine distance value and the second cosine distance value comprises: For each candidate image in the image retrieval library, obtaining a cosine sum result after summing the corresponding first cosine distance value and the second cosine distance value; According to the sum of the cosine sum results, the corresponding candidate images are arranged from large to small, and the candidate image at the previously set ranking is determined as the image retrieval result corresponding to the image text description information.

8. An image retrieval device, characterized in that: include: An information receiving module, used for receiving image text description information for image retrieval; A feature determination module, used to determine the description feature vector of the image text description information, and respectively determine the candidate image feature vector of the candidate image in the image retrieval library and determine the text content feature vector of the text extraction content corresponding to the candidate image; The result retrieval module is used to determine the image retrieval results that meet the retrieval conditions from the image retrieval library according to the description feature vector, the candidate image feature vector and the text content feature vector.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the image retrieval method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the image retrieval method according to any one of claims 1 to 7 when executed.

Images