Multi-modal image retrieval method, system and equipment, medium and program product

Abstract

The invention relates to the technical field of image retrieval, and discloses a multi-modal image retrieval method, system and device, a medium and a program product, the method comprises the following steps: obtaining a to-be-retrieved image and / or character description information of the image, inputting the to-be-retrieved image and / or the character description information of the image into a pre-constructed multi-modal image retrieval model, and taking a feature vector output by the model as a query vector; performing similarity calculation on the query vector and a feature vector corresponding to each image in a preset image library, and taking the image of which the similarity is greater than a preset threshold as a matched image; and displaying the matched images as retrieval results according to the sequence of the similarity from high to low. The multi-mode image retrieval method can provide retrieval modes of searching images by text, searching images by images and searching images by image-text mixing, realizes multi-mode image retrieval capability, provides ideas of attribute learning and sorting learning, realizes image retrieval of information of other dimensions except categories, adapts to the characteristic of complexity of an image retrieval task, and improves the image retrieval efficiency. And the convenience, accuracy and robustness of image retrieval are improved.

Description

Technical Field

[0001] The present invention relates to the field of image retrieval technology, and in particular to a multimodal image retrieval method, system, device, medium and program product. Background Art

[0002] Image retrieval refers to the technology of finding target images through image recognition technology. This technology can be used in many scenarios. The information conveyed by images is often unstructured and has always faced many challenges, mainly:

[0003] (1) Image diversity: Images may present great diversity due to factors such as shooting angle, lighting conditions, resolution, occlusion, wear, etc., which increases the difficulty of recognition.

[0004] (2) Image openness: Image information is an unstructured open set, and massive images are often difficult to retrieve due to the lack of precise content description.

[0005] (3) Similarity: Different images may have many similarities, which makes it difficult for existing recognition systems to distinguish them.

[0006] The current mainstream image retrieval methods generally treat image retrieval as a common classification task or metric learning task, which is achieved by predicting image categories or calculating image similarity. Existing image retrieval methods have the following disadvantages:

[0007] (1) Image retrieval methods based on image classification require manual annotation of image categories. The inherent requirement is that the category boundaries of images can be divided. Due to the existence of a large number of similar images, the similarity of images is a degree indicator, and there may be non-conductive similarities, such as A is similar to B, B is similar to C, but C is not similar to A.

[0008] (2) The classification algorithm cannot correctly identify images outside the training category set. In most scenarios, the category set of images is not a closed set and cannot be used to predict unknown categories.

[0009] (3) To retrieve images through metric learning, it is necessary to prepare image templates in advance for feature comparison with the image to be retrieved. Since the template image and the image to be retrieved may have large style differences, the retrieval effect is greatly affected by the template itself. Different template images of the same category will produce different retrieval results. Summary of the invention

[0010] In view of this, the present invention provides a multimodal image retrieval method, system, device, medium and program product, which can overcome the dependence of image search on templates, improve the convenience of image retrieval and the accuracy of retrieval quality, and be capable of performing the task of retrieving images of unknown categories.

[0011] In a first aspect, the present invention provides a multimodal image retrieval method, the method comprising:

[0012] Obtain the image to be retrieved and / or the text description information of the image, input it into a pre-built multimodal image retrieval model, and use the feature vector output by the model as the query vector;

[0013] Calculate the similarity between the query vector and the feature vectors corresponding to each image in the preset image library, and take the image whose similarity is greater than a preset threshold as a matching image;

[0014] The matching images are displayed as search results in descending order of similarity.

[0015] The present embodiment provides a retrieval method that can provide text-to-image search, image-to-image search, and image-text mixed search, realizes multimodal image retrieval capability, realizes image retrieval of multi-dimensional information, adapts to the characteristics of the complexity of image retrieval tasks, and improves the convenience, accuracy, and robustness of image retrieval.

[0016] In an optional implementation, the multimodal image retrieval model is an end-to-end multi-task learning neural network model, and the construction process of the multimodal image retrieval model includes:

[0017] Obtain image samples of multiple categories and text description information corresponding to each image, and input them into a neural network model for training, and obtain a trained model as a multimodal image retrieval model for outputting a query vector; wherein the neural network model includes:

[0018] Text encoder, used to extract text features of text description information and generate text feature vectors;

[0019] An image encoder is used to extract image features of an image, generate an image feature vector, and obtain a memory feature vector based on the image features of all image samples;

[0020] The text-image alignment module is used to achieve text-image alignment by calculating the classification learning loss between text features, image features and image categories respectively;

[0021] An image-to-image alignment module is used to achieve image-to-image alignment by performing metric learning, ranking learning, and image feature classification learning between image features;

[0022] A category center calculation module is used to obtain category features by calculating the mean of memory features of images of the same category;

[0023] The centroid alignment module is used to realize centroid alignment by calculating the similarity between image features and category features;

[0024] Feature fusion module, used to fuse text features and image features to form text-image fusion features;

[0025] The text-image center module is used to realize text-image center alignment by calculating the similarity between the text-image fusion features and the category features;

[0026] The attribute selection module is used to extract the qualifier information from the text features to form attribute features;

[0027] A feature decomposition module is used to separate the attribute-related features in the image features and memory features through linear transformation to form image decomposition features;

[0028] The attribute alignment module is used to achieve attribute alignment by calculating the similarity between attribute features and image decomposition features.

[0029] The embodiment of the present invention introduces the category feature, a relatively stable feature, to represent the regression reference of each category, and uses proximity to the center of the same category and distance from the center of the different category as a constraint, thereby improving the effect of distance metric learning and reducing the influence of the quality of a single sample on metric learning; decomposing the limiting conditions in the text description to form text attribute features, and linearly decomposing the image features into different image attribute features, and aligning the text attribute features with the image attribute features, so that the model can learn the image features corresponding to different limiting conditions, thereby realizing multi-dimensional retrieval of images; through similarity sorting, a more fine-grained feature measurement is realized, and the problem of distance learning of sample features with fuzzy category boundaries is solved.

[0030] In an optional implementation, the text encoder is a Transformer, the image encoder is a Vision Transformer, and the text-image alignment module implements text-image alignment based on a CLIP algorithm through feature vectors output by the text encoder and the image encoder;

[0031] The image-to-image alignment module includes a distance metric, a batch normalization, and a classifier;

[0032] The feature decomposition module is a fully connected layer, and the decomposition dimension is the sum of the number of all attribute information;

[0033] The image centroid alignment module, the text-image centroid alignment module and the attribute alignment module are all implemented through distance metric learning;

[0034] The feature fusion module is a convolutional layer.

[0035] The embodiment of the present invention closely integrates image search by image and image search by text, and at the same time realizes local feature alignment through attribute alignment, thereby realizing image retrieval technology in different dimensions.

[0036] In an optional implementation, the loss of the text-image alignment module and the classification learning loss of the image-image alignment module are both represented by cross entropy, which is expressed as follows:

[0037]

[0038] Among them, g(x ij ) is the true value of the i-th sample in the j-th category, p(x ij ) is the predicted value of the i-th sample in the j-th category.

[0039] In an optional implementation, the losses of the image centroid alignment module, the text-image centroid alignment module, and the attribute alignment module and the metric learning loss of the image-image alignment module are implemented using triplet loss, and the functional formula is as follows:

[0040]

[0041] in, represents the i-th image, Representation and Positive samples of the same category, Representation and Negative samples of different categories; represents the intra-class distance, that is The distance from the positive sample;

[0042] represents the distance between classes, that is The distance from the negative sample; α is the lower limit of the difference between the intra-class distance and the inter-class distance; [*] + Indicates when If it is greater than 0, take this value; if it is less than 0, take 0.

[0043] In an optional implementation, the ranking loss in the image-to-image alignment module is implemented by constraining the feature distance, and the ranking loss is expressed as:

[0044]

[0045] in, and are close distance samples and long distance samples respectively, β j is the similarity difference, and its size is constrained according to different similarity levels.

[0046] The present invention implements the feature alignment task of each alignment model through a loss function, and a better feature vector can be obtained as a query vector through the constraint of the loss function.

[0047] In a second aspect, the present invention provides a multimodal image retrieval system, the system comprising:

[0048] A query vector output module is used to obtain the image to be retrieved and / or the text description information of the image, input it into a pre-built multimodal image retrieval model, and use the feature vector output by the model as the query vector;

[0049] A search result output module is used to calculate the similarity between the query vector and the feature vectors corresponding to each image in the preset image library, and take the image with a similarity greater than a preset threshold as a matching image;

[0050] The retrieval result display module is used to display the matching images in descending order of similarity as retrieval results.

[0051] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the multimodal image retrieval method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0052] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the multimodal image retrieval method of the first aspect or any corresponding embodiment thereof.

[0053] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the multimodal image retrieval method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 is a flowchart of a multimodal image retrieval method according to an embodiment of the present invention;

[0056] Figure 2 is a schematic structural diagram of a multimodal image retrieval model according to an embodiment of the present invention;

[0057] Figure 3 is a structural block diagram of a multimodal image retrieval system according to an embodiment of the present invention;

[0058] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0060] Existing image retrieval technology mainly focuses on the problem of searching images by images, and faces many limitations in practical applications. Although many shopping platforms also have retrieval technologies for searching images by text, they are all based on matching the image description text. In many scenarios, the images to be retrieved are often not equipped with introduction text. This method proposes a multimodal image retrieval method, introduces multidimensional learning and ranking learning, and opens up new ideas for retrieval with different limiting conditions. It is specifically implemented using a neural network model. The model supports searching images by image, searching images by text, and searching images by a mixture of text and images. The overall idea is to retrieve by combining image and text information. The model fuses the provided image and text information into a representation, and the similarity between this representation and all the image features to be matched is calculated. When the similarity is higher than a given threshold, the image matching is completed. The technical problems solved by the present invention include:

[0061] 1. Open category recognition. Image data is huge and open in categories. It is difficult to classify a large number of images using image classification technology, and it is not possible to perform open category image retrieval tasks.

[0062] 2. Image search relies on templates. The retrieval technology based on metric learning realizes the image search function and solves the problem of identifying open category targets. However, the premise of image search is to have images in hand. When there are no images in hand, retrieval cannot be performed. In addition, the retrieval effect of image search is restricted by the characteristics of the template image itself. For example, a template image with darker colors will preferentially match images with darker colors. When there is a certain color difference, the retrieved image may be wrong. In addition to color difference, other characteristics of the image (such as texture, clarity, etc.) will also affect the retrieval results.

[0063] 3. The quality of search results. Due to the complexity of the search task itself, in practical applications, the accuracy of the results and the convenience of the search process must be considered. Strengthening the search conditions through text descriptions will increase the reliability of the search results and the convenience of the search process.

[0064] 4. Ranking quality. The general image retrieval learning process only requires that the intra-class distance is smaller than the inter-class distance. When the similarity boundary between images is not clear enough, only considering the intra-class and inter-class distances cannot reflect the image similarity relationship. When these methods are applied to general tasks, there may be a huge deviation between the image scores retrieved by these methods and the actual similarity order.

[0065] According to an embodiment of the present invention, a multimodal image retrieval method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer device such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0066] In this embodiment, a multimodal image retrieval method is provided, which can be used in the above-mentioned computer device. Figure 1 is a flowchart of a multimodal image retrieval method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0067] Step S101, obtaining the image to be retrieved and / or the text description information of the image, inputting it into a pre-built multimodal image retrieval model, and the model outputs a query vector.

[0068] Specifically, the embodiment of the present invention supports image retrieval by image search, image search by text search, and mixed text and image search. Therefore, when searching, the user can enter the image, the text description of the image, the image and the corresponding text description in the search area, and then input the information entered by the user into the pre-built multimodal image retrieval model, and the model outputs the query vector (i.e., the vector representation of the input information). The pre-built multimodal image retrieval model in the embodiment of the present invention is implemented using an end-to-end multi-task learning neural network model, which combines the advantages of metric learning and image classification, while also overcoming the shortcomings of these two methods.

[0069] In an embodiment of the present invention, the process of constructing a multimodal image retrieval model includes: obtaining image samples of multiple categories and text description information corresponding to each image, and inputting them into a neural network model for training, and obtaining a trained model as a multimodal image retrieval model for outputting a query vector.

[0070] In a specific embodiment, for example, the image samples are various types of cigarette packs, and the text description information corresponding to "Lotus Cigarette" is, for example: a pack of lotus, green packaging, printed with the word "lotus" and "green lotus leaves and pink lotus", and the text description information is manually generated.

[0071] like Figure 2 As shown, the multimodal image retrieval model structure provided by the embodiment of the present invention comprises:

[0072] 1. Text Encoder, used to extract text features of text description information and generate text feature vectors;

[0073] 2. Image Encoder: It is used to extract the image features of the image, generate the image feature vector, and obtain the memory feature vector based on the image features of all image samples. In the actual model training process, in order to speed up the training and save computing resources, the image features are calculated by the image encoder parameter weights that are updated in real time, and the memory features are calculated by the most recently saved image encoder parameter weights (the model parameter weights are generally saved by the number of iterations or by iteration rounds, for example, one model parameter weight is saved every 1,000 iterations of learning, or one model parameter weight is saved after each round of training data is learned).

[0074] 3. A text-image alignment module is used to achieve text-image alignment by calculating the classification learning loss between text features, image features and image categories respectively; specifically, it is achieved by calculating the loss between text features and image categories, and between image features and image categories. The image category this time is a category label that people annotate the image, such as "dog", "landscape", "shirt", etc. Both the text features and the image features are fitted to a target (image category), thereby achieving text-image alignment. The text-image alignment module can realize the retrieval of images by text description under the condition of no image, that is, searching for images by text. The implementation method of Text Encoder, Image Encoder and text-image alignment in the embodiment of the present invention is the same as the CLIP (Contrastive Language-Image Pre-training) algorithm, wherein Text Encoder and Image Encoder are respectively implemented by Transformer and Vision Transformer. The CLIP algorithm is a multimodal learning algorithm proposed by OpenAI, which is mainly used for the understanding and association of images and texts. Its core idea is to map images and texts to the same feature space by contrastive learning, so that the model can understand the semantic association between images and texts, thereby realizing cross-modal understanding and generation tasks.

[0075] 4. An image-image alignment module is used to achieve image-image alignment by performing metric learning, ranking learning and image feature classification learning between image features; specifically, metric learning can measure the intra-class feature distance and inter-class feature distance of a group of images, ranking learning can sort the similarity between images, and classification learning can map image features to categories. The image-image alignment module in the embodiment of the present invention can realize the image search function, which includes distance measurement, batch normalization and classifier. Among them, the distance measurement method is cosine similarity, and it is required that the intra-class distance is smaller than the inter-class distance, the sample distance with high similarity is smaller than the sample distance with low similarity, and the classifier is realized through a fully connected layer. The present invention proposes the idea of ​​similarity ranking learning, and realizes finer-grained feature measurement through similarity ranking, so as to solve the problem of sample feature distance learning under blurred category boundaries.

[0076] 5. Category center calculation module, used to obtain category features by calculating the mean of memory features of images of the same category. The category features in the embodiment of the present invention represent the cluster center of the category (which can be regarded as category features). Single image sample features are often biased. The method based on metric learning generally optimizes model parameters by constraining the intra-class and inter-class distances of a set of image sample features. This learning process is greatly affected by the style, lighting, clarity and other characteristics of a single sample, making it difficult for the model loss to converge to a lower level. The embodiment of the present invention introduces the category feature, a relatively stable feature, to represent the regression reference of each category, and uses the constraint of being close to the center of the same category and away from the center of the different category to improve the effect of distance metric learning. The category center feature is more representative than the single image feature, which can overcome the style deviation of a single image sample and reduce the impact of the quality of a single sample on metric learning. It should be noted that the data set has been labeled with categories during model training, and the model can calculate the memory features of images of the same category based on the category labels.

[0077] 6. A centroid alignment module is used to achieve centroid alignment by calculating the similarity between image features and category features; in the embodiment of the present invention, the centroid alignment module is used to achieve alignment between image features and category features, so that the image features are close to the cluster center of the category to which they belong and away from the cluster centers of other categories. Centroid alignment is achieved through distance metric learning, and the distance metric method is cosine similarity, which also requires that the intra-class distance is smaller than the inter-class distance.

[0078] 7. A feature fusion module is used to fuse text features and image features to form text-image fusion features, which can realize image retrieval given a text description and an image template. The feature fusion module in the embodiment of the present invention is a convolutional layer, which uses the Conv(Concat(x)) operation to realize feature fusion.

[0079] 8. A text-image center module is used to realize text-image center alignment by calculating the similarity between the text-image fusion feature and the category feature; the text-image center alignment in the embodiment of the present invention makes the image feature close to the cluster center of its own category and away from the cluster centers of other categories, which is specifically realized by distance metric learning. The distance metric method is cosine similarity, and it is also required that the intra-class distance is smaller than the inter-class distance. The text-image center alignment module can realize the retrieval task of combining text search with image search.

[0080] 9. An attribute selection module is used to extract the qualifier information in the text features to form attribute features. Specifically, the attributes are modifying qualifiers or short sentences in the text description, such as "green packaging" and "lotus" printed on them. The attribute selection module of the embodiment of the present invention extracts the text features of these positions according to the corresponding positions of the attributes in the text description, and categorizes the sequence features to form attribute features.

[0081] 10. A feature decomposition module is used to separate attribute-related features from image features and memory features through linear transformation to form image decomposition features; the feature decomposition module in the embodiment of the present invention is a fully connected layer, and the decomposition dimension is the total number of all attribute information.

[0082] 11. An attribute alignment module is used to achieve attribute alignment by calculating the similarity between attribute features and image decomposition features. In the embodiment of the present invention, attribute alignment is achieved through distance metric learning. The distance metric method is cosine similarity, and the intra-class distance is also required to be smaller than the inter-class distance. By decomposing the limiting conditions in the text description to form text attribute features, and linearly decomposing the image features into different image attribute features, and then aligning the text attribute features with the image attributes, the model can learn the image features corresponding to different limiting conditions, thereby achieving broad and multi-dimensional retrieval of images, such as retrieving images with green packaging through the attribute of "green packaging".

[0083] Each alignment module in the embodiment of the present invention is actually implemented by a loss function, specifically:

[0084] The loss of the text-image alignment module and the classification learning loss of the image-image alignment module are expressed by cross entropy, which is expressed as:

[0085]

[0086] Among them, g(x ij ) is the true value of the i-th sample in the j-th category, p(x ij ) is the predicted value of the i-th sample in the j-th category.

[0087] The losses of the image centroid alignment module, the text-image centroid alignment module, the attribute alignment module, and the metric learning loss of the image-image alignment module are implemented using triplet loss, and the functional formula is as follows:

[0088]

[0089] in, represents the i-th image; Representation and Positive samples of the same category; Representation and Negative samples of different categories; Represents the intra-class distance (i.e. distance from the positive sample); Represents the distance between classes (i.e. distance from negative samples); α is the lower limit of the difference between the intra-class distance and the inter-class distance, [*] + Indicates when If it is greater than 0, take this value; if it is less than 0, take 0.

[0090] The sorting loss in the image alignment module is implemented by constraining the feature distance, specifically:

[0091]

[0092] in, and are close distance samples and long distance samples respectively, β j is the similarity difference, and its size is constrained according to different similarity levels.

[0093] The sum of the losses of the above modules is taken as the total loss of the model. During the training process, whether the model training is completed is determined according to the set number of iterations or the convergence of the loss. No specific restrictions are made here.

[0094] Step S102: Calculate similarity between the query vector and the feature vectors corresponding to each image in the preset image library, and use images with similarity greater than a preset threshold as matching images.

[0095] The feature vectors corresponding to each image in the preset image library of the embodiment of the present invention can be obtained by inputting the image and / or the text description information of the image in the image library into the feature vector output by the above-mentioned multimodal image retrieval model, which is used as the sample feature vector for matching in the image retrieval process. The cosine similarity between the query vector and the feature vector corresponding to each image in the image library is calculated. When the similarity is greater than the set similarity threshold, it indicates that a matching image can be retrieved.

[0096] Step S103: display the matching images in descending order of similarity as search results.

[0097] In practical applications, there may be multiple matching images with a similarity greater than a threshold value. In the embodiment of the present invention, all queried images are given in order of similarity as the result of image retrieval.

[0098] The multimodal image retrieval method provided by the embodiment of the present invention can provide retrieval methods of text-to-image search, image-to-image search, and image-text mixed image search, realizes the complex image retrieval capability across modalities, overcomes the dependence of image-to-image search on template quality, proposes the ideas of attribute learning and ranking learning, realizes image retrieval of other dimensional information besides categories, adapts to the characteristics of the complexity of image retrieval tasks, and improves the convenience, accuracy and robustness of image retrieval.

[0099] In the present embodiment, a multimodal image retrieval system is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0100] This embodiment provides a multimodal image retrieval system. Figure 3 As shown, including:

[0101] The retrieval input module 301 is used to obtain the image to be retrieved and / or the text description information of the image, and input it into the pre-built multimodal image retrieval model, and the feature vector output by the model is used as the query vector;

[0102] A search and matching module 302 is used to calculate the similarity between the query vector and the feature vectors corresponding to each image in the preset image library, and to take the image whose similarity is greater than a preset threshold as a matching image;

[0103] The search result display module 303 is used to display the matching images in descending order of similarity as the search results.

[0104] In some optional implementations, the multimodal image retrieval model of the retrieval input module 301 is an end-to-end multi-task learning neural network model. The construction process of the multimodal image retrieval model includes:

[0105] Obtain image samples of various categories and the text description information corresponding to each image, and input them into the neural network model for training. The trained model is used as a multimodal image retrieval model to output the query vector; the neural network model includes:

[0106] Text encoder, used to extract text features of text description information and generate text feature vectors;

[0107] An image encoder is used to extract image features of an image, generate an image feature vector, and obtain a memory feature vector based on the image features of all image samples;

[0108] The text-image alignment module is used to achieve text-image alignment by calculating the classification learning loss between text features, image features and image categories respectively;

[0109] An image-to-image alignment module is used to achieve image-to-image alignment by performing metric learning, ranking learning, and image feature classification learning between image features;

[0110] A category center calculation module is used to obtain category features by calculating the mean of memory features of images of the same category;

[0111] The centroid alignment module is used to realize centroid alignment by calculating the similarity between image features and category features;

[0112] Feature fusion module, used to fuse text features and image features to form text-image fusion features;

[0113] The text-image center module is used to realize text-image center alignment by calculating the similarity between the text-image fusion features and the category features;

[0114] The attribute selection module is used to extract the qualifier information from the text features to form attribute features;

[0115] A feature decomposition module is used to separate the attribute-related features in the image features and memory features through linear transformation to form image decomposition features;

[0116] The attribute alignment module is used to achieve attribute alignment by calculating the similarity between attribute features and image decomposition features.

[0117] In some optional implementations, the text encoder is a Transformer, the image encoder is a VisionTransformer, and the text-image alignment module implements text-image alignment based on a CLIP algorithm using feature vectors output by the text encoder and the image encoder;

[0118] The image-image alignment module includes distance metric, batch normalization and classifier; the feature decomposition module is a fully connected layer, and the decomposition dimension is the sum of the number of all attribute information; the image centroid alignment module, the text-image centroid alignment module and the attribute alignment module are all implemented through distance metric learning; the feature fusion module is a convolutional layer.

[0119] In some optional implementations, the loss of the text-image alignment module and the classification learning loss of the image-image alignment module are expressed by cross entropy, which is expressed as follows:

[0120]

[0121] Among them, g(x ij ) is the true value of the i-th sample in the j-th category, p(x ij ) is the predicted value of the i-th sample in the j-th category.

[0122] The losses of the image centroid alignment module, the text-image centroid alignment module, the attribute alignment module, and the metric learning loss of the image-image alignment module are implemented using triplet loss, and the functional formula is as follows:

[0123]

[0124] in, represents the i-th image, Representation and Positive samples of the same category, Representation and Negative samples of different categories; represents the intra-class distance, that is distance from the positive sample);

[0125] represents the distance between classes, that is distance from negative samples); α is the lower limit of the difference between the intra-class distance and the inter-class distance, [*] + Indicates when If it is greater than 0, take this value; if it is less than 0, take 0.

[0126] The sorting loss in the image alignment module is implemented by constraining the feature distance, specifically:

[0127]

[0128] in, and are close distance samples and long distance samples respectively, β j is the similarity difference, and its size is constrained according to different similarity levels.

[0129] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0130] The multimodal image retrieval system in this embodiment is presented in the form of functional units, where the units refer to ASIC (Application Specific Integrated Circuit) circuits, processors and memories that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0131] The embodiment of the present invention also provides a computer device having the above Figure 3 The multimodal image retrieval system shown.

[0132] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other, and can be installed on a common mainboard or installed in other ways as required. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output system (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 A processor 10 is taken as an example.

[0133] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0134] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0135] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0136] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0137] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0138] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0139] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0140] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A multimodal image retrieval method, characterized in that: include: Obtain the image to be retrieved and / or the text description information of the image, input it into the pre-built multimodal image retrieval model, and use the feature vector output by the model as the query vector; Calculate the similarity between the query vector and the feature vectors corresponding to each image in the preset image library, and take the image whose similarity is greater than a preset threshold as a matching image; The matching images are displayed as search results in descending order of similarity.

2. The method according to claim 1, characterized in that: The multimodal image retrieval model is an end-to-end multi-task learning neural network model. The construction process of the multimodal image retrieval model includes: Obtain image samples of multiple categories and text description information corresponding to each image, and input them into a neural network model for training, and obtain a trained model as a multimodal image retrieval model for outputting a query vector; wherein the neural network model includes: Text encoder, used to extract text features of text description information and generate text feature vectors; An image encoder is used to extract image features of an image, generate an image feature vector, and obtain a memory feature vector based on the image features of all image samples; The text-image alignment module is used to achieve text-image alignment by calculating the classification learning loss between text features, image features and image categories respectively; An image-to-image alignment module is used to achieve image-to-image alignment by performing metric learning, ranking learning, and image feature classification learning between image features; A category center calculation module is used to obtain category features by calculating the mean of memory features of images of the same category; The centroid alignment module is used to realize centroid alignment by calculating the similarity between image features and category features; Feature fusion module, used to fuse text features and image features to form text-image fusion features; The text-image center module is used to realize text-image center alignment by calculating the similarity between the text-image fusion features and the category features; The attribute selection module is used to extract the qualifier information from the text features to form attribute features; A feature decomposition module is used to separate the attribute-related features in the image features and memory features through linear transformation to form image decomposition features; The attribute alignment module is used to achieve attribute alignment by calculating the similarity between attribute features and image decomposition features.

3. The method according to claim 2, characterized in that The text encoder is a Transformer, the image encoder is a Vision Transformer, and the text-image alignment module realizes text-image alignment based on the CLIP algorithm through the feature vectors output by the text encoder and the image encoder; The image-to-image alignment module includes a distance metric, a batch normalization, and a classifier; The feature decomposition module is a fully connected layer, and the decomposition dimension is the sum of the number of all attribute information; The image centroid alignment module, the text-image centroid alignment module and the attribute alignment module are all implemented through distance metric learning; The feature fusion module is a convolutional layer.

4. The method according to claim 3, characterized in that The loss of the text-image alignment module and the classification learning loss of the image-image alignment module are both expressed by cross entropy, and the expression is: Among them, g(x ij ) is the true value of the i-th sample in the j-th category, p(x ij ) is the predicted value of the i-th sample in the j-th category.

5. The method according to claim 3, characterized in that: The losses of the image centroid alignment module, the text-image centroid alignment module, the attribute alignment module, and the metric learning loss of the image-image alignment module are implemented using triplet loss, and the functional formula is as follows: in, represents the i-th image, Representation and Positive samples of the same category, Representation and Negative samples of different categories; represents the intra-class distance, that is The distance from the positive sample; represents the distance between classes, that is The distance from the negative sample; α is the lower limit of the difference between the intra-class distance and the inter-class distance; [*] + Indicates when If it is greater than 0, take this value; if it is less than 0, take 0.

6. The method according to claim 5, characterized in that The ranking loss in the image-to-image alignment module is implemented by constraining the feature distance, and the ranking loss is expressed as: in, and are close distance samples and long distance samples respectively, β j is the similarity difference, and its size is constrained according to different similarity levels.

7. A multimodal image retrieval system, characterized in that: The system comprises: A query vector output module is used to obtain the image to be retrieved and / or the text description information of the image, input it into a pre-built multimodal image retrieval model, and use the feature vector output by the model as the query vector; A search result output module is used to calculate the similarity between the query vector and the feature vectors corresponding to each image in the preset image library, and take the image with a similarity greater than a preset threshold as a matching image; The retrieval result display module is used to display the matching images in descending order of similarity as retrieval results.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multimodal image retrieval method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the multimodal image retrieval method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the multimodal image retrieval method according to any one of claims 1 to 6.

Images