Image-text retrieval method and device based on multi-modal large model
By introducing an adversarial granular text feature mechanism into the pre-trained model, the problem of inaccurate semantic matching of key attribute combinations in fine-grained retrieval is solved, achieving efficient fine-grained semantic matching and improved retrieval accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-15
- Publication Date
- 2026-04-28
AI Technical Summary
Existing pre-trained models based on global contrastive learning struggle to capture and accurately match the combined semantics of key attributes in fine-grained retrieval tasks. This results in the models being insensitive to subtle semantic differences, and fine-tuning the models leads to a decline in generalization performance.
By introducing an adversarial granular text feature mechanism, key modifiers are extracted from the query text, positive and negative granular text features are constructed, and similarity calculation and data cleaning strategies are combined with a pre-trained model to improve fine-grained semantic matching capabilities.
It significantly improves the accuracy and robustness of fine-grained multimodal retrieval, reduces the probability of false selection, and has low computational overhead and is easy to deploy.
Smart Images
Figure CN121935402A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of computer vision, natural language processing and information retrieval technology, and more specifically, to a method, apparatus, electronic device and computer-readable storage medium for image and text retrieval based on a multimodal large model. Background Technology
[0002] With the rapid development of artificial intelligence technology, multimodal information retrieval, especially image retrieval based on text descriptions, has become an important research direction at the intersection of computer vision and natural language processing. This technology aims to establish an effective connection between text semantics and visual content, and is widely needed in various scenarios such as intelligent assistants that can perform "text-based image search".
[0003] To achieve this goal, researchers have proposed multimodal alignment models based on large-scale pre-training. Among them, CLIP (Contrastive Language-Image Pre-Training) has attracted much attention due to its excellent zero-shot generalization ability. Typical models such as CLIP can map massive amounts of images and text to similar positions in a common feature space. This learning paradigm gives the model strong general image-text matching capabilities, enabling multimodal retrieval without training for specific tasks.
[0004] However, such pre-trained models based on global contrastive learning have significant limitations when dealing with fine-grained retrieval tasks. Because the training objective focuses on bringing positive sample pairs closer and pushing negative sample pairs further away, but does not explicitly constrain the model's ability to accurately distinguish detailed features such as object attributes, states, and relationships, the model becomes insensitive to subtle semantic differences in query statements in real-world applications. For example, when a user queries "an old man walking a yellow dog," the model might return images that partially match but not completely match, such as "an old man walking a black dog" or "a middle-aged man walking a yellow dog." This is because the model struggles to fully capture and accurately match the combined semantics of key attributes such as "old man," "yellow," and "dog."
[0005] To improve model performance on specific fine-grained tasks, a common approach in existing technologies is to fine-tune pre-trained models on dedicated datasets. However, this method often leads to the "catastrophic forgetting" problem, where the model loses the general representational abilities learned on the original large-scale data while adapting to new tasks, resulting in decreased generalization performance. Furthermore, fine-tuning requires additional labeled data and computational resources, and it struggles to flexibly handle diverse fine-grained query requirements in open domains. Therefore, how to significantly improve fine-grained query performance without retraining or fine-tuning the model, while fully utilizing existing efficient inference strategies, has become a pressing technical problem. This application is therefore proposed to address this issue. Summary of the Invention
[0006] The purpose of this application is to address the shortcomings of the prior art by providing a text and image retrieval method, apparatus, electronic device, and computer-readable storage medium based on a multimodal large model, so as to solve the technical problems existing in the prior art.
[0007] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a text and image retrieval method based on a multimodal large model, characterized in that the method includes: The query text is obtained and processed using a pre-trained image-text pair-based contrastive learning model. The top N image features in the base database that have a semantic feature similarity to the query text in descending order are matched to form a candidate image set; where N is a positive integer. Extract key modifying components from the query text, and form multiple adversarial granular text features based on the key modifying components to obtain an adversarial granular text feature group; wherein, the adversarial granular text features include at least one positive granular feature that is semantically consistent with the key modifying component, and multiple negative granular features that are semantically different from the key modifying component; Each image feature in the candidate image set is compared with each adversarial granular text feature in the adversarial granular text feature group to calculate the similarity, thereby obtaining the granular similarity set of each image feature corresponding to the adversarial granular text feature group. Based on the granularity similarity set and the adversarial granularity text feature group, images that meet the preset conditions are selected from the candidate image set according to the preset data cleaning strategy and output as the retrieval results.
[0008] Optionally, based on the granularity similarity set and the adversarial granularity text feature group, images that meet preset conditions are selected from the candidate image set as retrieval results according to a preset data cleaning strategy, including: In multiple sets of granularity similarity, it is determined whether the adversarial granular text feature relative to the maximum value of the granularity similarity in the set is semantically consistent with the positive granular feature in the adversarial granular text feature group; If so, the image corresponding to the set of granular similarity is determined as a candidate hit image, and among all the candidate hit images, it is determined whether the maximum value of the corresponding granular similarity is greater than the preset threshold. If so, the image corresponding to the maximum granularity similarity of that class is determined as the hit image, and the first X images in descending order of the corresponding maximum granularity similarity are taken as the final search results; where X is a preset positive integer less than N.
[0009] Optionally, the query text is obtained and processed using a pre-trained image-text pair-based contrastive learning model. The model then matches the top N image features in the base database that have a descending similarity to the semantic features represented by the query text, forming a candidate image set, including: Get the query text; The image-text pair-based contrastive learning model is used to extract features from the query text and all images in the base database, resulting in a set of image features composed of query text features and all image features. The feature similarity between the query text features and each image feature in the image feature set is calculated sequentially, and the images are sorted in descending order according to the feature similarity. The top N images are selected from the image feature set to form the candidate image set.
[0010] Optionally, the image-text pair-based contrastive learning model includes a text editor and a visual editor. The query text features are obtained by the text editor extracting text features from the query text, and the image features are obtained by the visual editor extracting visual features from the images in the base database.
[0011] Optionally, the method for calculating the feature similarity between the query text features and each image feature in the image feature set is matrix multiplication.
[0012] Optionally, each image feature in the candidate image set is compared with each adversarial granular text feature in the adversarial granular text feature group to calculate similarity, resulting in a granular similarity set for each image feature corresponding to the adversarial granular text feature group, including: Perform matrix multiplication between each image feature in the candidate image set and the adversarial granular text group; The matrix multiplication result is subjected to softmax normalization to obtain the granular similarity set of each image feature corresponding to the adversarial granular text feature group.
[0013] Optionally, key modifying components are extracted from the query text, and multiple adversarial granular text features are formed based on these key modifying components, resulting in an adversarial granular text feature set including: Based on a preset grammatical analysis strategy, the query text is analyzed to identify modifiers, and the modifiers that meet the conditions are identified as the key modifiers. For the semantics represented by the key modifying component, at least one positive example granular text consistent with the semantics of the key modifying component is formed, and multiple negative example granular texts different from the semantics of the key modifying component are formed. Feature extraction processing is performed on the positive example granular text and multiple negative example granular texts to obtain at least one positive example granular feature and multiple negative example granular features, which constitute the adversarial granular text feature group.
[0014] Secondly, embodiments of this application also provide a text and image retrieval device based on a multimodal large model, characterized in that the device comprises: The feature extraction module is configured to acquire the query text and process the query text using a pre-trained image-text pair-based contrastive learning model. It then matches the top N image features in the base database that have a semantic feature similarity to the query text in descending order to form a candidate image set; where N is a positive integer. The adversarial granularity processing module is configured to extract key modifying components from the query text, and form multiple adversarial granularity text features based on the key modifying components to obtain an adversarial granularity text feature group; wherein, the adversarial granularity text features include at least one positive granularity feature that is semantically consistent with the key modifying component, and multiple negative granularity features that are semantically different from the key modifying component. The calculation module is configured to perform similarity calculations between each image feature in the candidate image set and each adversarial granular text feature in the adversarial granular text feature group, so as to obtain a granular similarity set of each image feature corresponding to the adversarial granular text feature group; The data cleaning module is configured to select images that meet preset conditions from the candidate image set and output them as search results based on the granularity similarity set and the adversarial granularity text feature group, according to a preset data cleaning strategy.
[0015] Optionally, the data cleaning module is specifically used to: determine, in multiple granularity similarity sets, whether the adversarial granular text feature corresponding to the maximum granularity similarity value in the set is semantically consistent with the positive granularity feature in the adversarial granular text feature group; if so, determine the image corresponding to this type of granularity similarity set as a candidate hit image, and in all the candidate hit images, determine whether the corresponding maximum granularity similarity value is greater than the preset threshold; if so, determine the image corresponding to the maximum granularity similarity value in this type as the hit image, and take the first X images in descending order according to the corresponding maximum granularity similarity value as the final retrieval result; where X is a preset positive integer less than N.
[0016] Optionally, the feature extraction module is specifically used for: obtaining query text; using the image-text pair-based contrastive learning model to extract features from the query text and all images in the base database, respectively obtaining a set of image features composed of query text features and all image features; sequentially calculating the feature similarity between the query text features and each image feature in the set of image features, and sorting them in descending order according to the feature similarity, selecting the top N images from the set of image features to form the candidate image set.
[0017] Optionally, the image-text pair-based contrastive learning model includes a text editor and a visual editor. The query text features are obtained by the text editor extracting text features from the query text, and the image features are obtained by the visual editor extracting visual features from the images in the base database.
[0018] Optionally, the calculation module is specifically used to: perform matrix multiplication on each image feature in the candidate image set and the adversarial granular text group; perform softmax normalization on the matrix multiplication result to obtain a granular similarity set for each image feature corresponding to the adversarial granular text feature group.
[0019] Optionally, the adversarial granularity processing module is specifically used for: identifying modifiers in the query text based on a preset grammatical analysis strategy, and determining the modifiers that meet the conditions as the key modifiers; constructing at least one positive example granular text consistent with the semantics of the key modifiers, and multiple negative example granular texts different from the semantics of the key modifiers, based on the semantics represented by the key modifiers; performing feature extraction processing on the positive example granular texts and the multiple negative example granular texts to obtain at least one positive example granular feature and multiple negative example granular features, constituting the adversarial granular text feature group.
[0020] Thirdly, embodiments of this application provide an electronic device, which includes a memory for storing one or more programs and a processor. When the one or more programs are executed by the processor, the above-described image and text retrieval method based on a multimodal large model is implemented.
[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described image and text retrieval method based on a multimodal large model.
[0022] The beneficial effects of this application are: This application provides a method, apparatus, electronic device, and computer-readable storage medium for image-text retrieval based on a multimodal large model. By introducing an adversarial granular text feature mechanism into a retrieval framework based on image-text contrastive learning, it extracts key modifiers from the query text and constructs semantically consistent positive granular text and semantically dissimilar negative granular text. This enables the retrieval process to not only rely on overall semantic similarity but also to explicitly characterize fine-grained semantic constraints in the query statement, thereby effectively improving fine-grained semantic matching ability and retrieval accuracy. Simultaneously, this application fully utilizes the zero-shot inference capability of pre-trained image-text models, overcoming the shortcomings of existing models such as CLIP in expressing complex semantically constrained scenarios without requiring additional model training or fine-tuning.
[0023] Furthermore, this application calculates the similarity between candidate images and the adversarial granular text features, and then makes a judgment based on the obtained granular similarity set. This ensures that candidate images not only satisfy positive semantic consistency but also remain distinguishable from non-target semantics, thereby effectively enhancing the discriminativeness of the retrieval results, reducing the probability of semantically similar but unrelated images being mistakenly selected, and improving the accuracy and reliability of the retrieval results. Since this method only introduces a small amount of adversarial granular text feature calculation and similarity comparison, and does not rely on additional model structures or complex inference modules, it has low computational overhead, is simple to implement, and easy to deploy, making it suitable for practical applications such as large-scale image retrieval.
[0024] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0025] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0026] Figure 1 A flowchart illustrating a text and image retrieval method based on a multimodal large model, provided for an embodiment of this application; Figure 2 A schematic diagram of the structure of a text and image retrieval device based on a multimodal large model provided in this application embodiment; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.
[0028] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.
[0029] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0030] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0031] In the description of this application, it should be noted that the terms "upper", "lower", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship that the product of this application is usually placed in. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.
[0032] In the description of this application, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.
[0033] First, the background technology involved in this application will be introduced.
[0034] With the continuous growth of multimedia data scale, image-text retrieval technology based on multimodal large-scale models has been widely applied in scenarios such as image search, content moderation, intelligent recommendation, and multimodal understanding. Among existing technologies, CLIP (Contrastive Language–Image Pre-Training) serves as an example. As a typical image-text contrastive learning model, CLIP pre-trains on large-scale image-text pair data, mapping images and text to a unified feature space, thereby enabling multimodal retrieval tasks such as "searching for images with text." Due to its strong versatility and zero-shot capability, image-text pair contrastive learning models have been widely used in practical systems.
[0035] However, the training objective of image-text pair contrastive learning models is primarily to bring semantically related samples closer together and distance unrelated samples through contrastive learning. The representations they learn tend to be more inclined towards global semantic matching rather than strictly distinguishing fine-grained semantic combinations. Therefore, in retrieval scenarios involving attributes, modifiers, or multiple constraints, existing models often struggle to accurately depict the fine semantic relationships between various modifiers in the text, easily leading to semantic incomplete matching problems.
[0036] Therefore, while fine-tuning the image-text pair contrastive learning model on a specific fine-grained dataset can improve its performance on this type of task, such targeted optimization often leads to a decline in the model's general representation ability learned on the original broad dataset, i.e., the problem of "catastrophic forgetting" or degradation of basic capabilities, thereby weakening its original generalization performance.
[0037] To address the aforementioned issues, this application proposes a multimodal retrieval method based on adversarial granularity. This method extends existing image-text contrast learning models, requiring no changes to the model structure or retraining; it achieves retrieval solely through vector similarity calculation. This approach offers advantages such as low implementation cost, high versatility, and ease of deployment. It overcomes the reduced generalization performance issues associated with existing fine-tuning techniques for the model.
[0038] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0039] Please see Figure 1 , Figure 1 This application provides an embodiment of a multimodal large-scale image and text retrieval method. The executing entity of this method can be an electronic device with data processing capabilities, such as a computer, server, or smartphone. Figure 1 As shown, the method includes: Step S101: Obtain the query text and process it using a pre-trained image-text pair-based contrastive learning model. Match the top N image features in the base database that have the semantic feature similarity to the query text in descending order to form a candidate image set.
[0040] Where N is a positive integer. It should be noted that the contrastive learning model for image-text pairs can be the CLIP model, or ALIGN (A Large-scale Image and Noisy-text embedding), FILIP (Fine-grained Language-Image Pre-training), etc., and is not limited here.
[0041] In one implementation, the method for implementing step S101 may be: obtaining query text; using an image-text pair-based contrastive learning model to extract features from the query text and all images in the base database, respectively obtaining query text features and image feature sets composed of all image features; sequentially calculating the feature similarity between the query text features and each image feature in the image feature set, and sorting them in descending order according to feature similarity, selecting the top N images from the image feature set to form a candidate image set.
[0042] In this implementation, the image-text pair-based contrastive learning model includes a text editor and a visual editor. Query text features are extracted by the text editor from the query text, while image features are extracted by the visual editor from the images in the base database. The method for calculating the feature similarity between the query text features and each image feature in the image feature set is matrix multiplication. Matrix multiplication can be calculated as a vector dot product or cosine similarity, and the choice can be flexibly set according to computational needs; no limitation is made here.
[0043] This implementation fully leverages the advantage of image-text pair contrastive learning models, which map images and text to a unified semantic space. It measures cross-modal similarity through efficient vector operations (such as matrix multiplication), and then performs feature extraction and global similarity calculation and ranking on the entire base database. This ensures that the batch of images most semantically relevant to the query text are included in the candidate image set without omission. This step minimizes the risk of missed detections due to premature limitation of the search scope, guaranteeing a sufficiently large and relevant candidate image set for deep filtering in subsequent adversarial granular processing stages. This is a prerequisite for the method to achieve high accuracy.
[0044] To facilitate understanding, a practical example is used below to illustrate the above implementation. Assume the image-text pair contrastive learning model is the CLIP model, and the task is to retrieve an image from the database corresponding to "a person walking a black dog." First, the visual editor of the CLIP model extracts visual features from the images in the database, obtaining image features for each image. These image features can be float vectors, each consisting of 1024 float values. Assuming there are M images in the database, the visual editor will generate M float vectors. Next, the CLIP model's text editor extracts text features from the input query text, obtaining text features with the same dimensionality as the image features. After completing the feature processing of the query text and the base database images, a similarity calculation is performed. The query text features are multiplied by the image feature set composed of the image features of all images in the base database to obtain the similarity between the query text and all images in the base database. The similarity between the query text features and M image features is sorted in descending order (from high to low). A similarity of 0.5 is set, and the images corresponding to the top N features with similarity greater than 0.5 are selected to form a candidate image set. Step S101 only performs a preliminary screening of images representing "a person walking a black dog" in the base database, which often results in incomplete matching or even complete inconsistencies. Possible scenarios include: the candidate image set containing not only images representing "a person walking a black dog" but also images representing "a person walking a white dog" and "a person walking a yellow dog." Subsequent steps S102-S104 will significantly improve this problem.
[0045] Step S102: Extract key modifiers from the query text, and form multiple adversarial granular text features based on the key modifiers to obtain an adversarial granular text feature group.
[0046] Among these, key modifiers, as contrastive learning models of image-text pairs, are more prone to identification blind spots. Generally, adjectives, adverbs, and other modifiers in a sentence are the parts that are most likely to cause model illusions and errors. Adversarial granular text features are used to characterize constraint information at different semantic granularities in the query text. Adversarial granular text features include at least one positive granularity feature semantically consistent with the key modifier, and multiple negative granularity features semantically different from the key modifier.
[0047] In one implementation, S102 can be achieved by: identifying modifiers in the query text based on a preset syntax analysis strategy, and determining the modifiers that meet the conditions as key modifiers; constructing at least one positive example granular text consistent with the semantics of the key modifiers, and multiple negative example granular texts different from the semantics of the key modifiers; performing feature extraction processing on the positive example granular text and multiple negative example granular texts to obtain at least one positive example granular feature and multiple negative example granular features, forming an adversarial granular text feature group. In this implementation, the existence of negative example granular texts, derived from the key modifiers, provides a clear "counterexample" reference for the cleaning strategy. A qualified image not only needs to be close to the positive example granular text, but also must be significantly far away from the negative example granular text. Through the adversarial design of "pulling closer to positive examples and pushing away negative examples," the accuracy of the model in judging detailed attributes is greatly improved.
[0048] To facilitate understanding, a practical example is used below to illustrate the above implementation. Suppose the query text is still "a person walking a black dog." This query text is easily misidentified by the model as having the modifier "black dog." Therefore, we construct a positive example text with the granularity of "black," and simultaneously construct adversarial granularities for "black": white dog, yellow dog, orange dog, brown dog, black and white dog, etc., as negative example texts. The specific number of granularities can be determined according to specific circumstances and needs. Feature extraction processing is performed on the determined positive and negative example texts to obtain a set of adversarial granularity text features.
[0049] Step S103: Calculate the similarity between each image feature in the candidate image set and each adversarial granular text feature in the adversarial granular text feature group, to obtain the granular similarity set of each image feature corresponding to the adversarial granular text feature group.
[0050] In one implementation, step S103 can be achieved by: performing matrix multiplication on each image feature in the candidate image set and the adversarial granular text group; performing softmax normalization on the matrix multiplication result to obtain the granular similarity set of each image feature corresponding to the adversarial granular text feature group.
[0051] In this implementation, a Softmax function is independently applied to normalize the granular similarity set of each image with all adversarial text features. The Softmax function exponentially amplifies the difference between the highest and second-highest scores. This means that if an image has a slightly higher match on a positive granular feature (such as "brown") than on all negative features (such as "black" or "white"), this slight advantage will be significantly amplified after Softmax, making its corresponding probability value very prominent. This method not only improves the confidence of the model's judgment but also makes the subsequent retrieval results obtained based on data cleaning strategies more robust and effective, significantly enhancing the system's ability to distinguish subtle semantic differences. For ease of understanding, a practical example is provided below to illustrate the above implementation. Each image feature in the candidate image set is multiplied by a matrix with the adversarial granular text group. If the candidate image set obtained through step S101 contains image features of N images, each of the N image features is multiplied by a matrix with each adversarial granular text feature in the adversarial granular text feature group obtained through S102. The matrix multiplication result is then normalized by softmax to obtain the granular similarity set of the N image features corresponding to the adversarial granular text feature group. Suppose that an image in the candidate image set has a semantic representation of a white dog, and its corresponding granular similarity set is {"black dog 0.031"; "white dog 0.520"; "brown dog 0.004"; "green dog 0.038"; "blue dog 0.069"; "red dog 0.030"; "purple dog 0.066"; "pink dog 0.065"; "gray dog 0.060"}. If the adversarial granular text feature is the granular similarity corresponding to "white dog" which is the maximum similarity value in this set, then the semantic meaning of the image is "a person walking a white dog". That is, the granular category corresponding to the maximum granular similarity value in the adversarial granular text feature set is determined as the semantic description to which the image belongs.
[0052] Step S104: Based on the preset data cleaning strategy, select the images corresponding to the granularity similarity set whose similarity to the positive example granularity features meets the preset conditions as the retrieval result output.
[0053] In one implementation, step S104 can be achieved by: determining, in multiple granularity similarity sets, whether the adversarial granular text feature corresponding to the maximum granularity similarity in the set is semantically consistent with the positive granularity feature in the adversarial granular text feature group; if so, determining the image corresponding to that type of granularity similarity set as a candidate hit image, and among all candidate hit images, determining whether its corresponding maximum granularity similarity is greater than a preset threshold; if so, determining the image corresponding to the maximum granularity similarity in that type as the hit image, and using the first X images in descending order based on the corresponding maximum granularity similarity as the final retrieval result. X is a preset positive integer less than N.
[0054] Similarity calculation in multimodal large models has inherent noise. Some erroneous details (negative examples) may accidentally generate high similarity scores with image features. By using the data cleaning strategy mentioned in this embodiment, the maximum value of the fine-grained feature similarity of the positive examples that passed the test is used as the sorting basis. The images are sorted in descending order of this score and a threshold is set for output. This ensures that the top-ranked images are not only generally relevant, but also the images with the highest matching degree in the details that users care about most. Thus, the accuracy of image and text retrieval results is achieved without changing the model's capabilities.
[0055] As mentioned earlier, the candidate image set contains images that do not represent "a person walking a black dog." Therefore, it may also include images representing "a person walking a white dog," "a person walking a yellow dog," and "a person walking a black and white dog." Assume the adversarial granular text feature set contains 10 adversarial strength text features, where the feature for "black dog" is a positive granular feature, and the features for "white dog," "brown dog," "yellow dog," "green dog," "blue dog," "red dog," "purple dog," "pink dog," and "gray dog" are negative granular features. The semantics of the adversarial granular text feature corresponding to the maximum granular similarity value in the granular similarity set are the same as the semantics represented in its corresponding image. That is, if an image in the candidate image set represents "a person walking a white dog", the adversarial granular text corresponding to the maximum granularity similarity value in the adversarial granular text feature group for that image is "white dog"; if an image in the candidate image set represents "a person walking a yellow dog", the adversarial granular text corresponding to the maximum granularity similarity value in the adversarial granular text feature group for that image is "yellow dog"; and if an image in the candidate image set represents "a person walking a black dog", the adversarial granular text corresponding to the maximum granularity similarity value in the adversarial granular text feature group for that image is "black dog". For ease of understanding, a practical example will be used to illustrate the above embodiment below. After determining the granularity category corresponding to the maximum granularity similarity value in the adversarial granular text feature group as the semantic description of the image, if the semantic category of an image in the candidate image set is consistent with the semantics represented by the key modifier (e.g., "black dog"), the image corresponding to that granularity similarity set is determined as a candidate hit image. Then, among all candidate hit images, it is determined whether the maximum granularity similarity value is greater than a preset threshold. The larger the maximum granularity similarity value, the closer the semantics represented by the corresponding image is to the semantics of the query text. Assuming the preset threshold is 0.5 and X is 5, images with a maximum granularity similarity value greater than 0.5 are determined as hit images, and the top 5 images with the maximum granularity similarity value in descending order are used as the final search results.
[0056] Steps S102-S104 introduce an adversarial granular text feature mechanism into the retrieval framework based on image-text contrast learning. This extracts key modifiers from the query text and constructs positive granular text with consistent semantics and negative granular text with different semantics. This enables the retrieval process to not only rely on overall semantic similarity but also to explicitly characterize the fine-grained semantic constraints in the query statement, effectively improving the fine-grained semantic matching capability and accuracy in the image-text retrieval process based on a multimodal large model.
[0057] Optionally, refer to Figure 2The diagram shown is a structural schematic of the image and text retrieval device based on a multimodal large model provided in this application. The device includes: The feature extraction module 201 is configured to acquire the query text and process the query text using a pre-trained image-text pair-based contrastive learning model. It matches the top N image features in the base database that have a semantic feature similarity to the query text in descending order to form a candidate image set; where N is a positive integer.
[0058] The adversarial granularity processing module 202 is configured to extract key modifying components from the query text, and form multiple adversarial granular text features based on the key modifying components to obtain an adversarial granular text feature group; wherein, the adversarial granular text features include at least one positive granular feature that is semantically consistent with the key modifying component, and multiple negative granular features that are semantically different from the key modifying component.
[0059] The calculation module 203 is configured to perform similarity calculations between each image feature in the candidate image set and each adversarial granular text feature in the adversarial granular text feature group, so as to obtain a granular similarity set of each image feature corresponding to the adversarial granular text feature group.
[0060] The data cleaning module 204 is configured to select images corresponding to the set of granular similarity that meet the preset conditions with respect to the positive example granular text as the retrieval result output based on the preset data cleaning strategy.
[0061] Optionally, the data cleaning module 204 is specifically used for: In multiple granularity similarity sets, it is determined whether the adversarial granular text feature with the maximum granularity similarity in the set is semantically consistent with the positive granularity feature in the adversarial granular text feature group. If so, the image corresponding to this type of granularity similarity set is determined as a candidate hit image. In all candidate hit images, it is determined whether the corresponding maximum granularity similarity is greater than a preset threshold. If so, the image corresponding to the maximum granularity similarity is determined as the hit image, and the first X images in descending order of the corresponding maximum granularity similarity are taken as the final retrieval result. Here, X is a preset positive integer less than N.
[0062] Optionally, the feature extraction module 201 is specifically used for: Obtain the query text; use the image-text pair-based contrastive learning model to extract features from the query text and all images in the base database, respectively obtaining the query text features and the image feature set composed of all image features; calculate the feature similarity between the query text features and each image feature in the image feature set in turn, and sort them in descending order according to the feature similarity, and select the top N images from the image feature set to form the candidate image set.
[0063] Optionally, the image-text pair-based contrastive learning model includes a text editor and a visual editor. The query text features are obtained by the text editor extracting text features from the query text, and the image features are obtained by the visual editor extracting visual features from the images in the base database.
[0064] Optionally, the calculation module 203 is specifically used for: Each image feature in the candidate image set is multiplied by the adversarial granular text group; the matrix multiplication result is then normalized using softmax to obtain the granular similarity set of each image feature corresponding to the adversarial granular text feature group.
[0065] Optionally, the adversarial granularity processing module 202 is specifically used for: Based on a pre-defined grammatical analysis strategy, the query text is analyzed to identify modifiers that meet certain conditions, which are then identified as key modifiers. The semantics represented by the key modifiers are used to construct at least one positive example granular text with semantics consistent with the key modifiers, and multiple negative example granular texts with semantics different from the key modifiers. Feature extraction is performed on the positive example granular text and the multiple negative example granular texts to obtain at least one positive example granular feature and multiple negative example granular features, which together form an adversarial granular text feature set.
[0066] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.
[0067] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).
[0068] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may be a computing device with data processing capabilities.
[0069] The electronic device includes: processor 301 and memory 302.
[0070] The memory 302 is used to store programs, and the processor 301 calls the programs stored in the memory 302 to execute the above method embodiments. The specific implementation and technical effects are similar, and will not be described in detail here.
[0071] Optionally, this application also provides a program product, such as a computer-readable storage medium, including a program that, when executed by a processor, performs the above-described method embodiments.
[0072] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0073] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0074] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0075] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
[0076] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.
Claims
1. A text and image retrieval method based on a multimodal large model, characterized in that, The method includes: The query text is obtained and processed using a pre-trained image-text pair-based contrastive learning model. The top N image features in the base database that have a semantic feature similarity to the query text in descending order are matched to form a candidate image set; where N is a positive integer. Extract key modifiers from the query text, and form multiple adversarial granular text features based on the key modifiers to obtain an adversarial granular text feature group; wherein, the adversarial granular text features include at least one positive granular feature that is semantically consistent with the key modifier, and multiple negative granular features that are semantically different from the key modifier; Each image feature in the candidate image set is compared with each adversarial granular text feature in the adversarial granular text feature group to calculate the similarity, thereby obtaining the granular similarity set of each image feature corresponding to the adversarial granular text feature group. Based on a preset data cleaning strategy, images corresponding to the set of granular similarity that meet preset conditions with the granular features of positive examples are selected as the retrieval results output.
2. The method according to claim 1, characterized in that, Based on a preset data cleaning strategy, images corresponding to the granularity similarity set whose similarity to the positive example granularity features meets preset conditions are selected as the retrieval results output, including: In multiple sets of granularity similarity, it is determined whether the adversarial granular text feature relative to the maximum value of the granularity similarity in the set is semantically consistent with the positive granular feature in the adversarial granular text feature group; If so, the image corresponding to the set of granular similarity is determined as a candidate hit image, and among all the candidate hit images, it is determined whether the maximum value of the corresponding granular similarity is greater than the preset threshold. If so, the image corresponding to the maximum granularity similarity of that class is determined as the hit image, and the first X images in descending order of the corresponding maximum granularity similarity are taken as the final search results; where X is a preset positive integer less than N.
3. The method according to claim 1, characterized in that, The query text is obtained and processed using a pre-trained image-text pair-based contrastive learning model. The top N image features in the base database with the semantic similarity to the query text in descending order are matched to form a candidate image set, including: Get the query text; The image-text pair-based contrastive learning model is used to extract features from the query text and all images in the base database, resulting in a set of image features composed of query text features and all image features. The feature similarity between the query text features and each image feature in the image feature set is calculated sequentially, and the images are sorted in descending order according to the feature similarity. The top N images are selected from the image feature set to form the candidate image set.
4. The method according to claim 3, characterized in that, The image-text pair-based contrastive learning model includes a text editor and a visual editor. The query text features are obtained by the text editor extracting text features from the query text, and the image features are obtained by the visual editor extracting visual features from the images in the base database.
5. The method according to claim 3, characterized in that, The method for calculating the feature similarity between the query text features and each image feature in the image feature set is matrix multiplication.
6. The method according to claim 1, characterized in that, Each image feature in the candidate image set is compared with each adversarial granular text feature in the adversarial granular text feature group to calculate its similarity, resulting in a granular similarity set for each image feature corresponding to the adversarial granular text feature group, including: Perform matrix multiplication between each image feature in the candidate image set and the adversarial granular text group; The matrix multiplication result is subjected to softmax normalization to obtain the granular similarity set of each image feature corresponding to the adversarial granular text feature group.
7. The method according to claim 1, characterized in that, Extract key modifiers from the query text, and form multiple adversarial granular text features based on these key modifiers, resulting in an adversarial granular text feature set including: Based on a preset grammatical analysis strategy, the query text is analyzed to identify modifiers, and the modifiers that meet the conditions are identified as the key modifiers. For the semantics represented by the key modifying component, at least one positive example granular text consistent with the semantics of the key modifying component is formed, and multiple negative example granular texts different from the semantics of the key modifying component are formed. The positive example granular text and multiple negative example granular texts are subjected to feature extraction processing to obtain at least one positive example granular feature and multiple negative example granular features, which constitute the adversarial granular text feature group.
8. A text and image retrieval device based on a multimodal large model, characterized in that, The device includes: The feature extraction module is configured to acquire the query text and process the query text using a pre-trained image-text pair-based contrastive learning model. It then matches the top N image features in the base database that have a semantic feature similarity to the query text in descending order to form a candidate image set; where N is a positive integer. The adversarial granularity processing module is configured to extract key modifying components from the query text, and form multiple adversarial granular text features based on the key modifying components to obtain an adversarial granular text feature group; wherein, the adversarial granular text features include at least one positive granular feature that is semantically consistent with the key modifying component, and multiple negative granular features that are semantically different from the key modifying component. The calculation module is configured to perform similarity calculations between each image feature in the candidate image set and each adversarial granular text feature in the adversarial granular text feature group, so as to obtain a granular similarity set of each image feature corresponding to the adversarial granular text feature group; The data cleaning module is configured to select, based on a preset data cleaning strategy, the images corresponding to the granular similarity set that meet the preset conditions with the similarity to the positive example granular text as the retrieval results output.
9. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is in operation, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is executed by a processor to perform the method as described in any one of claims 1-7.