Large model enhanced cross-modal retrieval method and system and related equipment

Abstract

The invention discloses a large model enhanced cross-modal retrieval method and system and related equipment. The cross-modal retrieval method comprises the following steps: acquiring an image-text pair; based on the image-text pair, obtaining large model enhancement information of the image-text pair, combining an original image, a text and the enhancement information, adopting a multi-branch encoder of a pre-training CLIP model to obtain a plurality of feature vectors, constructing a multi-stage collaborative alignment loss function, and performing collaborative alignment on an image mode and a text mode in a public semantic space; and training a model through the multi-stage collaborative alignment loss function and a pre-constructed training database, and performing retrieval through the trained model. According to the method, collaborative learning is carried out on the image-text features obtained by the encoder, auxiliary semantic enhancement information is added, and collaborative alignment is carried out on the image modality and the text modality in the public semantic space, so that a better retrieval network is trained, and the accuracy of image-text retrieval is improved.

Description

Technical Field

[0001] The present application relates to the technical field of cross-modal retrieval, and specifically to a large-model enhanced cross-modal retrieval method, system and related equipment. Background Art

[0002] With the rapid increase in the amount of multimodal data (such as text, images, and videos) from various sources, the demand for multimodal data analysis is also increasing. Cross-modal retrieval aims to search for corresponding text given a large-scale image as a query, or to search for corresponding images given a text as a query.

[0003] In recent years, great progress has been made in cross-modal image-text retrieval by advancing the structure of convolutional neural networks and LSTM to Transformer and pre-trained vision and language models. With better model structures and models pre-trained on large-scale data, the retrieval performance has been greatly improved, but these works rarely focus on a fundamental perspective of the task, namely cross-modal data bias. Specifically, text and its corresponding image may contain inconsistent information with each other, which is potentially harmful to retrieval performance.

[0004] On the one hand, the description in the text does not necessarily completely match the content in the image, that is, there is an inherent semantic gap between the two. Therefore, the text may contain redundant information that is irrelevant to the visual appearance of the corresponding image. On the other hand, images usually also contain information that is irrelevant to the text, such as background and noise information.

[0005] In general, although the existing technology has made some progress, it lacks in-depth analysis of the cross-modal data deviation characteristics of the task, which limits the further development of retrieval. Therefore, how to effectively analyze and solve the existing cross-modal data deviation is a technical problem that cross-modal retrieval urgently needs to solve. Summary of the invention

[0006] In this embodiment, a large model-enhanced cross-modal retrieval method, system, electronic device and storage medium are provided to solve the problems of cross-modal data deviation and low accuracy of image-text retrieval in related technologies.

[0007] In a first aspect, an embodiment of the present invention provides a large model enhanced cross-modal retrieval method, the cross-modal retrieval method comprising: Get image-text pairs; Based on the image-text pair, obtaining large model enhancement information of the image-text pair, including using a SAM segmentation model to perform fine-grained semantic segmentation on the image and select semantic entities therein; using a large language model to obtain a summary of the text and extract key semantic fragments; Combining the original image, text and enhanced information, using a multi-branch encoder of a pre-trained CLIP model to obtain multiple feature vectors, the multiple feature vectors including image features, text features, fine-grained image entity features and text summary features; Construct a multi-level co-alignment loss function to co-align image and text modalities in a common semantic space; The model is trained using the multi-level collaborative alignment loss function and a pre-constructed training database, the multi-branch encoder is trained using a gradient descent algorithm, and retrieval is performed using the trained model.

[0008] In an optional embodiment, the SAM segmentation model is used to perform fine-grained semantic segmentation on the image to select semantic entities therein, including: The SAM segmentation model is used to perform semantic segmentation on the image, and points uniformly sampled from the image are used as prompts to make image segmentation suggestions and obtain a set of all segmentation masks; Filter all segmentation masks to obtain a set of filtered image fine-grained semantic segmentation instances; To supplement the missing enhancements of some images, a quarter of the original image is randomly cropped and sampled as a fine-grained instance, which supplements the length of the image fine-grained semantic segmentation instance set.

[0009] In an optional embodiment, a large language model is used to obtain a summary of the text and extract key semantic fragments, including: The Llama2 large language model is used to obtain key information from the input long text, including title, ingredients and introduction; Generate a text summary based on the title, components, and introduction.

[0010] In an optional embodiment, the multi-branch encoder construction of the pre-trained CLIP model includes: An adaptation layer is added to the Transformer structure in the text encoder and the image encoder of the CLIP model; the adaptation layer is obtained by downsampling projection and upsampling projection and a residual connection; Based on the encoder of the CLIP model with an adaptation layer, construct feature encoders for different input information; The feature encoder includes an image encoder, a text encoder, an instance encoder and a text summary encoder.

[0011] In an optional embodiment, the original image, text and enhanced information are combined, and a multi-branch encoder of a pre-trained CLIP model is used to obtain multiple feature vectors, including: Use an image encoder with an adaptation layer to extract features from the input image to obtain image features; The text encoder of the CLIP model with an adaptation layer extracts features from the title, components, and introduction of the input text respectively to obtain three partial feature vectors. The parallel feature vectors are then input into a two-layer Transformer structure for fusion. The three partial feature vectors are concatenated and passed through a fully connected layer and a nonlinear activation function to obtain the final text feature; The text summary is encoded by adding the text encoder of the CLIP model with an adaptation layer to obtain the text summary features; The image encoder of the CLIP model is used to encode the features of the fine-grained image segmentation entities to obtain fine-grained image entity features.

[0012] In an optional embodiment, a multi-level collaborative alignment loss function is constructed to collaboratively align the image modality and the text modality in a common semantic space, including: The matching images and texts are defined as positive sample pairs, the mismatched ones are defined as negative sample pairs, and the cosine similarity is used as the standard to measure the relationship between sample pairs; The ring loss function is used to increase the similarity between positive sample pairs and reduce the similarity between negative sample pairs; Construct alignment loss based on multiple feature vectors. The alignment loss includes semantic alignment loss between image and text, semantic alignment loss between text and image fine-grained segmentation, semantic alignment loss between image and text summary, and semantic alignment loss within long text. The weighted sum of the alignment losses is used to obtain a multi-level collaborative alignment loss function.

[0013] In an optional embodiment, the model is trained by using the multi-level collaborative alignment loss function and a pre-built database, and retrieval is performed by using the trained model, including: For samples in the database that have no corresponding images but only contain text, the semantic alignment loss within the long text is used to perform semantic alignment within the text; for samples containing image-text, the multi-level collaborative alignment loss function is used for training in combination with the generated large model enhancement information; The model parameters are updated alternately through two sample training methods.

[0014] Compared with the prior art, the large model enhanced cross-modal retrieval method of the present invention has the following beneficial effects: The present invention uses the SAM segmentation model to segment fine-grained semantic targets in the image; summarizes the key information in the long text through the Llama2 large language model and generates a key semantic summary text; based on the CLIP lightweight fine-tuning image-text multi-branch encoder, the pre-trained model CLIP is fine-tuned with a lightweight adaptation layer to obtain feature vectors such as image and text; and a multi-level ring loss is designed to collaboratively learn the image-text features obtained by the encoder, and by adding auxiliary semantic enhancement information, the image modality and the text modality are collaboratively aligned in the common semantic space to train a better retrieval network, thereby improving the accuracy of image-text retrieval.

[0015] In a second aspect, an embodiment of the present invention provides a large model enhanced cross-modal retrieval system, including: An acquisition module, used to acquire image-text pairs; An enhanced information acquisition module is used to acquire large model enhanced information of the image-text pair based on the image-text pair, including using the SAM segmentation model to perform fine-grained semantic segmentation on the image and select semantic entities therein; using the large language model to acquire a summary of the text and extract key semantic fragments; A feature extraction module, for combining the original image, text and enhanced information, using a multi-branch encoder of a pre-trained CLIP model to obtain a plurality of feature vectors, wherein the plurality of feature vectors include image features, text features, fine-grained image entity features and text summary features; The loss function design module is used to construct a multi-level co-alignment loss function to co-align the image modality and the text modality in a common semantic space; The retrieval module is used to train the model through the multi-level collaborative alignment loss function and the pre-constructed training database, train the multi-branch encoder using the gradient descent algorithm, and perform retrieval through the trained model.

[0016] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a communication interface, a memory and a bus, wherein the processor, the communication interface and the memory communicate with each other through the bus, and the processor can call logic instructions in the memory to execute the steps of the method provided in the first aspect.

[0017] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the large model enhanced cross-modal retrieval method as described in the first aspect.

[0018] Compared with the prior art, the beneficial effects of the large model enhanced cross-modal retrieval system, electronic device and storage medium of the present invention are the same as those of the large model enhanced cross-modal retrieval method described in the first aspect, so they will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flowchart of a large model enhanced cross-modal retrieval method in an embodiment of the present invention; Figure 2 It is a schematic diagram of a large model-enhanced cross-modal retrieval method in an embodiment of the present invention; Figure 3 A schematic diagram of generating content for a large model enhancement of an image in an embodiment of the present invention; Figure 4 A schematic diagram of generating content for a large model enhancement of text in an embodiment of the present invention; Figure 5 A schematic diagram of an image-text multi-branch encoder for collaboratively enhancing semantic information in an embodiment of the present invention; Figure 6 It is a structural block diagram of a large model-enhanced cross-modal retrieval system in an embodiment of the present invention; Figure 7 1 is a structural block diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0022] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0023] In an embodiment of the present invention, a large model-enhanced cross-modal retrieval method is provided. Figure 1 is a flow chart of the large model enhanced cross-modal retrieval method of the present invention, Figure 2 Schematic diagram of a large model-enhanced cross-modal retrieval method in an embodiment of the present invention; Figure 1 and Figure 2 As shown, the process includes the following steps: S100, obtaining an image-text pair; Specifically, paired image-text pairs are extracted from the database, and for many-to-one or one-to-many situations, a random selection method is adopted to learn better model robustness.

[0024] S200, based on the image-text pair, obtaining large model enhancement information of the image-text pair, including using the SAM segmentation model to perform fine-grained semantic segmentation on the image and select semantic entities therein; using the large language model to obtain a summary of the text and extract key semantic fragments; First of all, it should be noted that by introducing the basic large model and using its general semantic extraction capabilities, semantic enhancement operations are performed on images and texts respectively. The image segmentation model (SAM segmentation model) is used to perform semantic segmentation on the image, and then all segmentation instances are screened to obtain fine-grained semantic instances of the image; the large language model is used to summarize and extract key semantic information from the text to generate a summary text of the long text.

[0025] In this embodiment, the SAM segmentation model is used to perform fine-grained semantic segmentation on the image and select semantic entities therein, including: The SAM segmentation model is used to perform semantic segmentation on the image, and points uniformly sampled from the image are used as prompts to make image segmentation suggestions and obtain a set of all segmentation masks; Filter all segmentation masks to obtain a set of filtered image fine-grained semantic segmentation instances; To supplement the missing enhancements of some images, a quarter of the original image is randomly cropped and sampled as a fine-grained instance, which supplements the length of the image fine-grained semantic segmentation instance set.

[0026] Specifically, first, we introduce the SAM segmentation model and use the "everything" mode to segment the input image to achieve zero-shot segmentation object proposal. For example, a 64*64 pixel point grid is used to uniformly sample the image. The sampled points are used as prompt vectors to predict the generation of segmentation masks. For a large number of masks returned, the loU prediction module of the model selects masks with high confidence. After obtaining a stable mask, non-maximum suppression (NMS) is used to filter duplicate masks. The threshold of NMS is set to 0.9. For a food image, an average of about 900 masks can be generated. They are sorted according to the mean of confidence and stability scores to obtain a mask set. : ; For collections All masks in are first sorted from large to small according to their area, and mask instances with too small area and loss of semantic information are filtered out. The updated set It only contains 16 instances. Furthermore, the instances are input into the image encoder of CLIP in parallel to obtain the instance feature set: ; In order to further filter out fine-grained semantic instances of images, such as Figure 3As shown in the figure, the text encoder of the CLIP model is used to obtain the feature vector of the text "an image of {topic}", and the similarity between the features in the instance set and the text features is calculated. The instances that meet the similarity threshold are screened out, and the top 4 with the largest similarity scores are selected as key fine-grained semantic instances. For images with less than 4 fine-grained semantic instances that meet the threshold, a quarter of the original image area is randomly cropped as an instance supplement.

[0027] Furthermore, if Figure 4 As shown in the figure, a large language model is used to obtain a summary of the text and extract key semantic fragments, including: The Llama2 large language model is used to obtain key information from the input long text, including title, ingredients and introduction; Generate a text summary based on the title, components, and introduction.

[0028] Specifically, for the large language model Llama2, we use Model, through the design prompt engineering, input text information into the big model.

[0029] For example, first, let the big model play the role of a powerful assistant, and enter the system prompt: You are a helpful assistant, try to provide help when answering questions, and the answer should be within 30 words. Secondly, enter the text summary prompt: Provide a text as follows: Title: {Title}, Components: {Components}, Introduction: {Introduction}. Return the key summary information of the long text. The summary should be rich in information and as objective as possible. Among them, {Title}, {Components}, and {Introduction} are replaced with the content in the long text. Let the long text be , including title Element ,introduce , then there is a text summary for: ); The maximum number of input tokens is set to 2048, the maximum number of output tokens is set to 256, and the model temperature coefficient is set to 0.6.

[0030] S300, combining the original image, text and enhanced information, using a multi-branch encoder of a pre-trained CLIP model to obtain multiple feature vectors, wherein the multiple feature vectors include image features, text features, fine-grained image entity features and text summary features; Combination Figure 5 As shown, it should be noted that the multi-branch encoder construction of the pre-trained CLIP model includes: An adaptation layer is added to the Transformer structure in the text encoder and the image encoder of the CLIP model; the adaptation layer is obtained by downsampling projection and upsampling projection and a residual connection; Specifically, for the pre-trained CLIP model, a lightweight adaptation layer is added to it, all CLIP pre-training parameters are kept frozen, and only the newly added adaptation layer is fine-tuned and updated. Specifically, in each layer of the Transformer structure in the CLIP text encoder and image encoder, an adaptation layer is added after the multi-head attention layer and the MLP layer. The adaptation layer includes a downsampling projection Project the hidden layer to a lower dimension , after an upsampling projection Projected back to the original dimension, and a residual connection , of the following form: ; As the activation function, the rectified linear unit function is used here, where the scaling factor is set Control projection dimensions, The dimension is 512 and the scaling factor is set to 4: ； Based on the encoder of the CLIP model with an adaptation layer, construct feature encoders for different input information; The feature encoder includes an image encoder, a text encoder, an instance encoder and a text summary encoder.

[0031] Combining the original image, text, and enhanced information, a multi-branch encoder from the pre-trained CLIP model is used to obtain multiple feature vectors, including: Use an image encoder with an adaptation layer to extract features from the input image to obtain image features; The text encoder of the CLIP model with an adaptation layer extracts features from the title, components, and introduction of the input text respectively to obtain three partial feature vectors. The parallel feature vectors are then input into a two-layer Transformer structure for fusion. The three partial feature vectors are concatenated and passed through a fully connected layer and a nonlinear activation function to obtain the final text feature; The text summary is encoded by adding the text encoder of the CLIP model with an adaptation layer to obtain the text summary features; The image encoder of the CLIP model is used to encode the features of the fine-grained image segmentation entities to obtain fine-grained image entity features.

[0032] Specifically, combined Figure 5 As shown, for the input image , feature extraction is performed using an image encoder with an adaptation layer. The image features are: ; For input text For example, it includes three parts: title, composition, and introduction, that is, , each of the three parts is encoded to extract independent features, using the text encoder of the CLIP model with an adaptation layer added.

[0033] Furthermore, for the title ,have: ; The text of ingredients and instructions is longer. For these two parts, they are first divided into sentences and then encoded in parallel. The maximum length of each sentence is 20 tokens, and the maximum length is 100 tokens. For 15 sentences.

[0034] ; ; The parallel feature vectors are then input into the two-layer Transformer structure for fusion, as follows: ; ; Finally, the three partial feature vectors are concatenated and passed through a fully connected layer and a nonlinear activation function. Finally, the hyperbolic tangent function is used here to obtain the final text features ( The dimensions are all consistent with 512 dimensions): ; For the introduced image fine-grained segmentation enhancement information ( ), use the image encoder of the completely frozen CLIP model for feature encoding, and then average all the segmentation enhancement features in the set to obtain (512 dimensions), there are: ; S400, construct a multi-level collaborative alignment loss function to collaboratively align image modality and text modality in a common semantic space; In this embodiment, a multi-level collaborative alignment loss function is constructed to collaboratively align the image modality and the text modality in a common semantic space, including: The matching images and texts are defined as positive sample pairs, the mismatched ones are defined as negative sample pairs, and the cosine similarity is used as the standard to measure the relationship between sample pairs; The ring loss function is used to increase the similarity between positive sample pairs and reduce the similarity between negative sample pairs; Construct alignment loss based on multiple feature vectors. The alignment loss includes semantic alignment loss between image and text, semantic alignment loss between text and image fine-grained segmentation, semantic alignment loss between image and text summary, and semantic alignment loss within long text. The weighted sum of the alignment losses is used to obtain a multi-level collaborative alignment loss function.

[0035] Specifically, a loss function is designed to coordinate multiple feature vectors to align text and visual modality spaces. In the retrieval process, matching images and texts are defined as positive sample pairs, and mismatched ones are defined as negative sample pairs. Cosine similarity is used as the criterion for measuring the relationship between sample pairs. express: ; The similarity between positive sample pairs is recorded as , the similarity between negative sample pairs is recorded as ,The ring loss function is used to increase the similarity between positive sample pairs and reduce the similarity between negative sample pairs, as follows: ; in, , are the marginal values ​​of positive and negative samples respectively. is the scaling factor, and are independent weighting factors, and A linear function of , is the relaxation factor, and its value is 0.25. Furthermore, the bidirectional ring loss is adopted: ; Next, based on the semantic enhancement vector and the original text and image vectors, a semantic alignment loss between image and text is designed. , semantic alignment loss between text and image fine-grained segmentation , semantic alignment loss between image and text summary In addition, for the three components within the long text, semantic alignment losses are constructed between each two to form the semantic alignment loss within the long text. Finally, the four groups of losses are combined to construct a multi-level large model enhanced collaborative alignment loss function : ; S500, training the model through the multi-level collaborative alignment loss function and the pre-constructed training database, training the multi-branch encoder using the gradient descent algorithm, and performing retrieval through the trained model.

[0036] The model is trained by using the multi-level collaborative alignment loss function and the pre-built database, and retrieval is performed by using the trained model, including: For samples in the database that have no corresponding images but only contain text, the semantic alignment loss within the long text is used to perform semantic alignment within the text; for samples containing image-text, the multi-level collaborative alignment loss function is used for training in combination with the generated large model enhancement information; The model parameters are updated alternately through two sample training methods.

[0037] Specifically, during the training phase, for samples that have no corresponding images in the database but only contain text, only the semantic alignment loss within the long text is used. Semantic alignment is performed within the text. For samples containing image-text, the generated large model is combined with enhanced information and the collaborative alignment loss function is used. For training, the two sample training methods are used to update the model parameters alternately.

[0038] In the test phase, the trained multi-branch encoder is used to obtain the trained image features I and text features . Calculate the cosine similarity of the feature vectors between the two , sort the candidate texts / images according to the similarity scores, and select the sorted samples according to the retrieval needs.

[0039] In order to illustrate the effectiveness of the present invention, the following experiments were conducted for verification.

[0040] Experimental verification was conducted on the Recipe1M dataset, using the commonly used recall indicators Recall@1, Recall@5, Recall@10 in the retrieval field, as well as the median of the correct position medR. The database size of the test model was 1,000 pairs and 10,000 pairs. The model performance was compared with other baseline models on these two test scales, proving the superior performance of the cross-modal retrieval method enhanced by the large model. Finally, ablation experiments were conducted on the proposed image fine-grained segmentation enhancement, text summary enhancement, multi-branch feature encoder, and collaborative alignment loss to verify the effectiveness of the design of each step.

[0041] As shown in Table 1 and Table 2, the experiments show the performance comparison of the model with other methods on the scale of 1000 and 10000 test pairs of Recipe1M, showing the results of the model in retrieving text from images.

[0042] Table 1 Performance comparison of models on Recipe1M with 1000 pairs of samples

[0043] Table 2 Performance comparison of models on Recipe1M with 10,000 sample pairs It can be seen from Tables 1 and 2 that the large model-enhanced cross-modal retrieval method shows excellent performance on both scales of test data. In addition to the retrieval performance exceeding other methods, the parameter update in the training phase is only about 14% of that of other full-scale fine-tuning CLIP methods (VLPCOOK, etc.), reflecting the excellent implementation of this method.

[0044] The embodiment of the present invention also provides a large model enhanced cross-modal retrieval system, which is used to implement the above method embodiment, and will not be repeated hereafter. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware or a combination of software and hardware is also possible and conceivable.

[0045] like Figure 6 As shown, Figure 6 : is a structural block diagram of the large model enhanced cross-modal retrieval system of the present invention, the system includes: An acquisition module 101 is used to acquire an image-text pair; The enhanced information acquisition module 102 is used to acquire large model enhanced information of the image-text pair based on the image-text pair, including using the SAM segmentation model to perform fine-grained semantic segmentation on the image and select semantic entities therein; using the large language model to obtain a summary of the text and extract key semantic fragments; A feature extraction module 103 is used to combine the original image, text and enhanced information, and use a multi-branch encoder of a pre-trained CLIP model to obtain multiple feature vectors, wherein the multiple feature vectors include image features, text features, fine-grained image entity features and text summary features; A loss function design module 104, for constructing a multi-level collaborative alignment loss function to collaboratively align the image modality and the text modality in a common semantic space; The retrieval module 105 is used to train the model through the multi-level collaborative alignment loss function and the pre-constructed training database, train the multi-branch encoder using the gradient descent algorithm, and perform retrieval through the trained model.

[0046] Figure 7A structural block diagram of an electronic device provided by an embodiment of the present invention, such as Figure 7 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the following method: Get image-text pairs; Based on the image-text pair, obtaining large model enhancement information of the image-text pair, including using a SAM segmentation model to perform fine-grained semantic segmentation on the image and select semantic entities therein; using a large language model to obtain a summary of the text and extract key semantic fragments; Combining the original image, text and enhanced information, using a multi-branch encoder of a pre-trained CLIP model to obtain multiple feature vectors, the multiple feature vectors including image features, text features, fine-grained image entity features and text summary features; Construct a multi-level co-alignment loss function to co-align image and text modalities in a common semantic space; The model is trained using the multi-level collaborative alignment loss function and a pre-constructed training database, the multi-branch encoder is trained using a gradient descent algorithm, and retrieval is performed using the trained model.

[0047] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0048] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including: Get image-text pairs; Based on the image-text pair, obtaining large model enhancement information of the image-text pair, including using a SAM segmentation model to perform fine-grained semantic segmentation on the image and select semantic entities therein; using a large language model to obtain a summary of the text and extract key semantic fragments; Combining the original image, text and enhanced information, using a multi-branch encoder of a pre-trained CLIP model to obtain multiple feature vectors, the multiple feature vectors including image features, text features, fine-grained image entity features and text summary features; Construct a multi-level co-alignment loss function to co-align image and text modalities in a common semantic space; The model is trained using the multi-level collaborative alignment loss function and a pre-constructed training database, the multi-branch encoder is trained using a gradient descent algorithm, and retrieval is performed using the trained model.

[0049] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0050] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A large model enhanced cross-modal retrieval method, characterized in that: The cross-modal retrieval method comprises: Get image-text pairs; Based on the image-text pair, obtaining large model enhancement information of the image-text pair, including using a SAM segmentation model to perform fine-grained semantic segmentation on the image and select semantic entities therein; using a large language model to obtain a summary of the text and extract key semantic fragments; Combining the original image, text and enhanced information, using a multi-branch encoder of a pre-trained CLIP model to obtain multiple feature vectors, the multiple feature vectors including image features, text features, fine-grained image entity features and text summary features; Construct a multi-level co-alignment loss function to co-align image and text modalities in a common semantic space; The model is trained using the multi-level collaborative alignment loss function and a pre-constructed training database, and retrieval is performed using the trained model.

2. The large model enhanced cross-modal retrieval method according to claim 1, characterized in that: Use the SAM segmentation model to perform fine-grained semantic segmentation on the image and select the semantic entities, including: The SAM segmentation model is used to perform semantic segmentation on the image, and points uniformly sampled from the image are used as prompts to make image segmentation suggestions and obtain a set of all segmentation masks; Filter all segmentation masks to obtain a set of filtered image fine-grained semantic segmentation instances; To supplement the missing enhancements of some images, a quarter of the original image is randomly cropped and sampled as a fine-grained instance, which supplements the length of the image fine-grained semantic segmentation instance set.

3. The large model enhanced cross-modal retrieval method according to claim 1, characterized in that: Use a large language model to obtain a summary of the text and extract key semantic fragments, including: The Llama2 large language model is used to obtain key information from the input long text, including title, ingredients and introduction; Generate a text summary based on the title, components, and introduction.

4. The large model enhanced cross-modal retrieval method according to claim 1, characterized in that: The multi-branch encoder construction of the pre-trained CLIP model includes: An adaptation layer is added to the Transformer structure in the text encoder and the image encoder of the CLIP model; the adaptation layer is obtained by downsampling projection and upsampling projection and a residual connection; Based on the encoder of the CLIP model with an adaptation layer, construct feature encoders for different input information; The feature encoder includes an image encoder, a text encoder, an instance encoder and a text summary encoder.

5. The large model enhanced cross-modal retrieval method according to claim 4, characterized in that: Combining the original image, text, and enhanced information, a multi-branch encoder from the pre-trained CLIP model is used to obtain multiple feature vectors, including: Use an image encoder with an adaptation layer to extract features from the input image to obtain image features; The text encoder of the CLIP model with an adaptation layer extracts features from the title, components, and introduction of the input text respectively to obtain three partial feature vectors. The parallel feature vectors are then input into a two-layer Transformer structure for fusion. The three partial feature vectors are concatenated and passed through a fully connected layer and a nonlinear activation function to obtain the final text feature; The text summary is encoded by adding the text encoder of the CLIP model with an adaptation layer to obtain the text summary features; The image encoder of the CLIP model is used to encode the features of the fine-grained image segmentation entities to obtain fine-grained image entity features.

6. The large model enhanced cross-modal retrieval method according to claim 1, characterized in that: Construct a multi-level collaborative alignment loss function to collaboratively align image and text modalities in a common semantic space, including: The matching images and texts are defined as positive sample pairs, the mismatched ones are defined as negative sample pairs, and the cosine similarity is used as the standard to measure the relationship between sample pairs; The ring loss function is used to increase the similarity between positive sample pairs and reduce the similarity between negative sample pairs; Alignment loss is constructed based on multiple feature vectors. The alignment loss includes semantic alignment loss between image and text, semantic alignment loss between text and image fine-grained segmentation, semantic alignment loss between image and text summary, and semantic alignment loss within long text. The weighted sum of the alignment losses is used to obtain a multi-level collaborative alignment loss function.

7. The large model enhanced cross-modal retrieval method according to claim 1, characterized in that: The model is trained by using the multi-level collaborative alignment loss function and the pre-built database, and retrieval is performed by using the trained model, including: For samples in the database that have no corresponding images but only contain text, the semantic alignment loss within the long text is used to perform semantic alignment within the text; for samples containing image-text, the multi-level collaborative alignment loss function is used for training in combination with the generated large model enhancement information; The model parameters are updated alternately through two sample training methods.

8. A large model-enhanced cross-modal retrieval system, characterized in that: include: An acquisition module, used to acquire image-text pairs; An enhanced information acquisition module is used to acquire large model enhanced information of the image-text pair based on the image-text pair, including using a SAM segmentation model to perform fine-grained semantic segmentation on the image and select semantic entities therein; using a large language model to acquire a summary of the text and extract key semantic fragments; A feature extraction module, for combining the original image, text and enhanced information, using a multi-branch encoder of a pre-trained CLIP model to obtain a plurality of feature vectors, wherein the plurality of feature vectors include image features, text features, fine-grained image entity features and text summary features; The loss function design module is used to construct a multi-level co-alignment loss function to co-align the image modality and the text modality in a common semantic space; The retrieval module is used to train the model through the multi-level collaborative alignment loss function and the pre-constructed training database, train the multi-branch encoder using the gradient descent algorithm, and perform retrieval through the trained model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the large model enhanced cross-modal retrieval method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the large model enhanced cross-modal retrieval method as described in any one of claims 1 to 7 are implemented.

Images