Cross-modal retrieval enhancement method and system based on visual large language model and medium

Abstract

The invention provides a cross-modal retrieval enhancement method based on a visual large language model. The method comprises the following steps: acquiring a preliminary cross-modal retrieval result of a text and an image; performing relevancy scoring on the preliminary cross-modal retrieval result by utilizing a visual big language model; screening the preliminary cross-modal retrieval results according to the relevancy score; and sorting the screening results, and displaying the most relevant result to the user. According to the method, the accuracy and reliability of cross-modal retrieval are remarkably improved, and high data annotation and training cost is effectively avoided. Meanwhile, as a universal retrieval enhancement scheme, the method can be seamlessly integrated into various existing cross-modal retrieval systems, and the application flexibility and economical efficiency of the method are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of cross-modal retrieval, and in particular to a cross-modal retrieval enhancement method, system and medium based on a visual large language model. Background Art

[0002] Existing cross-modal retrieval technologies are divided into image encoders and text encoders. The encoder encodes images and texts into vectors of the same dimension, and calculates the distance between the vectors to achieve the purpose of text retrieval of images or image retrieval of texts. However, due to the limited model capabilities of these cross-modal retrieval models themselves or the limited scope of the training datasets themselves, the image vectors and text vectors encoded by the models cannot be fully aligned, which leads to the retrieval of some images or texts with low relevance in the results of cross-modal retrieval, which seriously affects the effect of cross-modal retrieval. Many studies have begun to explore ways to improve the alignment capabilities of models. For example, contrastive learning is introduced to enhance the similarity between images and texts in the vector space. By optimizing the loss function, similar pairs of images and texts are closer and dissimilar pairs are farther away. In addition, pre-training with large-scale multimodal datasets can also significantly improve the performance of the model in cross-modal retrieval tasks. However, these improvement schemes still face problems such as high computing resource requirements and slow inference speed in practical applications. Therefore, in actual deployment, they usually have to balance between retrieval accuracy and computing efficiency. Summary of the invention

[0003] In view of the problems existing in the prior art, a cross-modal retrieval enhancement method, system and medium based on a visual big language model are provided, which perform secondary screening of existing retrieval results based on the visual big language model to effectively improve the accuracy.

[0004] The first aspect of the present invention proposes a cross-modal retrieval enhancement method based on a visual large language model, comprising:

[0005] Obtain preliminary cross-modal retrieval results of text and images;

[0006] Use the visual large language model to score the relevance of preliminary cross-modal retrieval results;

[0007] Filter the preliminary cross-modal search results based on relevance scores;

[0008] Sort the filter results to display the most relevant results to the user.

[0009] As a preferred solution, the preliminary cross-modal retrieval results are obtained using a general cross-modal retrieval method.

[0010] As a preferred solution, the use of the visual large language model to score the relevance of the preliminary cross-modal retrieval results specifically includes:

[0011] The preliminary cross-modal retrieval results are input into the pre-trained visual big language model, which performs semantic analysis on the image and text, and according to the input prompt words, the reranking task is converted into a "true" or "false" question to judge whether the retrieval results meet the query requirements. The visual big language model outputs the probability distribution of "true" and "false", and the probability of "true" is used as the relevance score of the retrieval result to judge the relevance of each retrieval result.

[0012] As a preferred solution, the design method of the prompt words specifically includes: for texts that specifically describe scenes, prompt words are added to a semantic framework; for texts that describe abstract descriptions, prompt words are used to strengthen sentiment analysis; prompt words designed for different scenarios are ultimately converted into "true" or "false" questions.

[0013] As a preferred solution, the screening of the preliminary cross-modal search results according to the relevance score specifically includes: eliminating the results with a relevance score lower than a preset threshold.

[0014] As a preferred solution, the sorting of the screening results includes: displaying the search results to the user in descending order of relevance scores.

[0015] As a preferred solution, the general cross-modal retrieval method at least includes: converting images and texts into vector forms through an image encoder and a text encoder, and then calculating the distance between the vectors to generate preliminary cross-modal retrieval results.

[0016] As a preferred solution, the method of using the visual large language model to score the relevance of the preliminary cross-modal retrieval results also includes:

[0017] The relevance scores of the preliminary cross-modal retrieval results are scored by combining multiple visual large language models, and the scoring weights of the multiple visual large language models are adjusted according to the specific application scenarios to calculate the relevance scores of the preliminary cross-modal retrieval results by weighted calculation.

[0018] A second aspect of the present invention proposes a system, including a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executes the cross-modal retrieval enhancement method based on a large visual language model as described in the first aspect.

[0019] The third aspect of the present invention proposes a computer-readable storage medium having computer program instructions stored thereon, characterized in that when the program instructions are executed by a processor, they are used to implement the process corresponding to the cross-modal retrieval enhancement method based on a visual large language model described in the first aspect.

[0020] Compared with the prior art, the beneficial effects of adopting the above technical solution are: the present invention not only significantly improves the accuracy and reliability of cross-modal retrieval, but also effectively avoids the high cost of data annotation and training. At the same time, as a general retrieval enhancement solution, the present invention can be seamlessly integrated into various existing cross-modal retrieval systems, greatly improving their application flexibility and economy. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of the cross-modal retrieval enhancement method based on the visual large language model proposed in the present invention.

[0022] Figure 2 Schematic diagram of cross-modal retrieval in one embodiment of the present invention.

[0023] Figure 3 FIG. 1 is a schematic diagram of relevance scoring of a visual large language model in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar modules or modules with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be construed as limitations on the present application. On the contrary, the embodiments of the present application include all changes, modifications and equivalents that fall within the spirit and connotation of the appended claims.

[0025] In order to solve the problem of inaccurate retrieval caused by model capabilities or data limitations in existing cross-modal retrieval technologies, the embodiment of the present invention proposes a cross-modal retrieval enhancement method based on a visual large language model, which can perform secondary screening based on existing cross-modal retrieval results, thereby optimizing the retrieval results. The specific scheme is as follows:

[0026] Step 1: Obtain preliminary cross-modal retrieval results of text and images.

[0027] In this embodiment, any existing cross-module search method can be used to obtain preliminary cross-modal search results. Figure 2 The method shown in FIG. 1 is used to illustrate that first, the image and text are converted into vector form through the image encoder and the text encoder, and then the distance between the vectors is calculated to generate a preliminary cross-modal retrieval result. Specifically, assuming that the vector after image encoding is represented as V i , the text encoded vector is represented as T i , then the preliminary search results can be obtained by calculating the cosine similarity between vectors:

[0028]

[0029] Among them, Sim(V i ,T i ) represents the image vector V i and the text vector T i The similarity value is used to obtain the image most relevant to each text, and then obtain the preliminary cross-modal retrieval results.

[0030] In practical applications, other methods may also be used to obtain cross-modal retrieval results, which is not required by the present invention.

[0031] Step 2: Use the visual large language model to score the relevance of the preliminary cross-modal retrieval results.

[0032] The visual language model has a deep understanding of images and texts, can focus on the details and overall semantics of the image at the same time, and accurately match the contextual semantics of the text, especially the understanding of complex semantics (such as abstract descriptions, metaphors, and contextual dependencies) far exceeds the general cross-modal model; at the same time, the visual language model provides a flexible mechanism (through Prompt Engineering) to adapt to different tasks, so that the model can dynamically adjust its evaluation strategy for relevance according to scene requirements; in addition, the visual language model also has reasoning capabilities and can identify implicit relationships (such as the association between the character actions in the image and the emotions described in the text), which is also a capability that traditional multimodal models do not have. Based on this, the visual language model is used in this embodiment to eliminate low-relevance results in the preliminary retrieval results.

[0033] It should be noted that although the visual language model has the ability to understand images and text, it is necessary to design specific prompt words and scoring mechanisms to apply it to the tasks in the scenario of this embodiment, because the model is usually trained for generation tasks or direct question-answering tasks, and cannot be directly used for relevance scoring tasks. In addition, the visual language model often has high computational costs. How to effectively integrate the preliminary search results and screen them requires optimizing the processing method of input data and the efficiency of model calls.

[0034] Please refer to Figure 3In this embodiment, after obtaining the preliminary cross-modal retrieval results, the retrieval text / image and the corresponding preliminary retrieval results are input into the visual big language model, and the model is used to perform in-depth semantic analysis of the image and text. At this time, it is also necessary to construct specific prompt words for each query and corresponding candidate item, and transform the re-ranking task into a "true" or "false" question to determine whether the candidate item (i.e., the preliminary retrieval result) meets the query requirements. For example, in the image title retrieval task, given an image query q_img and a text candidate c_txt, the prompt word may be: "Q: The title of this picture is: c_txt. A: Does this title correctly describe this picture? Please answer 'true' or 'false'."

[0035] According to the designed prompt words, let the visual language model output the relevance score between the image and the text to determine the relevance of each search result. The process of calculating the relevance score is as follows: input the prompt word into the visual language model, and the model will output the probability distribution of "true" and "false". At this time, the probability of "true" is used as the relevance score of the candidate, thereby converting the visual model into a model that outputs the relevance score. Specifically, the relevance score function used by the visual language model is f(V,T,P), and the similarity between the image and the text is expressed as Rel(V,T):

[0036] Rel(V,T)=f(V,T,P)

[0037] Among them, V represents the input image, T represents the input text, and P represents the prompt word of the large language model.

[0038] Among them, the prompt words can be flexibly adjusted according to the scene to obtain the best effect. Specifically:

[0039] For texts that specifically describe a scene (such as “a person is playing the piano”), prompt words can be added to a semantic framework (such as “please match the action or object contained in the description”).

[0040] For texts with abstract descriptions (such as "lonely dusk"), prompt words can enhance sentiment analysis (such as "analyzing whether the image conveys emotional characteristics").

[0041] The prompt words designed for different scenarios will eventually be converted into "true" or "false" questions to determine the relevance score, such as: "Analyze whether the image conveys the emotion of loneliness, and answer true or false."

[0042] In one embodiment, the prompt words are generated by algorithmic design. Specifically, the prompt words are dynamically generated by a generative model, and the best prompt words are adaptively generated according to the characteristics of the input data (such as the complexity of the text or the details of the image). At the same time, reinforcement learning can also be used to optimize the prompt words, so that the model can learn to generate more efficient prompt words in continuous training. In practical applications, different visual large language models can be selected according to demand. Since the visual large language model often has a high computational cost, in this embodiment, the model is optimized by engineering means to ensure reasonable resource overhead and prediction speed, including but not limited to low-bit model quantization, kernel operator optimization and other optimization means.

[0043] Step 3: Filter the preliminary cross-modal retrieval results based on the relevance score.

[0044] After obtaining the relevance score of each search result, the results with lower relevance scores are eliminated, that is, a filtering threshold θ is set to retain only the results where Rel(V,T) is greater than θ:

[0045] {(V i ,T i )|Rel(V,T)>θ}

[0046] Compared with traditional methods, the filtering function implemented by the visual large language model in the present invention ensures that only highly relevant retrieval results are retained.

[0047] Step 4: Sort the filtered results and display the most relevant results to the user.

[0048] Finally, the retained search results are scored for relevance and sorted according to the relevance score so that the most relevant content is displayed first. In this embodiment, different sorting algorithms can be selected to achieve different sorting accuracy and speed requirements, such as sorting based on relevance score, sorting based on cross entropy, etc.

[0049] In other embodiments, when using the visual big language model to perform relevance analysis on preliminary cross-modal retrieval results, visual big language models of different architectures can be integrated to enhance the semantic understanding ability of the model and improve the accuracy of screening and sorting.

[0050] When integrating visual language models of different architectures, the scoring weights of different visual language models can be dynamically adjusted according to different application scenarios to meet specific retrieval requirements. Similarly, the scoring weights of visual language models can also be dynamically adjusted in sorting to meet different application scenarios. Assuming the dynamic weight is [α 1 ,α 2 ,α 3 ,…,α N ], then the sorting function is:

[0051]

[0052] The cross-modal retrieval enhancement method based on the visual large language model eliminated by the present invention has the following advantages through secondary screening and sorting of preliminary cross-modal retrieval results:

[0053] 1. Substantial improvement in accuracy and relevance: The present invention uses the visual large language model to perform secondary screening and re-ranking of the initial search results, which can automatically filter out erroneous results with low relevance and re-rank high relevance results. Compared with the existing method of directly searching based on encoder distance, secondary screening and ranking greatly improves the accuracy of the results, making the search results more in line with user needs.

[0054] 2. Savings on data annotation and training costs: Traditional methods usually rely on large-scale annotated data sets and deep model training to improve retrieval accuracy and model alignment. This invention uses the auxiliary analysis capabilities of the visual large language model to perform secondary optimization on the initial retrieval results without retraining the model, thereby effectively saving high data annotation and training costs and reducing overall resource consumption and operating costs.

[0055] 3. Improved applicability and flexibility: The secondary retrieval and ranking method proposed in this invention is compatible with different types of cross-modal retrieval systems without relying on specific model structures. Unlike the specific optimization models in the prior art, this method is universal and can be applied to any cross-modal retrieval system, significantly improving its application scope and adaptability, and meeting the needs of different application scenarios.

[0056] 4. Significantly reduce the demand for computing resources: In the prior art, improving retrieval results usually means increasing the model size and computing resource investment. The present invention achieves performance optimization through secondary processing of existing retrieval results, avoiding repeated model training and high computing resource consumption. This allows the method to maintain efficient retrieval while having low demand for computing resources, saving system operating costs.

[0057] An embodiment of the present invention further proposes a system, including a memory and a processor, wherein the memory stores a computer program corresponding to the aforementioned cross-modal retrieval enhancement method based on a large visual language model, which can be loaded by the processor and executed.

[0058] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program includes a program code for executing the method shown in the flowchart.

[0059] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0060] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0061] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0062] As another aspect, the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the cross-modal retrieval enhancement method based on a visual large language model described in the above embodiment.

[0063] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the cross-modal retrieval enhancement method based on the visual large language model described in the above embodiment.

[0064] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0065] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0066] For those skilled in the art, the specific meanings of the above terms in the present invention can be understood in specific situations; the drawings in the embodiments are used to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0067] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A cross-modal retrieval enhancement method based on a visual large language model, characterized in that: include: Obtain preliminary cross-modal retrieval results of text and images; Use the visual large language model to score the relevance of preliminary cross-modal retrieval results; Filter the preliminary cross-modal search results based on relevance scores; Sort the filter results to display the most relevant results to the user.

2. The cross-modal retrieval enhancement method based on a visual large language model according to claim 1 is characterized in that: The preliminary cross-modal retrieval results are obtained using a general cross-modal retrieval method.

3. The cross-modal retrieval enhancement method based on a visual large language model according to claim 1 is characterized in that: The use of the visual large language model to score the relevance of the preliminary cross-modal retrieval results specifically includes: The preliminary cross-modal retrieval results are input into the pre-trained visual big language model, which performs semantic analysis on the image and text. Based on the input prompt words, the reranking task is converted into a "true" or "false" question to determine whether the retrieval results meet the query requirements. The visual big language model outputs the probability distribution of "true" and "false", and the probability of "true" is used as the relevance score of the retrieval result to determine the relevance of each retrieval result.

4. The cross-modal retrieval enhancement method based on a visual large language model according to claim 1 is characterized in that: The design method of the prompt words specifically includes: for texts that describe specific scenes, prompt words are added with semantic frameworks; for texts that describe abstract descriptions, prompt words are used to strengthen sentiment analysis; prompt words designed for different scenes are ultimately converted into "true" or "false" questions.

5. The cross-modal retrieval enhancement method based on a visual large language model according to claim 1, characterized in that: The screening of the preliminary cross-modal search results according to the relevance score specifically includes: eliminating the results with a relevance score lower than a preset threshold.

6. The cross-modal retrieval enhancement method based on a visual large language model according to claim 1, characterized in that: The sorting of the screening results includes: displaying the search results to the user according to the relevance scores from large to small.

7. The cross-modal retrieval enhancement method based on a visual large language model according to claim 2 is characterized in that: The general cross-modal retrieval method at least includes: converting images and texts into vector forms through an image encoder and a text encoder, and then calculating the distance between the vectors to generate preliminary cross-modal retrieval results.

8. The cross-modal retrieval enhancement method based on a visual large language model according to claim 1, characterized in that: The method of using the visual large language model to score the relevance of the preliminary cross-modal retrieval results also includes: The relevance scores of the preliminary cross-modal retrieval results are scored by combining multiple visual large language models, and the scoring weights of the multiple visual large language models are adjusted according to the specific application scenarios to calculate the relevance scores of the preliminary cross-modal retrieval results by weighted calculation.

9. A system, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executes the cross-modal retrieval enhancement method based on a large visual language model as described in any one of claims 1 to 8.

10. A computer-readable storage medium having computer program instructions stored thereon, wherein the program instructions, when executed by a processor, are used to implement a process corresponding to the cross-modal retrieval enhancement method based on a visual large language model as described in any one of claims 1 to 8.

Images