Complex text image retrieval method and system based on multi-modal large model thinking chain

Abstract

The invention discloses a complex text image retrieval method and system based on a multi-modal large model thinking chain. The method comprises the following steps: S1, carrying out adaptive semantic disassembly; s2, reconstruction and optimization are carried out; s3, constructing a matching probability matrix; elements of the matching probability matrix represent probability scores of matching degrees between the candidate images and the matching texts; combining the matched text with the candidate image, and sending the combined image to a pre-training visual language model pair by pair to obtain a corresponding matching score; then a binary discriminant scoring mechanism is used to convert the matching score into a probability score that the cue word is' YES '; and S4, according to the matching probability matrix, calculating the matching degree between each candidate image and the original proposition, and selecting the candidate image with the highest matching degree as an image retrieval result. The invention aims to extract and optimize semantic features from complex text description by using a self-adaptive local deconstruction and global optimization method and a binary discriminant scoring mechanism guided by cue words, so as to improve the accuracy and generalization ability of image retrieval.

Description

Technical Field

[0001] The present invention relates to the field of image retrieval, and in particular to a local-global information adaptive complex text image retrieval method and system based on multimodal large model thinking chain technology. Background Art

[0002] In recent years, with the rapid development of deep learning and multimodal technology, text-to-image retrieval (TIR) has become an important research direction in the fields of information retrieval and computer vision.

[0003] With the advent of the big data era, we are surrounded by massive amounts of image and text information. How to effectively retrieve the content users need from this information has become a pressing issue. Image Retrieval based on Contextual Description (IRCD) aims to extract key, effective information from highly abstract and semantically diverse text descriptions and accurately match them with images. This task is crucial in many practical scenarios, such as intelligent search engines that can locate specific images through natural language queries, autonomous driving systems that can quickly identify traffic scenes through text descriptions, and medical image analysis that uses text information to accurately locate lesions.

[0004] Traditional image retrieval methods mainly rely on low-level visual features of the image itself, such as color, texture, and shape, and achieve image matching by calculating the similarity between these features. However, this low-level feature-based method has obvious limitations, especially when facing the retrieval needs of complex text descriptions, it often fails to accurately capture the high-level semantic information expressed by the text. For example, when the retrieval text entered by the user involves scene context, object relationships, behavioral actions, or abstract concepts, retrieval methods that rely solely on low-level features find it difficult to find the image that best matches the expected semantics. With the advancement of deep learning technology, especially the emergence of vision-language models, people have begun to be able to handle more complex image retrieval tasks, such as retrieving images from text descriptions containing rich context and subtle visual clues.

[0005] However, the drawback of existing methods is that they usually require large-scale labeled data for supervised training of deep learning models, which requires a lot of hardware resources, computing resources, and time resources. Obtaining high-quality labeled data is also a huge and complex challenge. Data labeling is expensive, and labeled data is often limited in practical applications. Finally, the generalization ability of the model is often relatively limited. For image and text pairs that have not been seen or have a different distribution from the training data, the performance of the model often decreases significantly. Summary of the Invention

[0006] In light of the aforementioned shortcomings of the existing technology, the present invention provides a complex text image retrieval method and system based on a multimodal large-scale model thinking chain. This method aims to extract and optimize semantic features from complex text descriptions using adaptive local deconstruction, global optimization methods, and a binary discriminant scoring mechanism guided by prompt words, thereby improving the accuracy and generalization of image retrieval. Through hierarchical parsing of text information, this method effectively enhances cross-modal matching capabilities without the need for additional fine-tuning training, thereby enabling efficient image retrieval applications.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is:

[0008] First, a complex text image retrieval method based on a multimodal large model thinking chain includes the following steps:

[0009] S1. Adaptive semantic decomposition:

[0010] Adaptively decompose the original proposition using a large language model to generate multiple splitting plans; input the multiple splitting plans into the large language model for voting; and select the splitting plan with the highest vote rate as the optimal splitting plan;

[0011] S2. Reconstruction and optimization:

[0012] Perform a global rewrite of the original proposition;

[0013] Inputting the optimal splitting plan and the original proposition into a large language model to generate multiple local descriptor propositions and an optimized global representation;

[0014] S3. Construct matching probability matrix:

[0015] The dimension of the matching probability matrix is M×N, where M rows represent M candidate images and N columns represent N matching texts; the matching text includes multiple local descriptor propositions and one optimized global representation;

[0016] The elements of the matching probability matrix represent probability scores of the matching degree between the candidate image and the matching text;

[0017] The probability score is obtained by combining the matching text and the candidate image and feeding each pair into a pre-trained visual language model to obtain a corresponding matching score; then using a binary discriminant scoring mechanism to convert the matching score into the probability score of the prompt word being "yes";

[0018] S4. Output image retrieval results:

[0019] According to the matching probability matrix, the matching degree between each candidate image and the original proposition is calculated, and the candidate image with the highest matching degree is selected as the image retrieval result.

[0020] Preferably, in step S4, the matching algorithm is:

[0021] Normalize each column element in the matching probability matrix to obtain the probability matching score p between the candidate image and the matching text; perform weighted addition of the probability scores by row to obtain the matching degree S(I) of the i-th candidate image. i ):

[0022]

[0023] Among them, the weight m represents the number of the local descriptor propositions, I i represents the i-th candidate image, T g represents the optimized global representation, Represents the jth local descriptor proposition.

[0024] In the second aspect, a complex text image retrieval system based on a multimodal large model thinking chain includes: an input module, a semantic decomposition module, a reconstruction optimization module and a matching scoring module; the input module inputs the candidate image and the original proposition into the system; the semantic decomposition module performs adaptive semantic decomposition on the original proposition through the large language model to generate multiple splitting schemes; the semantic decomposition module inputs multiple splitting schemes into the large language model for voting; the splitting scheme with the highest vote rate is selected as the optimal splitting plan; the reconstruction optimization module globally rewrites the original proposition to generate multiple local descriptor propositions and an optimized global representation; the matching scoring module constructs a matching probability matrix: the elements of the matching probability matrix represent the probability scores of the matching degree between the candidate image and the matching text; the matching scoring module calculates the matching degree between each candidate image and the original proposition based on the matching probability matrix, and selects the candidate image with the highest matching degree as the image retrieval result;

[0025] The probability score is obtained by combining the matching text with the candidate image and feeding each pair into a pre-trained visual language model to obtain the corresponding matching score; then using a binary discriminant scoring mechanism, the matching score is converted into the probability score that the prompt word is "yes".

[0026] Compared with the prior art, the beneficial effects of the present invention are embodied in:

[0027] 1. This method first generates multi-angle splitting plans based on the thinking chain reasoning ability of the large language model for the characteristics of different complex descriptions, and introduces a self-consistency mechanism to cross-vote multiple plans, and finally selects the optimal disassembly plan to generate several sub-descriptions with clear semantics and concise structures. At the same time, with the help of the powerful semantic construction and optimization capabilities of the language model, the original description is restructured and linguistic errors are corrected to further improve the text quality and semantic clarity. Through the method proposed in the present invention, the original semantically complex text description can be effectively disassembled and optimized and rewritten, thereby significantly enhancing the visual language model's ability to understand the text content and improving the accuracy and robustness of image-text matching.

[0028] 2. During the image-text matching phase, a carefully designed cue word template is used to guide the visual language model to generate controlled binary outputs, which are then used to measure the degree of match between the image and the text. By constructing an image-text matching probability score matrix, a global score is generated for each image in the candidate set to determine its match with complex semantics, ultimately enabling accurate retrieval of images that best match the semantic intent. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic diagram of the method of Example 1;

[0030] Figure 2 is a schematic diagram of an adaptive proposition splitter in Example 2;

[0031] Figure 3 This is a schematic diagram of the probability scoring of the multimodal large model image and text based on prompt word restrictions in Example 2. DETAILED DESCRIPTION

[0032] In order to make the technical means, creative features, objectives and effects of the invention easier to understand, the present invention is further described with reference to specific figures. However, the present invention is not limited to the following implementation cases.

[0033] It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings in this specification are only used to match the contents disclosed in the specification so that people familiar with this technology can understand and read them. They are not used to limit the conditions under which the present invention can be implemented. Therefore, they have no substantive technical significance. Any structural modification, change in proportional relationship or adjustment of size should still fall within the scope of the technical content disclosed in the present invention without affecting the efficacy and purpose that can be achieved by the present invention.

[0034] Example 1:

[0035] like Figure 1 The complex text image retrieval method shown in FIG. 1 is based on a multimodal large model thinking chain, and includes the following steps:

[0036] S1. Adaptive semantic decomposition:

[0037] The large language model is used to adaptively decompose the original proposition and generate multiple splitting plans. The multiple splitting plans are input into the large language model for voting. The splitting plan with the highest vote rate is selected as the optimal splitting plan.

[0038] For the complex natural language description of the input, an adaptive proposition decomposition mechanism based on the large language model (LLM) is constructed. The mechanism first performs deep semantic modeling and deconstruction on the description of the original proposition, and then splits it into n independent sub-descriptions that focus more on detailed information. In this process, the model generates an initial decomposition plan through multi-perspective understanding, and iteratively enhances it with the help of the self-consistency mechanism of the language model, that is, a cross-voting strategy is performed on multiple decomposition versions, and finally the plan with the highest vote rate is selected as the optimal fine-grained decomposition result, that is, the optimal decomposition plan. This step utilizes the powerful semantic understanding and parsing capabilities of the LLM, so that when dealing with complex descriptions in different fields, it can adaptively understand complex descriptions and comprehensively split them from multiple angles without the need to manually preset specific decomposition rules, which significantly improves the interpretability and adaptability of the original text in cross-modal retrieval tasks.

[0039] S2. Reconstruction and optimization:

[0040] Perform a global rewrite of the original proposition;

[0041] The optimal splitting plan and the original proposition are input into the large language model to generate multiple local descriptor propositions and an optimized global representation;

[0042] In order to further refine the semantic core of complex propositions and enhance their logical coherence, an optimization strategy combining syntactic rewriting and semantic reconstruction is proposed. This strategy relies on the powerful language modeling capabilities of the large language model. Without changing the original semantics, it enables the large language model to globally rewrite the original complex proposition in a more precise and complete manner, eliminating potential grammatical errors such as missing components and mixing in the original description, thereby improving the clarity and processability of the text. On this basis, the optimal splitting plan and the globally rewritten original proposition are jointly input into the large language model to generate a final local information set with higher semantic consistency and more reasonable structure. Ultimately, two outputs corresponding to each complex original proposition can be obtained: a set of fine-grained local description sub-propositions, and an optimized global representation after semantic optimization.

[0043] S3. Construct matching probability matrix:

[0044] The dimension of the matching probability matrix is M×N, where M rows represent M candidate images and N columns represent the sum of the number of local descriptor propositions and optimized global representations, which is N. The element values of the matching probability matrix represent the probability scores of the degree of match between the candidate image and the optimized global representation or local descriptor proposition. The element values are obtained by combining the optimized global representation and local descriptor proposition with the candidate image respectively, and sending them one by one into the pre-trained visual language model to obtain the corresponding matching scores. Then, a binary discriminant scoring mechanism is used to convert the matching scores into probability scores that the prompt word is "yes".

[0045] The optimized global representation and its decomposed local descriptor propositions are combined with candidate images and fed pairwise into a pre-trained visual language model (VLM). The VLM embeds the image and text into a unified feature space, models the image-text relationship through a cross-attention mechanism, and outputs a matching score to measure the semantic fit between the text proposition (optimized global representation and local descriptor proposition) and the candidate image. To overcome the score compression problem caused by high semantic similarity between images, a binary discriminant scoring mechanism guided by a cue word is designed:

[0046] Since most candidate images are very similar, directly letting the VLM model score the degree of image-text matching will lead to the problem of different score discrimination. Therefore, we first use a specially designed prompt word to strictly limit the output of the multimodal model to a binary decision of "yes" or "no". in , image I i , and the processed optimized global representation and local descriptor proposition T p Input the VLM model together and get the model output logits (the original output score of the model without normalization), z:

[0047] z=[z1,z2,…,z k ]=VLM(P in ,I i ,T p )

[0048] Then, only the logit values corresponding to “yes” and “no” in the logits output by the open source visual language model are extracted (z yes , z no ), and normalize the two values through the SoftMax operation to convert them into two probability scores with a sum of 1, and take the probability score p representing "yes" yes As a model to measure the image-text pair (I i ,T j )’s image-text matching probability score s(I i ,T j ), s(I i ,Tj ) is expressed as:

[0049]

[0050] Reasoning is performed on all combinations of local descriptor propositions and optimized global representations with candidate images, resulting in an M×N matching probability matrix, where M rows correspond to M candidate images and N columns correspond to N text inputs. This matrix includes N-1 fine-grained local descriptor propositions and a semantically optimized global representation. This matrix fully captures the multi-level matching relationship between images and complex semantics, providing detailed support for subsequent image retrieval results.

[0051] S4. Output image retrieval results:

[0052] According to the matching probability matrix, the matching degree between each candidate image and the original proposition is calculated, and the candidate image with the highest matching degree is selected as the image retrieval result.

[0053] Based on the obtained matching probability matrix, the matching degree between each candidate image and the original proposition is calculated, and the candidate image with the highest score is used as the image retrieval result.

[0054] Specifically, normalize each column element in the matching probability matrix to obtain image I i With processing text T j The probability matching score p(I I ,T j ) to eliminate the impact of extreme abnormal score values on the final retrieval effect:

[0055]

[0056] Then, the probability scores of each row in the matching probability matrix are weighted and added together to form the degree of image-text matching between a single candidate image and the complex text, where the weight m is the number of simple propositions after splitting, and I i represents the i-th candidate image, T g represents the optimized global representation, Represents the jth local descriptor proposition split out, the i-th candidate image matching degree S(I i ) is expressed as:

[0057]

[0058] Example 2:

[0059] This embodiment provides an image retrieval system that takes as input a set of structurally similar images and a semantically complex natural language description. The system outputs an image that best matches the semantics of the description. The following describes the specific process of this system:

[0060] S1. The input module inputs the candidate image and the original proposition into the system:

[0061] Ten candidate images with highly similar visual content are provided. The images contain multiple male characters and background elements such as plants. The image layouts are similar to each other, making semantic distinction difficult.

[0062] The original query (complex text description) to be retrieved is: "The man below is standing behind the plant to the right, in the lower left corner. His legs are not fully visible. He is above the image. The image on the left looks forward, and the image on the right looks left." The present invention considers such grammatically incorrect or ambiguous queries, so this embodiment uses an example that may cause misunderstanding in the large model.

[0063] S2, the semantic decomposition module performs adaptive semantic decomposition on the original proposition:

[0064] The semantic decomposition module includes an adaptive proposition splitter. It uses a pre-installed large language model (such as the ChatGLM series) to automatically deconstruct the complex text description. By invoking the chain thinking mechanism of the large language model, it generates multiple versions of candidate split plans. Based on the self-consistency principle of the large language model, it performs consistency voting on these multiple versions of plans. Finally, it uses the optimal split plan to split the following fine-grained local sub-propositions:

[0065] (1) The man on the lower left is standing behind the potted plant. (2) We cannot see the man's legs on the lower left.

[0066] (3) The man on the upper right is looking left. (4) The man on the upper left is looking forward.

[0067] S3, the reconstruction and optimization module performs semantic structure reconstruction and fusion optimization on the optimal splitting plan:

[0068] The system uses prompts to guide the large language model to maintain the semantics of the original description while optimizing and correcting potential grammatical errors such as semantic confusion and missing components that may affect model matching. To achieve this goal, we have designed the following prompts for rewriting and optimizing the original description:

[0069] You are a professional text optimization assistant, skilled at rewriting text with grammatical problems to improve clarity. Please help me with the following tasks: Analyze user-entered sentences / paragraphs and identify potential issues such as grammatical errors or errors, ambiguous or unclear references, overly complex nested structures, and a mixture of technical terminology and colloquialisms. Rewrite the text while maintaining the original meaning. Requirements include: breaking down long and complex sentences into shorter ones, clarifying ambiguous references, reorganizing sentences using more standardized grammatical structures, and preserving key information while improving readability.

[0070] The original description mentioned in S1 and the carefully set prompt words are input into the large language model for semantic rewriting and global structure optimization, generating the following global semantic optimization proposition:

[0071] "In the upper half of the picture, the man on the left is looking forward, and the man on the right is looking to the left. The man in the lower half of the picture is standing to the right of the potted plant, and his legs are not fully visible."

[0072] Compared with the original text, the optimized text has stronger language clarity, semantic coherence and interpretability.

[0073] S4. The matching scoring module constructs a matching probability matrix through a multimodal matching scoring mechanism.

[0074] Each of the above local descriptor propositions and optimized global representations is paired with candidate images and fed into a pre-trained visual language model (such as LLaVA or BLIP-2). The pre-trained visual language model uses the prompt "Does the image content match the description?" as an input prefix, transforming the task into a binary judgment task (yes / no) and returning the raw output probability values corresponding to "yes" and "no." After softmax normalization, the probability score for each image-text match is obtained.

[0075] For example, the output of a candidate image matching the local descriptor proposition "the man in the upper right corner is looking left" is:

[0076] z yes =3.2,z no =1.1

[0077]

[0078] The system performs the above matching process for each candidate image against all local descriptor propositions and the optimized global representation, constructing a probability matrix of dimension 10×5 (4 local descriptor propositions + 1 optimized global representation). This matrix reflects the multi-level match degree of each candidate image at both fine-grained and global semantic levels.

[0079] S5. The matching scoring module calculates the matching degree and determines the search results:

[0080] The matching scoring module calculates the matching degree between each candidate image and the original proposition and selects the candidate image with the highest matching degree as the image retrieval result. By normalizing each column in the matching probability matrix, the interference caused by the high scores of individual propositions on the overall impact is eliminated. Then, the weighted matching score of each row is calculated. The matching function is as follows:

[0081]

[0082] Where m = 4, which represents the number of sub-propositions. The matching scoring module returns the image with the highest matching score as the retrieval result.

[0083] About the search results analysis:

[0084] In practice, this method successfully decomposes complex, semantically ambiguous text into multiple highly recognizable sub-propositions. Combined with global semantic optimization, it achieves deep multimodal understanding and high-precision image-text matching. Experiments show that compared to retrieval methods without semantic decomposition and optimization, the proposed solution demonstrates stronger differentiation and higher retrieval accuracy in scenarios with highly similar image content.

[0085] The present invention will be further described below with reference to the accompanying drawings:

[0086] The overall process of the present invention is as follows Figure 1 shown.

[0087] The present invention implements a three-level process of "planning-voting-splitting" for original semantically complex propositions by designing an adaptive proposition splitter. First, based on the multi-angle understanding ability of the large language model, multiple sets of splitting schemes are generated for complex text descriptions, and a self-consistency mechanism is introduced to conduct multiple rounds of voting to select the optimal splitting scheme, thereby obtaining a number of semantically focused and concisely structured local propositions. Secondly, combined with the language model's deep modeling ability for text semantics, the original proposition is rewritten and optimized to eliminate redundant components and correct grammatical errors, generating a global semantic representation with clear expression and reasonable structure.

[0088] During the matching phase, the local propositions obtained above, along with the optimized global propositions, are combined with image samples from the candidate set and fed into the visual language model pair by pair. By designing a prompt word template with binary output constraints, the model is guided to probabilistically score image-text matching relationships. Subsequently, an image-text matching score matrix is constructed, normalized, and weighted, and the overall matching score for each image is calculated under the guidance of multidimensional semantics. Finally, the image with the highest score is selected as the retrieval result image that best matches the complex input semantic description.

[0089] (2) Adaptive proposition splitter, such as Figure 2 shown.

[0090] In image retrieval tasks for complex texts, the original propositions usually contain a large amount of detailed descriptions and potential implicit semantics, such as spatial orientation, positional relationships, action states, and attribute features. These fine-grained local information plays a vital role in achieving high-precision image retrieval. Therefore, in response to this challenge, the present invention proposes an adaptive proposition splitter based on the consistency principle of a large language model, which aims to deeply analyze the multi-level semantic structure in complex propositions and enhance the accuracy and discriminability of image-text matching through fine-grained splitting and optimization. Its workflow is as follows:

[0091] First, the powerful language understanding ability of the large language model (LLM) is utilized to generate multiple sets of semantic splitting plans for the proposition based on the input complex proposition. Each set of splitting plans starts from a different semantic dimension, and strives to capture the multi-level details and potential implicit logic in the proposition. For example, for propositions describing space, some splitting plans focus on capturing the spatial relationship between objects, that is, determining the position, orientation and relative relationship of different objects; while other splitting plans may pay more attention to action or state characteristics, such as describing the dynamic changes or visual state of objects. Through this multi-angle and multi-dimensional splitting strategy, the present invention can effectively cover all potential semantic levels in complex propositions, thereby improving the semantic expression ability of complex propositions.

[0092] Let the original proposition be C. Then, through the semantic understanding ability of the large language model (LLM), the generated split plans P1, P2, ..., P n It can be expressed as:

[0093]

[0094] Among them, θ i Represents the split plan P i Different semantic dimensions used in .

[0095] To further ensure the accuracy and internal consistency of the final split plan, this paper introduces a voting strategy for split plans based on the majority consensus mechanism of the large language model. Specifically, the multiple split plans are re-input into the large language model for multiple rounds of judgment and voting. By counting the frequency of selection of different plans in the model's judgment, the split plan with the highest vote rate is selected as the final implementation plan.

[0096] Let P i Represents the split plan P i , then the optimal splitting solution P is finally selected * It can be calculated as follows:

[0097]

[0098] Among them, m represents the voting round, is the split plan P in round j i The number of votes.

[0099] This strategy fully utilizes the stability and consistency judgment capabilities of the large model in semantic judgment, thereby significantly enhancing the reliability and scientific nature of the proposition splitting results, and providing more expressive local semantic units for subsequent image-text matching.

[0100] (3) Multimodal large model image and text probability scoring method based on prompt word restriction, such as Figure 3 shown.

[0101] This paper proposes a multimodal large-model image-text probability scoring method based on prompt word restrictions. This method fully exploits the potential discrimination ability of the visual language large model (VLM) in the image-text alignment task. Through the structured prompt word design and output space restriction mechanism, the original open-ended generation problem is transformed into a closed-form discrimination problem, thereby improving the discrimination accuracy and robustness of the model in fine-grained image-text matching.

[0102] Specifically, this method introduces a structured prompt word template, feeding an image, a text description (which can be a global or local proposition), and a specially constructed prompt word into a multimodal model. This guides the model to make decisions only within a predefined space of candidate outputs, such as a binary classification of "yes" and "no." Compared to traditional free generation or open-ended question answering, this method establishes explicit semantic constraints within the language generation space through "restrictive prompt words," significantly improving the controllability and interpretability of image-text relevance judgments.

[0103] The logits output by the model only retains the original scores of the two words corresponding to "yes" and "no" (z yes , z no ), and then through the SoftMax operation, the two scores are first exponentialized and then normalized to a probability value with a sum of 1, thus obtaining the image-text pair (I i ,T j ) under the “positive correlation” semantics (i.e., the model chooses “yes”), which reflects the model’s confidence in the degree of semantic matching between the image and text, i.e., the image-text matching score:

[0104]

[0105] This scoring mechanism can not only be used for global semantic matching, but can also be extended to parallel matching tasks of multiple local fine-grained propositions. As mentioned in Example 1, all the fine-grained sub-propositions obtained and a global optimization representation can be combined with each picture and sent to the VLM to obtain the matching probability score of the global-local parallel matching of each picture. Finally, all the matching scores are formed into a probability matrix, and the degree of fusion matching of the image under global and local text is calculated in combination with normalization and weighting strategies, thereby achieving more accurate, interpretable and robust complex text-driven image retrieval tasks.

[0106] For each image I i , combining the matching scores of all local and global information, and taking a weighted sum, we get the final image matching score:

[0107]

[0108] Among them, p( I i,T g ) represents image I i With the global description T g The matching score, Represents image I i With the jth local description The matching score of m is the number of local descriptions.

[0109] Finally, through normalization and weighted calculation, the matching probability of each image and all text information will be summarized and used as the basis for image retrieval, so as to accurately match the image most relevant to the given complex text.

Claims

1. A complex text image retrieval method based on a multimodal large model thinking chain, characterized by: The steps include: S1. Adaptive semantic decomposition: Adaptively decompose the original proposition using a large language model to generate multiple splitting plans; input the multiple splitting plans into the large language model for voting; and select the splitting plan with the highest vote rate as the optimal splitting plan; S2. Reconstruction and optimization: Perform a global rewrite of the original proposition; Inputting the optimal splitting plan and the original proposition into the large language model to generate multiple local descriptor propositions and an optimized global representation; S3. Construct matching probability matrix: The dimension of the matching probability matrix is M×N, where M rows represent M candidate images and N columns represent N matching texts; the matching text includes multiple local descriptor propositions and one optimized global representation; The elements of the matching probability matrix represent probability scores of the matching degree between the candidate image and the matching text; The probability score is obtained by combining the matching text with the candidate image and feeding each pair into a pre-trained visual language model to obtain a corresponding matching score; Then, a binary discriminant scoring mechanism is used to convert the matching score into the probability score that the prompt word is "yes"; S4. Output image retrieval results: According to the matching probability matrix, the matching degree between each candidate image and the original proposition is calculated, and the candidate image with the highest matching degree is selected as the image retrieval result.

2. The complex text image retrieval method based on multimodal large model thinking chain according to claim 1 is characterized in that: In step S4, the matching algorithm is: Normalize each column element in the matching probability matrix to obtain the probability matching score p between the candidate image and the matching text; perform weighted addition of the probability scores by row to obtain the matching degree S(I) of the i-th candidate image. i ): Among them, the weight m represents the number of the local descriptor propositions, I i represents the i-th candidate image, T g represents the optimized global representation, Represents the jth local descriptor proposition.

3. A complex text image retrieval system based on a multimodal large model thinking chain, characterized by: include: Input module, semantic decomposition module, reconstruction optimization module and matching scoring module; The input module inputs the candidate image and the original proposition into the system; The semantic decomposition module performs adaptive semantic decomposition on the original proposition through the large language model to generate multiple decomposition schemes; The semantic decomposition module inputs the multiple splitting schemes into the large language model for voting; the splitting scheme with the highest vote rate is selected as the optimal splitting plan; the reconstruction optimization module globally rewrites the original proposition to generate multiple local description sub-propositions and an optimized global representation; the matching scoring module constructs a matching probability matrix: the elements of the matching probability matrix represent the probability scores of the matching degree between the candidate image and the matching text; The matching scoring module calculates the matching degree between each candidate image and the original proposition according to the matching probability matrix, and selects the candidate image with the highest matching degree as the image retrieval result; The probability score is obtained by combining the matching text with the candidate image and feeding each pair into a pre-trained visual language model to obtain the corresponding matching score; Then, a binary discriminant scoring mechanism is used to convert the matching score into the probability score when the prompt word is "yes"; the complex text image retrieval system based on the multimodal large model thinking chain is used to implement the complex text image retrieval method based on the multimodal large model thinking chain described in claim 1.

Images