Complex text image retrieval method and system based on multi-modal large model thought chain

Abstract

The application discloses a complex text image retrieval method and system based on a multimodal large model thought chain, and the method comprises the following steps: S1, adaptive semantic disintegration; S2, reconstruction and optimization; S3, constructing a matching probability matrix; the elements of the matching probability matrix represent the probability score of the matching degree between the candidate image and the matching text; the matching text and the candidate image are combined, and are sent into a pre-trained visual language model pair by pair to obtain a corresponding matching score; then, a binary discriminant scoring mechanism is used to convert the matching score into a probability score of the prompt word being "yes"; and S4, according to the matching probability matrix, the matching degree of each candidate image and the original proposition is calculated, and the candidate image with the highest matching degree is selected as the image retrieval result. The application aims to extract and optimize semantic features from complex text descriptions by using an adaptive local disintegration, global optimization method and a binary discriminant scoring mechanism guided by a prompt word, so as to improve the accuracy and generalization ability of image retrieval.

Description

Technical Field

[0001] This invention relates to the field of image retrieval, and in particular to a method and system for adaptive complex text image retrieval based on local-global information using multimodal large model thinking chain technology. Background Technology

[0002] In recent years, with the rapid development of deep learning and multimodal technologies, text-to-image retrieval (TIR) ​​has become an important research direction in the fields of information retrieval and computer vision.

[0003] With the advent of the big data era, we are surrounded by massive amounts of image and text information. How to effectively retrieve the content users need from this information has become an urgent problem to solve. Image Retrieval based on Contextual Description (IRCD) aims to extract key and effective information from highly abstract and semantically diverse text descriptions and achieve accurate matching with images. This task is of great significance in many practical scenarios. For example, intelligent search engines can locate specific images through natural language queries, autonomous driving systems can quickly identify traffic scenes through text descriptions, and medical image analysis can accurately locate lesions using text information.

[0004] Traditional image retrieval methods primarily rely on low-level visual features of images, such as color, texture, and shape, to achieve image matching by calculating the similarity between these features. However, this low-level feature-based approach has significant limitations, especially when faced with retrieval needs involving complex text descriptions. It often fails to accurately capture the high-level semantic information expressed by the text. For example, when the user's input search text involves scene context, object relationships, actions, or abstract concepts, retrieval methods relying solely on low-level features struggle to find the semantically most appropriate image. With advancements in deep learning technology, particularly the emergence of visual-language models, it has become possible to handle more complex image retrieval tasks, such as retrieving images from text descriptions containing rich context and subtle visual cues.

[0005] However, the shortcomings of existing methods are that they usually require large-scale labeled data for supervised training of deep learning models, which requires a lot of hardware, computing and time resources. Obtaining high-quality labeled data is also a huge and complicated challenge. Data labeling is expensive and labeled data is often limited in practical applications. Finally, the generalization ability of the model is often limited. For unseen image and text pairs or images with different distributions from the training data, the performance of the model often drops significantly. Summary of the Invention

[0006] In view of the aforementioned shortcomings of existing technologies, this invention provides a method and system for complex text image retrieval based on a multimodal large-scale model thinking chain. It aims to extract and optimize semantic features from complex text descriptions using adaptive local deconstruction, global optimization methods, and a prompt-word-guided binary discriminant scoring mechanism, thereby improving the accuracy and generalization ability of image retrieval. Through hierarchical parsing of text information, this method effectively enhances cross-modal matching capabilities without requiring a series of additional fine-tuning training steps, thus achieving efficient image retrieval applications.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] Firstly, a complex text and image retrieval method based on a multimodal large-scale model thinking chain includes the following steps:

[0009] S1. Adaptive semantic decomposition:

[0010] The original proposition is adaptively decomposed using a large language model to generate multiple decomposition schemes; the multiple decomposition schemes are input into the large language model for voting; the decomposition scheme with the highest vote rate is selected as the optimal decomposition plan.

[0011] S2, Refactoring and Optimization:

[0012] The original proposition is rewritten globally;

[0013] The optimal splitting plan and the original proposition are input into the large language model to generate multiple local descriptive sub-propositions and an optimized global representation.

[0014] S3. Construct the matching probability matrix:

[0015] The matching probability matrix has a dimension of M×N, where M rows represent M candidate images and N columns represent N matching texts; the matching texts contain multiple local descriptor propositions and an optimized global representation;

[0016] The elements of the matching probability matrix represent probability scores of the degree of matching between the candidate image and the matching text;

[0017] The probability score is obtained by combining the matching text with the candidate image and feeding them into a pre-trained visual language model one by one to obtain the corresponding matching score; then, a binary discriminant scoring mechanism is used to convert the matching score into the probability score of the prompt word "yes".

[0018] S4. Output image retrieval results:

[0019] Based on the matching probability matrix, the matching degree between each candidate image and the original proposition is calculated, and the candidate image with the highest matching degree is selected as the image retrieval result.

[0020] Preferably, in step S4, the algorithm for the matching degree is as follows:

[0021] Normalize the elements in each column of the matching probability matrix to obtain the probability matching score p between the candidate image and the matching text; sum the probability scores row by row to obtain the matching degree S(I) of the i-th candidate image. i ):

[0022]

[0023] Where, weight m represents the number of the local descriptor propositions, I i T represents the i-th candidate image. g This represents the optimized global representation. Let j represent the j-th local descriptor proposition.

[0024] Secondly, a complex text image retrieval system based on a multimodal large-scale model thinking chain includes: an input module, a semantic decomposition module, a reconstruction and optimization module, and a matching and scoring module. The input module inputs the candidate image and the original proposition into the system. The semantic decomposition module adaptively decomposes the original proposition using the large-scale language model, generating multiple decomposition schemes. The semantic decomposition module inputs the multiple decomposition schemes into the large-scale language model for voting; the decomposition scheme with the highest vote rate is selected as the optimal decomposition plan. The reconstruction and optimization module globally rewrites the original proposition, generating multiple local descriptive sub-propositions and an optimized global representation. The matching and scoring module constructs a matching probability matrix: the elements of the matching probability matrix represent the probability scores of the matching degree between the candidate image and the matching text. The matching and scoring module calculates the matching degree between each candidate image and the original proposition based on the matching probability matrix, and selects the candidate image with the highest matching degree as the image retrieval result.

[0025] The probability score is obtained by combining the matching text with the candidate image and feeding them into a pre-trained visual language model to obtain the corresponding matching score; then, a binary discriminant scoring mechanism is used to convert the matching score into the probability score of the prompt word "yes".

[0026] Compared with the prior art, the beneficial effects of the present invention are reflected in:

[0027] 1. This method first leverages the reasoning capabilities of a large language model to generate multi-faceted decomposition plans based on the characteristics of different complex descriptions. A self-consistency mechanism is introduced to cross-vote among these plans, ultimately selecting the optimal decomposition scheme to generate several semantically clear and structurally concise sub-descriptions. Simultaneously, utilizing the powerful semantic construction and optimization capabilities of the language model, the original description is structurally reorganized and grammatical errors corrected, further improving text quality and semantic clarity. Through the method proposed in this invention, original semantically complex text descriptions can be effectively decomposed and optimized for rewriting, thereby significantly enhancing the visual language model's ability to understand text content and improving the accuracy and robustness of image-text matching.

[0028] 2. In the image-text matching stage, a carefully designed prompt word template is used to guide the visual language model to generate controlled binary output, which is then used to measure the degree of matching between the image and the text. By constructing an image-text matching probability score matrix, a global score can be given for the matching of each image in the image candidate set with complex semantics, ultimately achieving accurate retrieval of the image that best matches the semantic intent. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the method in Example 1;

[0030] Figure 2 This is a schematic diagram of the adaptive proposition splitter in Example 2;

[0031] Figure 3 This is a schematic diagram of the multimodal large model image-text probability scoring based on prompt word restrictions in Example 2. Detailed Implementation

[0032] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further described below with reference to specific illustrations. However, the invention is not limited to the embodiments described below.

[0033] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0034] Example 1:

[0035] like Figure 1 The method for complex text and image retrieval based on a multimodal large model thinking chain, as shown, includes the following steps:

[0036] S1. Adaptive semantic decomposition:

[0037] The original proposition is adaptively decomposed using a large language model to generate multiple decomposition schemes; these schemes are then input into the large language model for voting; the scheme with the highest vote rate is selected as the optimal decomposition plan.

[0038] To address complex natural language input descriptions, an adaptive proposition decomposition mechanism based on Large Language Model (LLM) is constructed. This mechanism first performs deep semantic modeling and deconstruction of the original proposition description, then breaks it down into n independent sub-descriptions that focus more on detailed information. During this process, the model generates initial decomposition schemes through multi-perspective understanding and iteratively enhances them using the self-consistency mechanism of the language model. Specifically, it performs a cross-voting strategy on multiple decomposition versions, ultimately selecting the scheme with the highest vote rate as the optimal fine-grained decomposition result, i.e., the optimal decomposition plan. This step leverages the powerful semantic understanding and parsing capabilities of LLM, enabling adaptive understanding and comprehensive multi-perspective decomposition of complex descriptions across different domains without requiring manual pre-setting of specific decomposition rules. This significantly improves the interpretability and adaptability of the original text in cross-modal retrieval tasks.

[0039] S2, Refactoring and Optimization:

[0040] The original proposition is rewritten globally.

[0041] The optimal splitting plan and the original proposition are input into the large language model to generate multiple local descriptive sub-propositions and an optimized global representation.

[0042] To further refine the semantic core of complex propositions and enhance their logical coherence, an optimization strategy combining syntactic rewriting and semantic reconstruction is proposed. This strategy leverages the powerful language modeling capabilities of a large language model to globally rewrite the original complex propositions in a more accurate and complete manner without altering the original semantics. This eliminates potential grammatical errors such as missing or mixed components in the original description, improving text clarity and processability. Based on this, the optimal splitting plan and the globally rewritten original propositions are input into the large language model to generate a final set of local information with higher semantic consistency and a more rational structure. Ultimately, two outputs are obtained for each complex original proposition: a set of fine-grained local descriptive sub-propositions and a semantically optimized global representation.

[0043] S3. Construct the matching probability matrix:

[0044] The matching probability matrix has a dimension of M×N, where M rows represent M candidate images and N columns represent the sum of N local descriptor propositions and optimized global representations. The element values ​​of the matching probability matrix represent the probability score of the degree of matching between a candidate image and an optimized global representation or a local descriptor proposition. The element values ​​are obtained by combining the optimized global representation and the local descriptor proposition with the candidate image respectively, and feeding them into the pre-trained visual language model one pair at a time to obtain the corresponding matching scores. Then, a binary discriminant scoring mechanism is used to convert the matching scores into probability scores for the prompt word "yes".

[0045] The optimized global representation and its decomposed local descriptive propositions are combined with candidate images, and then fed into a pre-trained Visual Language Model (VLM) pairwise. The VLM model embeds the image and text into a unified feature space, models the image-text relationship through a cross-attention mechanism, and outputs a matching score to measure the semantic fit between the text proposition (optimized global representation and local descriptive proposition) and the candidate image. To overcome the scoring compression problem caused by high semantic similarity between images, a binary discriminative scoring mechanism guided by prompt words is designed:

[0046] Since most candidate images are very similar, directly scoring the image-text matching degree using the VLM model would lead to inconsistent score discrimination. Therefore, a specially designed cue word is first used to strictly limit the output of the multimodal model to a binary decision of "yes" or "no". in Image I i And the processed optimized global representation and local descriptor subproposition T p The VLM model is input as a common input, and the model output logits (the unnormalized raw output scores of the model) are obtained, z:

[0047] z = [z1, z2, ..., z k ] = VLM(P in ,I i ,T p )

[0048] Subsequently, only the logit values ​​(z) corresponding to "yes" and "no" in the output logits of the open-source visual language large model were extracted. yes , z no The two values ​​are then normalized using a SoftMax operation, transforming them into two probability scores that sum to 1. The probability score p representing "yes" is then selected. yes As a model to measure text-image pairs (I) i ,T j The probability score s(I) of the image-text matching degree. i ,T j ), s(I i ,Tj ) is represented as:

[0049]

[0050] Reasoning is performed on all local descriptor propositions and optimized global representations combined with candidate images to obtain a matching probability matrix of dimension M×N. M rows correspond to M candidate images, and N columns correspond to N text inputs, including N-1 fine-grained local descriptor propositions and one semantically optimized global representation. This matrix fully captures the multi-level matching relationships between images and complex semantics, providing detailed support for subsequent image retrieval results.

[0051] S4. Output image retrieval results:

[0052] Based on the matching probability matrix, the matching degree between each candidate image and the original proposition is calculated, and the candidate image with the highest matching degree is selected as the image retrieval result.

[0053] Based on the obtained matching probability matrix, the matching degree between each candidate image and the original proposition is calculated, and the candidate image with the highest score is taken as the image retrieval result.

[0054] Specifically, normalizing each column element in the matching probability matrix yields image I. i With processing text T j The probability matching score p(I) I ,T j This is to eliminate the impact of extreme outlier scores on the final search results.

[0055]

[0056] The probability scores of each row in the matching probability matrix are then weighted and summed to determine the image-text matching degree between a single candidate image and the complex text, where the weight m is the number of simple propositions after splitting, and I i Let T represent the i-th candidate image. g This indicates an optimization of the global representation. Let S(I) represent the j-th local descriptor proposition extracted from the split, and the matching degree S(I) of the i-th candidate image. i ) is represented as:

[0057]

[0058] Example 2:

[0059] This embodiment provides an image retrieval system. Its input consists of a set of structurally similar images and a semantically complex natural language description. The system outputs the image that best matches the semantics of the description. The specific process of this system will be described below:

[0060] S1. The input module inputs the candidate image and the original proposition into the system:

[0061] Ten candidate images with highly similar visual content are provided. The images contain multiple male figures and background elements such as plants. The image layouts are similar to each other, making semantic differentiation difficult.

[0062] The original proposition to be retrieved (complex text description) is: The man below is standing to the right rear of the plant in the lower left corner; his legs are not fully visible. He is at the top of the image; the left image looks forward, and the right image looks to the left. This invention believes that such grammatically incorrect or ambiguous queries should be taken into account, so this embodiment uses an example that might mislead a large model.

[0063] S2. The semantic decomposition module performs adaptive semantic decomposition on the original proposition:

[0064] The semantic decomposition module includes an adaptive proposition splitter. This splitter utilizes a pre-built large language model (such as the ChatGLM series) to automatically deconstruct the complex text description. By invoking the chain-like thinking mechanism of the large language model, it generates multiple versions of candidate splitting plans. Based on the self-consistency principle of the large language model, it performs a consistency vote on these multiple plans, and finally uses the optimal splitting plan to decompose the following fine-grained local sub-propositions:

[0065] (1) The man in the lower left corner is standing behind the potted plant. (2) We cannot see the man's legs in the lower left corner.

[0066] (3) The man in the upper right is looking to the left. (4) The man in the upper left is looking forward.

[0067] S3. The reconstruction and optimization module performs semantic structure reconstruction and fusion optimization on the optimal splitting plan:

[0068] The system guides the large language model to maintain the original semantic meaning of the description while optimizing and correcting potential grammatical errors such as semantic mixing and missing components that may affect model matching, through prompt words. To achieve this goal, we designed the following prompt words for rewriting and optimizing the original description:

[0069] You are a professional text optimization assistant, skilled at rewriting text with expression problems to make it clearer. Please help me complete the following tasks: Analyze user-input sentences / paragraphs, identify potential issues such as: grammatical errors or grammatical mistakes, logical ambiguities or unclear referents, overly complex nested structures, and a mixture of technical jargon and colloquialisms. Rewrite the text while preserving the original meaning, requiring: breaking down long and complex sentences into shorter ones, clarifying ambiguous referential relationships, reorganizing sentences with more standardized grammatical structures, and improving readability while retaining key information.

[0070] The original description mentioned in S1 and the carefully selected prompts above are input into a large language model for semantic rewriting and global structural optimization, generating the following global semantic optimization proposition:

[0071] "In the upper half of the image, the man on the left is looking forward, while the man on the right is looking to the left. In the lower half of the image, the man is standing to the right of the potted plant, and his legs are not fully visible."

[0072] The optimized text has greater linguistic clarity, semantic coherence, and interpretability compared to the original text.

[0073] S4. The matching scoring module constructs a matching probability matrix through a multimodal matching scoring mechanism.

[0074] For each of the aforementioned local descriptive propositions and optimized global representations, they are paired with candidate images and sequentially input into a pre-trained visual language model (such as LLaVA, BLIP-2, etc.). The pre-trained visual language model uses the prompt "Does the content of this image match the description?" as an input prefix, transforming the task into a binary judgment task (yes / no), and returns the original output probability values ​​for "yes" and "no". After Softmax normalization, the probability score of each image-text matching pair is obtained.

[0075] For example, the output after a candidate image is matched with the local descriptor proposition "The man in the upper right is looking to the left" is:

[0076] z yes =3.2,z no =1.1

[0077]

[0078] The system performs the above matching process on each candidate image with all local descriptor propositions and optimized global representations, constructing a 10×5 probability matrix (4 local descriptor propositions + 1 optimized global representation). This matrix reflects the multi-level matching degree of each candidate image at both the fine-grained and global semantic levels.

[0079] S5. The matching scoring module calculates the matching degree and determines the search results:

[0080] The matching and scoring module calculates the matching degree between each candidate image and the original proposition, and selects the candidate image with the highest matching degree as the image retrieval result. Normalization is performed on each column of the matching probability matrix to eliminate the interference of excessively high scores for individual propositions on the overall result. Then, the weighted matching degree score for each row is calculated, using the following matching degree function:

[0081]

[0082] Where m = 4, representing the number of sub-propositions. The matching scoring module returns the image with the highest matching degree as the search result.

[0083] Analysis of search results:

[0084] In practical application, this invention successfully decomposes semantically ambiguous complex text into multiple highly discriminative sub-propositions and, combined with global semantic optimization, achieves multimodal deep understanding and high-precision image-text matching. Experiments show that, compared with retrieval methods that do not perform semantic decomposition and optimization, the solution of this invention exhibits stronger discriminative ability and higher retrieval accuracy in scenarios with highly similar image content.

[0085] The present invention will be further described below with reference to the accompanying drawings:

[0086] The overall process of this invention is as follows: Figure 1 As shown.

[0087] This invention designs an adaptive proposition splitter to achieve a three-level process of "planning-voting-splitting" for complex original semantic propositions. First, based on the multi-faceted understanding capabilities of a large language model, multiple splitting schemes are generated for complex text descriptions. A self-consistency mechanism is introduced to conduct multiple rounds of voting to select the optimal splitting scheme, thereby obtaining several semantically focused and structurally concise local propositions. Second, combining the language model's deep semantic modeling capabilities, the original propositions are rewritten and optimized to remove redundant components and correct grammatical errors, generating a clear and structurally sound global semantic representation.

[0088] In the matching phase, the obtained local propositions and optimized global propositions are combined with image samples from the image candidate set and input into the visual language model pair by pair. By designing prompt word templates with binary output constraints, the model is guided to perform probability scoring on image-text matching relationships. Subsequently, an image-text matching score matrix is ​​constructed, and the matrix is ​​normalized and weighted to calculate the overall matching score for each image under multi-dimensional semantic guidance. Finally, the image with the highest score is selected as the retrieval result image that best matches the input complex semantic description.

[0089] (2) Adaptive proposition splitter, such as Figure 2 As shown.

[0090] In image retrieval tasks involving complex text, the original propositions often contain a wealth of detailed descriptions and latent semantics, such as spatial orientation, positional relationships, action states, and attribute features. This fine-grained local information plays a crucial role in achieving high-precision image retrieval. Therefore, to address this challenge, this invention proposes an adaptive proposition splitter based on the principle of large language model consistency. This splitter aims to deeply analyze the multi-level semantic structure of complex propositions and enhance the accuracy and discriminative power of image-text matching through fine-grained splitting and optimization. Its workflow is as follows:

[0091] First, leveraging the powerful language understanding capabilities of Large Language Models (LLM), multiple semantic segmentation plans are generated based on the input complex proposition. Each segmentation plan starts from a different semantic dimension, striving to capture the multi-layered details and potential implicit logic within the proposition. For example, for propositions describing space, some segmentation plans focus on capturing the spatial relationships between objects, i.e., determining the position, orientation, and relative relationships of different objects; while other segmentation plans may focus more on action or state features, such as describing the dynamic changes or visual states of objects. Through this multi-angle, multi-dimensional segmentation strategy, this invention can effectively cover all potential semantic levels in complex propositions, thereby improving the semantic expressiveness of complex propositions.

[0092] Let the original proposition be C. Then, using the semantic understanding capabilities of a Large Language Model (LLM), the generated splitting schemes P1, P2, ..., P... n It can be represented as:

[0093]

[0094] Where, θ i Indicates the split plan P i Different semantic dimensions are used in the text.

[0095] To further ensure the accuracy and internal consistency of the final splitting scheme, this invention introduces a voting strategy for selecting the splitting plan based on the majority consensus mechanism of a large language model. Specifically, the multiple splitting schemes mentioned above are re-inputted into the large language model for multiple rounds of judgment and voting. By statistically analyzing the selection frequency of different schemes in the model's judgment, the splitting plan with the highest vote rate is selected as the final execution scheme.

[0096] Let P i Indicates the split plan P i The final optimal splitting scheme P is selected. * It can be calculated in the following way:

[0097]

[0098] Where m represents the voting round, The splitting scheme P is in the j-th round. i The number of votes received.

[0099] This strategy fully leverages the stability and consistency of large models in semantic judgment, thereby significantly enhancing the reliability and scientific rigor of proposition splitting results and providing more expressive local semantic units for subsequent image-text matching.

[0100] (3) A multimodal large-scale image-text probability scoring method based on prompt word restrictions, such as Figure 3 As shown.

[0101] This invention proposes a multimodal large model image-text probability scoring method based on cue word constraints. This method fully explores the potential discriminative ability of the Visual Language Model (VLM) in image-text alignment tasks. Through structured cue word design and output space constraint mechanism, the original open generation problem is transformed into a closed discrimination problem, thereby improving the discrimination accuracy and robustness of the model in fine-grained image-text matching.

[0102] Specifically, this method introduces a structured cue word template, inputting images, text descriptions (which can be global or local propositions), and specifically constructed cue words into a multimodal model. The model is then guided to make decisions only within a predefined candidate output space, such as binary classification outputs of "yes" and "no." Compared to traditional free-generation or open-ended question answering, this method constructs explicit semantic constraints in the language generation space through "restricted cue words," thereby significantly improving the controllability and interpretability of image-text relevance judgment.

[0103] The model output logits only retains the original scores (z) of the two tokens corresponding to "yes" and "no". yes , z no Then, through the SoftMax operation, the two scores are first exponentially calculated, and then normalized to a probability value that sums to 1, thus obtaining the image-text pair (I). i ,T j The probability score under the semantic condition of "positive correlation" (i.e., the model selects "yes") reflects the model's confidence in judging the degree of semantic matching between the image and text, i.e., the image-text matching score:

[0104]

[0105] This scoring mechanism can be used not only for global semantic matching but also extended to parallel matching tasks involving multiple local fine-grained propositions. As mentioned in Example 1, all the obtained fine-grained sub-propositions and a global optimized representation can be paired with each image and fed into the VLM to obtain the matching probability score for global-local parallel matching of each image. Finally, all matching scores are formed into a probability matrix, and the degree of fusion matching of the image under global and local text is calculated by combining normalization and weighting strategies, thereby achieving a more accurate, interpretable, and robust complex text-driven image retrieval task.

[0106] For each image I i By combining the matching scores of all local and global information, and through a weighted summation, the final image matching score is obtained:

[0107]

[0108] Where, p( I i,T g ) represents image I i With global description T g Match score, Image I i With the j-th local description The matching score, where m is the number of local descriptions.

[0109] Finally, through normalization and weighted calculation, the matching probability of each image with all text information is summarized, which serves as the basis for image retrieval, thereby accurately matching the image most relevant to a given complex text.

Claims

1. A complex text image retrieval method based on a multimodal large-scale model thinking chain, characterized in that, The steps include the following: S1. Adaptive semantic decomposition: The original proposition is adaptively decomposed using a large language model to generate multiple decomposition schemes; the multiple decomposition schemes are input into the large language model for voting; the decomposition scheme with the highest vote rate is selected as the optimal decomposition plan. S2, Refactoring and Optimization: The original proposition is rewritten globally; The optimal splitting plan and the original proposition are input into the large language model to generate multiple local descriptive sub-propositions and an optimized global representation. S3. Construct the matching probability matrix: The matching probability matrix has a dimension of M×N, where M rows represent M candidate images and N columns represent N matching texts; the matching texts contain multiple local descriptor propositions and an optimized global representation; The elements of the matching probability matrix represent probability scores of the degree of matching between the candidate image and the matching text; The probability score is obtained by combining the matching text with the candidate image and feeding them into a pre-trained visual language model one by one to obtain the corresponding matching score. Then, a binary discriminant scoring mechanism is used to convert the matching score into the probability score of the prompt word "yes"; S4. Output image retrieval results: Based on the matching probability matrix, the matching degree between each candidate image and the original proposition is calculated, and the candidate image with the highest matching degree is selected as the image retrieval result. The algorithm for the matching degree is as follows: Normalizing each column element in the matching probability matrix yields the probability matching score between the candidate image and the matching text. The probability scores are weighted and summed row by row to obtain the matching degree of the i-th candidate image. : ； Wherein, the weight m represents the number of the local descriptor propositions. This represents the i-th candidate image. This represents the optimized global representation. Let j represent the j-th local descriptor proposition.

2. A complex text and image retrieval system based on a multimodal large-scale model thinking chain, characterized in that, include: The module consists of an input module, a semantic decomposition module, a reconstruction and optimization module, and a matching and scoring module. The input module inputs the candidate image and the original proposition into the system; The semantic decomposition module performs adaptive semantic decomposition of the original proposition through the large language model, generating multiple decomposition schemes. The semantic decomposition module inputs multiple decomposition schemes into the large language model for voting; the decomposition scheme with the highest vote rate is selected as the optimal decomposition plan; the reconstruction and optimization module globally rewrites the original proposition, generating multiple local descriptive sub-propositions and an optimized global representation; the matching scoring module constructs a matching probability matrix: the elements of the matching probability matrix represent the probability score of the degree of matching between the candidate image and the matching text; The matching scoring module calculates the matching degree between each candidate image and the original proposition based on the matching probability matrix, and selects the candidate image with the highest matching degree as the image retrieval result. The probability score is obtained by combining the matching text with the candidate image and feeding them into a pre-trained visual language model one by one to obtain the corresponding matching score. Subsequently, a binary discriminant scoring mechanism is used to convert the matching score into the probability score of the prompt word "yes"; the complex text image retrieval system based on multimodal large model thinking chain is used to implement the complex text image retrieval method based on multimodal large model thinking chain as described in claim 1.

Images