Image-text matching analysis method, system and device based on large model and multiple modes

Abstract

The invention relates to an image-text matching analysis method, system and device based on a large model and multiple modes. The method comprises the following steps: 1) image-text analysis: analyzing an image into an image and a text; 2) carrying out fine-tuning multi-mode large model processing on the image, and carrying out word segmentation processing on the characters; 3) constructing a bisection weight; 4) performing Few-shot cue word construction on the processed characters and the constructed equally-divided weight; 5) performing training / fine tuning on the characters and the pictures in the step 4 and the step 2 to serve as a large oracle model; according to the multi-mode image-text matching method, cooperative work of the multi-mode large model and the large language model is introduced, the advantages of the multi-mode large model and the large language model in semantic comprehension, context analysis and dynamic adjustment capacity are comprehensively utilized, a comprehensive image-text matching framework is established, and the accuracy, flexibility and interpretability of matching can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and specifically to a large model and multimodal image-text matching analysis method, system and device. Background Art

[0002] At present, multimodal image-text matching technology has been widely used in the fields of image understanding, information retrieval, and natural language processing. Traditional image-text matching methods usually rely on a specific convolutional neural network (CNN) to extract features from images, combine recurrent neural networks (RNN) or attention mechanisms to encode text, and then judge the relevance of images and texts through feature alignment or similarity calculations; there are also methods that use OCR character recognition technology to identify and match text in images. For example, the two-stream neural network method based on the image-text embedding space maps the image and text to the same feature space respectively, and then calculates feature similarity. However, this type of method has the following shortcomings: 1. Limited semantic understanding capabilities: Traditional methods for semantic understanding of images and texts rely on predefined feature extractors or numbers and texts present in images, and cannot flexibly capture complex multimodal semantic relationships.

[0003] 2. Insufficient scene generalization ability: When processing images and texts across domains or complex scenes, existing technologies often find it difficult to achieve satisfactory matching results.

[0004] 3. Lack of contextual analysis: For complex image-text matching tasks, existing methods find it difficult to fully utilize contextual information for deep semantic association analysis.

[0005] 4. Single evaluation index: Most methods are based on simple similarity scores or classification model outputs and cannot dynamically adjust the scoring criteria to adapt to different application scenarios.

[0006] Therefore, a large model and multimodal image-text matching analysis method, system and device are designed to overcome the above problems. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method, system and device for image-text matching analysis based on a large model and multimodality. The present invention uses a large multimodal model to perform semantic understanding and text generation on the image content, and combines the few-shot learning ability of a large language model to perform similarity scoring on the given text and the generated image description text, thereby achieving more accurate, intelligent and interpretable image-text matching, which can improve the accuracy, flexibility and interpretability of matching.

[0008] The present invention is implemented by the following technical solution: a large model and multi-modal image-text matching analysis method, the method comprising the following steps: 1) Image and text analysis: parse the image and text into images and text; 2) Fine-tune the multimodal large model processing for images and perform word segmentation processing for text; 3) Constructing equal weights; 4) The processed text is equally weighted with the construction to construct the few-shot prompt words; 5) Train / fine-tune the text and pictures in step 4 and step 2 as a large prediction model; 6) Similarity score; 7) Analyze the ratings.

[0009] Preferably, the image and text analysis in step 1) is divided into image analysis and text analysis, wherein the image processing method is: uniformly processing the input image, including resizing (such as fixing to 224×224 pixels), normalization, and color channel standardization (such as RGB format); the text processing method is: segmenting the prompt words, removing stop words, cleaning special characters, etc., and supports multi-language processing.

[0010] As a preferred embodiment, the specific method of constructing the equal weight in step 2) is: evaluating through multiple dimensions, which are composed of the following: 1. Content accuracyj: the degree of match between the description and the core content of the input text; 2. Description completeness k: whether the generated image description covers the key elements in the image; 3. Language style consistency u: whether the language style of the input text is consistent with that of the generated text; 4. Detail matching v: the degree of consistency between the image and the text in detail description; The score range of each scoring dimension is 0 to 1, and the final comprehensive score can be calculated by the preset weights. The formula is as follows: Overall score = j × content accuracy + k × description completeness + u × language style consistency + v × detail matching.

[0011] Preferably, the specific method in step 5) is: receiving input images and text prompt words, understanding the semantics, scenes and contents in the image through a fine-tuned multimodal large model, and generating a text description that is highly relevant to the image content.

[0012] As a preferred embodiment, the specific method of similarity scoring in step 6) is: utilizing the few-shot learning capability of the large language model to construct a prompt template containing multiple sets of known image-text matching pairs, providing context examples for the similarity scoring task, the large language model receives the multimodally generated image description text and the given text input by the user, performs semantic comparative analysis based on the context examples, and outputs a similarity score between 0 and 1.

[0013] Preferably, the method for explaining the rating in step 7) is as follows: while the system generates the rating, it provides a semantic explanation of the rating, indicating the degree of match between the text and the image description in terms of content, style or semantics, and helps users understand the basis for the rating through explanations in natural language.

[0014] A large model and multimodal image-text matching analysis system, the system comprising: Image and text parsing module: This module performs preprocessing, which includes image processing and text processing, where the image is uniformly processed, including resizing, fixed to 224×224 pixels, normalization, and color channel standardization format; Text processing: segment prompt words, remove stop words, clean up special characters, and support multi-language processing; Construct a scoring weight module: The scoring dimension module evaluates the generated text and input prompt words from multiple dimensions; Construct a few-shot prompt word module: Design a few-shot prompt word template for relevant scenario tasks; Multimodal semantic understanding module: receives input images and text prompts, understands the semantics, scenes, and content in the image through a fine-tuned multimodal large model, and generates a text description that is highly relevant to the image content; Similarity scoring module: Utilizes the few-shot learning capability of the large language model to construct a prompt template containing multiple sets of known image-text matching pairs, providing contextual examples for the similarity scoring task; Rating explanation module: While the system generates ratings, it also provides semantic explanations of the ratings; The scoring weight construction module is also provided with a dynamic scoring adjustment mechanism: the dynamic scoring adjustment mechanism flexibly adjusts the weight of each scoring dimension according to different application scenarios.

[0015] A large model and multimodal image-text matching analysis device, which is used for image-text matching in the education field or content review in industrial scenarios.

[0016] The beneficial effects of the present invention are as follows: Compared with the existing image-text matching analysis system, the image-text matching analysis system based on a large model and multimodality provided by the present invention has the following beneficial effects: 1. Significantly improved semantic understanding capabilities: The system can accurately understand the details and semantics in images and achieve more precise text matching through the generated image description text.

[0017] 2. Enhanced scene adaptability: It is not limited to specific fields or scenes and is applicable to a variety of complex application environments, such as image-text matching tasks in education, industry, advertising and other fields.

[0018] 3. The scoring mechanism is flexible and explainable: Compared with the single score of traditional methods, the similarity score generated by the present invention adopts a large language model, which can dynamically adapt to different task requirements and provide detailed scoring explanations.

[0019] 4. Balancing efficiency and accuracy: Through the efficient reasoning capabilities of large models, the system achieves high computing efficiency while ensuring scoring accuracy.

[0020] 5. Significant application value: The present invention is applicable to a variety of scenarios such as image and text retrieval, content review, and auxiliary teaching, providing users with a more intelligent and accurate image and text matching solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a system framework diagram of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to more clearly understand the objectives, technical solutions and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0023] In the description of the present invention, it is necessary to understand that the orientation or position relationship indicated by the terms such as "up", "down", "left", "right", "inside", "outside", "horizontal" and "vertical" is based on the orientation or position relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention, rather than indicating or implying that the device or original referred to must have a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0024] The present invention will be described in detail below with reference to the accompanying drawings: Figure 1 As shown, a large model and multi-modal image-text matching analysis method is provided, and the method comprises the following steps: 1) Image and text analysis: parse the image and text into images and text; 2) Fine-tune the multimodal large model processing for images and perform word segmentation processing for text; 3) Constructing equal weights; 4) The processed text is equally weighted with the construction to construct the few-shot prompt words; 5) Train / fine-tune the text and pictures in step 4 and step 2 as a large prediction model; 6) Similarity score; 7) Analyze the ratings.

[0025] Preferably, the image and text analysis in step 1) is divided into picture analysis and text analysis, wherein the image processing method is: uniformly processing the input image, including resizing, normalization, and color channel standardization, and the text processing method is: segmenting the prompt words, removing stop words, cleaning special characters, etc., while supporting multi-language processing.

[0026] As a preferred embodiment, the specific method of constructing the equal weight in step 2) is: evaluating through multiple dimensions, which are composed of the following: 1. Content accuracyj: the degree of match between the description and the core content of the input text; 2. Description completeness k: whether the generated image description covers the key elements in the image; 3. Language style consistency u: whether the language style of the input text is consistent with that of the generated text; 4. Detail matching v: the degree of consistency between the image and the text in detail description; The score range of each scoring dimension is 0 to 1, and the final comprehensive score can be calculated by the preset weights. The formula is as follows: Overall score = j × content accuracy + k × description completeness + u × language style consistency + v × detail matching.

[0027] Preferably, the specific method in step 5) is: receiving input images and text prompt words, understanding the semantics, scenes and contents in the image through a fine-tuned multimodal large model, and generating a text description that is highly relevant to the image content.

[0028] As a preferred embodiment, the specific method of similarity scoring in step 6) is: utilizing the few-shot learning capability of the large language model to construct a prompt template containing multiple sets of known image-text matching pairs, providing context examples for the similarity scoring task, the large language model receives the multimodally generated image description text and the given text input by the user, performs semantic comparative analysis based on the context examples, and outputs a similarity score between 0 and 1.

[0029] Preferably, the method for explaining the rating in step 7) is as follows: while the system generates the rating, it provides a semantic explanation of the rating, indicating the degree of match between the text and the image description in terms of content, style or semantics, and helps users understand the basis for the rating through explanations in natural language.

[0030] A large model and multimodal image-text matching analysis system, the system comprising: Image and text parsing module: This module performs preprocessing, which includes image processing and text processing, where the image is uniformly processed, including resizing, fixed to 224×224 pixels, normalization, and color channel standardization format; Text processing: segment prompt words, remove stop words, clean up special characters, and support multi-language processing; Construct a scoring weight module: The scoring dimension module evaluates the generated text and input prompt words from multiple dimensions; Construct a few-shot prompt word module: Design a few-shot prompt word template for relevant scenario tasks; For example, given the reference template: “Given text: If garbage sorting is done well, waste can also become treasure.” Multimodal generated image description text: The picture shows four stainless steel trash cans, arranged together, corresponding to recyclables, hazardous waste, kitchen waste and other waste. There is a corresponding label on the right side of each trash can, and the label includes the corresponding type of waste. The surface of the trash can is smooth and bright, without obvious scratches or stains.

[0031] Similarity Rating: Content accuracy j: 0.40 Description completeness k: 0.20 Language style consistency u: 0.10 Detail matching v: 0.30 Comprehensive score 0.92 = 0.40 × 0.90 + 0.20 × 0.90 + 0.10 × 0.95 + 0.30 × 0.95 Explanation: The image and text are highly relevant in terms of theme, content, and detail, with only minor differences in the consistency of the language style. " In this way, multiple groups of related prompt word templates are constructed, and the large language model learns to score and explain them.

[0032] Multimodal semantic understanding module: receives input images and text prompts, understands the semantics, scenes, and content in the image through a fine-tuned multimodal large model, and generates a text description that is highly relevant to the image content; Similarity scoring module: Using the few-shot learning capability of the large language model, a prompt template containing multiple sets of known image-text matching pairs is constructed to provide context examples for the similarity scoring task; the large language model receives multimodally generated image description text and given text input by the user, performs semantic comparison analysis based on the context examples, and outputs a similarity score between 0 and 1. If it is specialized knowledge in a niche field, it is necessary to use professional knowledge to fine-tune the large language model.

[0033] Rating explanation module: When the system generates a rating, it also provides a semantic explanation of the rating, such as indicating the degree of match between the text and the image description in terms of content, style, or semantics. This helps users understand the basis for the rating through explanations in natural language.

[0034] The scoring weight construction module is also provided with a dynamic scoring adjustment mechanism: the dynamic scoring adjustment mechanism flexibly adjusts the weight of each scoring dimension according to different application scenarios.

[0035] For example: 1. Educational scenario: Increase the weight of “description completeness” to ensure that the generated text covers all key information in the image.

[0036] 2. News scenario: Increase the weight of “content accuracy” and emphasize the topic accuracy and core matching of the description.

[0037] 3. Creative design scenario: Increase the weight of "language style consistency" to ensure that the generated text is consistent with the input text in expression style.

[0038] A large model and multimodal image-text matching analysis device, which is used for image-text matching in the education field or content review in industrial scenarios.

[0039] Example 1 Image-text matching applications in education In educational scenarios, users need to match courseware images with teaching texts. Content accuracy and description completeness are more important, so these dimensions are given higher weights: 1. Content accuracy j: 0.5 2. Description completeness k: 0.3 3. Language style consistency u: 0.2 4. Detail matching v: 0.0 (ignore irrelevant details) Practical Application: Given text: "The cell wall and nucleus were observed under a microscope" Multimodal generated image description text: "Schematic diagram of cell structure observed under a microscope" Similarity Rating: Content accuracy j: 0.8 Description completeness k: 0.7 Language style consistency u: 0.9 Comprehensive score 0.79 = 0.5×0.8+0.3×0.7+0.2×0.9 Explanation: “The image and text are highly matched in core semantics, but the image lacks specific annotations of cell names.” Example 2: Content review application in industrial scenarios In industrial content review scenarios, for example, it is used to review the matching degree between advertising images and promotional copy, with more emphasis on detail matching and consistency of language style: 1. Content accuracy j: 0.2 2. Description completeness k: 0.1 3. Language style consistency u: 0.4 4. Detail matching v: 0.3 Practical Application: Given text: "This is a new silver-gray refrigerator of a certain brand with voice control technology" Multimodal generated image description text: "Showing the voice control function of the silver gray refrigerator" Similarity Rating: Content accuracy j: 0.9 Description completeness k: 0.6 Language style consistency u: 0.8 Detail matching v: 0.7 Comprehensive score 0.77 = 0.2×0.9+0.1×0.6+0.4×0.8+0.3×0.7 Explanation: “The image content and the copy are exactly the same in terms of functional description, but it does not contain brand information.” Anyone familiar with the technology may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A large model and multi-modal image-text matching analysis method, characterized in that: The method comprises the following steps: 1) Image and text analysis: parse the image and text into images and text; 2) Fine-tune the multimodal large model processing for images and perform word segmentation processing for text; 3) Constructing equal weights; 4) The processed text is equally weighted with the construction to construct the few-shot prompt words; 5) Train / fine-tune the text and pictures in step 4 and step 2 as a large prediction model; 6) Similarity score; 7) Analyze the ratings.

2. The image-text matching analysis method based on large model and multi-modality according to claim 1 is characterized in that: In the step 1), the image and text analysis is divided into picture analysis and text analysis, wherein the image processing method is: uniformly processing the input image, including resizing, normalization, and color channel standardization, and the text processing method is: segmenting the prompt words, removing stop words, cleaning special characters, etc., and supports multi-language processing.

3. The image-text matching analysis method based on large model and multi-modality according to claim 1 is characterized in that: The specific method of constructing the equal weight in step 2) is: evaluating through multiple dimensions, which are composed of the following: (1) Content accuracy j: the degree of match between the description and the core content of the input text; (2) Description completeness k: whether the generated image description covers the key elements in the image; (3) Language style consistency u: whether the language style of the input text is consistent with that of the generated text; (4) Detail matching v: the degree of consistency between the image and the text in terms of detail description; The score range of each scoring dimension is 0 to 1, and the final comprehensive score can be calculated by the preset weights. The formula is as follows: Overall score = j × content accuracy + k × description completeness + u × language style consistency + v × detail matching.

4. The image-text matching analysis method based on large model and multi-modality according to claim 1 is characterized in that: The specific method in step 5) is: receiving input images and text prompt words, understanding the semantics, scenes and contents in the image through a fine-tuned multimodal large model, and generating a text description that is highly relevant to the image content.

5. The image-text matching analysis method based on large model and multi-modality according to claim 1 is characterized in that: The specific method of similarity scoring in step 6) is as follows: utilizing the few-shot learning capability of the large language model, constructing a prompt template containing multiple sets of known image-text matching pairs, providing context examples for the similarity scoring task, the large language model receiving the multimodally generated image description text and the given text input by the user, performing semantic comparative analysis based on the context examples, and outputting a similarity score between 0 and 1.

6. The image-text matching analysis method based on large model and multi-modality according to claim 1 is characterized by: The method of explaining the rating in step 7) is: while the system generates the rating, it provides a semantic explanation of the rating, indicating the degree of match between the text and the image description in terms of content, style or semantics, and helps users understand the basis for the rating through explanations in natural language.

7. A large model and multi-modal image-text matching analysis system, characterized in that: The system comprises: Image and text parsing module: This module performs preprocessing, which includes image processing and text processing, where the image is uniformly processed, including resizing, fixed to 224×224 pixels, normalization, and color channel standardization format; Text processing: segment prompt words, remove stop words, clean up special characters, and support multi-language processing; Construct a scoring weight module: The scoring dimension module evaluates the generated text and input prompt words from multiple dimensions; Construct a few-shot prompt word module: Design a few-shot prompt word template for relevant scenario tasks; Multimodal semantic understanding module: receives input images and text prompts, understands the semantics, scenes, and content in the image through a fine-tuned multimodal large model, and generates a text description that is highly relevant to the image content; Similarity scoring module: Utilizes the few-shot learning capability of the large language model to construct a prompt template containing multiple sets of known image-text matching pairs, providing contextual examples for the similarity scoring task; Rating explanation module: While the system generates ratings, it also provides semantic explanations of the ratings; The scoring weight construction module is also provided with a dynamic scoring adjustment mechanism: the dynamic scoring adjustment mechanism flexibly adjusts the weight of each scoring dimension according to different application scenarios.

8. A large model and multi-modal image-text matching analysis device, characterized in that: The device utilizes any one of the methods of claims 1 to 6 or the system of claim 7 to achieve image-text matching in the education field or content review in industrial scenarios.

Images