Image-text correlation detection method and device

Abstract

The invention provides an image-text correlation detection method and device, and relates to the technical field of artificial intelligence, in particular to the technical field of image processing, computer vision, natural language processing, large models and deep learning. A specific embodiment of the method comprises the following steps: performing multi-dimensional correlation detection on image context information and document structure data to generate an image-text correlation score; in response to determining that the image-text correlation score is not greater than a first preset score threshold and not less than a second preset score threshold, generating context summary information based on the image context information and the document structure data; generating cue words based on domain knowledge, similar examples and context summary information; and inputting the image context information, the document structure data and the cue word into the large language model, and outputting an image-text correlation detection result.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of image processing, computer vision, natural language processing, large models, and deep learning. Background Technology

[0002] Against the backdrop of explosive growth in digital content, the detection of image-text relevance in rich text documents has become a core requirement for content quality control, information retrieval optimization, and compliance review. With the continuous emergence of specialized content across multiple fields, image-text relevance detection not only needs to accurately determine the degree of association between images and text, but also needs to adapt to the terminology systems, scene characteristics, and document structures of different fields to meet the application needs in diverse scenarios. Its detection efficiency, adaptability, and practicality have attracted widespread attention in the industry.

[0003] Currently, multiple technical approaches have been developed in the field of image-text correlation detection. Among them, rule-based matching methods use fixed rules such as regular expressions and keyword matching to determine the correlation between images and text, requiring no machine learning technology support and having the advantage of rapid application in simple scenarios; traditional machine learning methods use classic models such as SVM (Support Vector Machine) and CRF (Conditional Random Field), relying on manually labeled features and a large amount of training data to build detection models, providing a basic technical solution for image-text matching; deep learning methods build end-to-end image-text matching networks based on deep learning models such as CNN (Convolutional Neural Network) and BERT (Bidirectional Encoder Representations from Transformers), achieving relevance judgment through the model's autonomous learning of features; single large model methods directly call multimodal large models such as GPT (Generative Pre-trained Transformer), using simple prompts to guide the model to complete the reasoning process, making full use of the cross-modal understanding capabilities of large models; hybrid methods combine rules and simple models for preliminary filtering, integrating the basic advantages of both types of technologies to carry out detection work. Summary of the Invention

[0004] This disclosure provides a method, apparatus, device, storage medium, and program product for detecting the correlation between images and text.

[0005] In a first aspect, embodiments of this disclosure propose a method for detecting image-text relevance, comprising: performing multi-dimensional relevance detection on image context information and document structure data to generate an image-text relevance score; in response to determining that the image-text relevance score is not greater than a first preset score threshold and not less than a second preset score threshold, generating context summary information based on the image context information and document structure data; generating prompt words based on domain knowledge, similar examples, and context summary information; and inputting the image context information, document structure data, and prompt words into a large language model to output the image-text relevance detection result.

[0006] Secondly, embodiments of this disclosure propose an image-text relevance detection device, comprising: a first detection module configured to perform multi-dimensional relevance detection on image context information and document structure data to generate an image-text relevance score; a first generation module configured to generate context summary information based on image context information and document structure data in response to determining that the image-text relevance score is not greater than a first preset score threshold and not less than a second preset score threshold; a second generation module configured to generate prompt words based on domain knowledge, similar examples, and context summary information; and a second detection module configured to input image context information, document structure data, and prompt words into a large language model and output an image-text relevance detection result.

[0007] Thirdly, embodiments of this disclosure provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described in the first aspect.

[0008] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described in the first aspect.

[0009] Fifthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0010] The key or essential features of the embodiments disclosed herein are not intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. Wherein: Figure 1This is a flowchart of an embodiment of the image-text relevance detection method according to the present disclosure; Figure 2 This is a flowchart of one embodiment of the multi-dimensional correlation detection steps; Figure 3 This is a flowchart of one embodiment of the prompt word generation process; Figure 4 This is a flowchart of one embodiment of the relevance reasoning steps; Figure 5 This is a schematic diagram of the structure of an embodiment of the image-text correlation detection device according to the present disclosure; Figure 6 This is a block diagram of an electronic device used to implement the image-text correlation detection method of the embodiments of this disclosure. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0014] In this disclosure, the key terms involved are defined as follows: Rich text documents refer to documents that contain various elements such as text, images, chapter structures, and captions, such as academic papers, technical manuals, and e-commerce product detail pages in HTML (HyperText Markup Language) format.

[0015] Image context information: refers to the surrounding information associated with the target image in the document, including the paragraph text before and after the image, the image caption (such as the alt (Alternative Text) attribute, the explanatory text below), the chapter title where the image is located, and the specific location of the image in the document.

[0016] Document structure data: refers to the overall framework information of the document, including the division of chapters, the topic classification of each chapter, and the logical relationships between chapters.

[0017] Image-text relevance score: A quantitative indicator calculated through multi-dimensional rule weighted fusion, with a value range of 0-1, used to characterize the degree of association between an image and its corresponding text and document structure.

[0018] Domain knowledge base: A collection of professional information categorized by industry domain, including key concepts, common image types, and image-text matching rules for that domain, and supports dynamic expansion.

[0019] Example Library: Stores a collection of image-text relevance detection cases in various fields. Each case includes the image-text context, judgment result, and reasoning process. New examples can be added online.

[0020] Dynamic CoT (Chain-of-Thought) reasoning: refers to a chain-of-thought reasoning method that dynamically adjusts the depth of reasoning based on the complexity of the case. It guides the model step by step to complete accurate reasoning and can correct the results through reverse verification.

[0021] Adaptive feedback optimization: Based on human feedback data, it automatically analyzes error patterns, updates the example library, and optimizes prompt word templates to achieve continuous iterative improvement of system performance.

[0022] The FilterResult object is the output data structure of the rule pre-filtering layer, which includes the decision result (match / non-match / uncertain), confidence score, list of matching rules, whether to skip LLM (Large Language Model) flag, and context summary.

[0023] Reverse verification cue words: cue words constructed based on the core conclusions and key evidence of the initial reasoning, used to guide the large model to reason from the opposite perspective.

[0024] Feedback buffer: Implemented based on a double-ended queue, used to cache manual feedback data, with a maximum cache size of 1000 records, and supports filtering data by time, domain, error type, and other dimensions.

[0025] Error pattern analysis: Statistical analysis of error types in manually fed data, including high-confidence errors (error cases with confidence > 0.8), boundary errors (error cases with confidence 0.4-0.6), missing context errors, domain mismatch errors, etc.

[0026] Cost estimation function: Based on API (Application Programming Interface) type, inference depth and context length, the function estimates the cost of a single API call in real time.

[0027] This disclosure aims to propose a training-free rich text-aware multimodal image-text relevance detection system based on prompts. Through a four-level collaborative architecture of rule pre-screening, meta-prompt adaptation, dynamic CoT inference, and adaptive feedback optimization, it achieves highly efficient and accurate detection with the following characteristics: Low cost: No model training required; most cases are solved at zero cost through the rule layer; High accuracy: Improves overall detection accuracy and has strong domain adaptability; High efficiency: Reduces redundant computation and lowers API call costs; Explainable: Structured reasoning process with clear decision-making basis; Self-optimization: Continuous iteration based on human feedback leads to steady performance improvement.

[0028] Typical applications include, but are not limited to, verification of images and text in academic papers, compliance review of technical documents, consistency detection of images and text in e-commerce products, quality inspection of news media content, management of image and text association in enterprise knowledge bases, and verification of matching images and text in medical documents.

[0029] Figure 1 A flowchart 100 of an embodiment of a text-image relevance detection method according to the present disclosure is shown. The text-image relevance detection method includes the following steps: Step 101: Perform multi-dimensional relevance detection on image context information and document structure data to generate image-text relevance scores.

[0030] In this embodiment, the entity executing the image-text relevance detection method can perform multi-dimensional relevance detection on image context information and document structure data to generate an image-text relevance score.

[0031] The core purpose of this step is to quickly filter out high-confidence matching or non-matching cases through multi-dimensional rule fusion, reducing the cost and efficiency losses caused by subsequent large model calls. Essentially, it involves a preliminary quantitative assessment of the correlation between the image and the corresponding text and structure in the document, providing a basis for selecting subsequent processing paths.

[0032] Before performing this step, the input rich text document must be parsed to extract the image context information and document structure data corresponding to each image. The image context information includes the paragraph text before and after the image, the image caption, the chapter title containing the image, and the image's specific location within the document. The document structure data includes the document's chapter divisions and the core themes of each chapter, forming a framework. The parsing process supports common rich text formats such as HTML, ensuring comprehensive extraction of the basic data required for image-text association.

[0033] Multi-dimensional relevance detection is based on rules across multiple dimensions, including positional proximity, explicit citation detection, chapter consistency, semantic density detection, and image caption information detection. Each rule assesses text-image relevance from different perspectives, and a weighted fusion is used to obtain a comprehensive text-image relevance score. The specific execution logic, scoring criteria, and weight configuration of each rule will be discussed later. Figure 2 The corresponding embodiments are described in detail.

[0034] In some embodiments, in response to determining that the image-text relevance score is greater than a first preset score threshold, the execution entity may output an image-text matching result; in response to determining that the image-text relevance score is less than a second preset score threshold, the execution entity may output an image-text mismatch result; in response to determining that the image-text relevance score is neither greater than the first preset score threshold nor less than the second preset score threshold, the execution entity may execute step 102.

[0035] For example, the first preset score threshold is set to 0.85, and the second preset score threshold is set to 0.25. This threshold configuration is optimized based on a large amount of test data and can effectively distinguish between high-confidence cases and uncertain cases. When the image-text relevance score > 0.85, it indicates that the evidence of association between the image and the text / document structure is sufficient, and it is judged as a high-confidence match, directly outputting a matching conclusion; when the image-text relevance score < 0.25, it indicates that the evidence of association between the image and the text is seriously insufficient, and it is judged as a high-confidence mismatch, directly outputting a mismatch conclusion. Neither of these two types of high-confidence cases needs to enter the subsequent large model processing flow. Detection can be completed quickly through the rule layer. Most cases are resolved directly through the rule layer without calling LLM, resulting in zero API cost, significantly improving detection efficiency and reducing API call costs.

[0036] Step 102: In response to determining that the image-text relevance score is not greater than a first preset score threshold and not less than a second preset score threshold, context summary information is generated based on image context information and document structure data.

[0037] In this embodiment, in response to determining that the image-text relevance score is not greater than a first preset score threshold and not less than a second preset score threshold, the aforementioned execution entity can generate context summary information based on image context information and document structure data.

[0038] When the image-text relevance score falls between the second and first preset thresholds (e.g., 0.25 ≤ image-text relevance score ≤ 0.85), it indicates that the evidence for image-text relevance in this case is insufficient or contradictory, classifying it as an uncertain case that requires further precise judgment through subsequent large-scale model reasoning. Generating contextual summary information at this point serves the core purpose of providing structured and concise input data to the large-scale model, reducing token consumption and reasoning interference caused by redundant information, and improving the reasoning efficiency and accuracy of the large-scale model.

[0039] Contextual summary information is a structured and concise extraction of image context information and document structure data, retaining only key information strongly related to the image-text relevance judgment. It typically includes four core parts: First, chapter information, namely the chapter title and core theme where the image is located, without needing to include the complete text within the chapter; second, concise surrounding text, removing redundant modifiers and irrelevant sentences from the paragraphs before and after the image, retaining the core description; third, figure caption information, directly using the original figure caption, only with formatting adjustments, without adding any extra content; and fourth, location information, clearly specifying the image's exact location in the document.

[0040] The process of generating contextual summaries requires no manual intervention; the system automatically completes the filtering and integration, ensuring the objectivity and relevance of the summary information and laying the foundation for subsequent prompt word generation and large-scale model inference.

[0041] Step 103: Generate prompt words based on domain knowledge, similar examples, and contextual summary information.

[0042] In this embodiment, the aforementioned execution entity can generate prompt words based on domain knowledge, similar examples, and contextual summary information.

[0043] The core of this step is to generate domain-adaptive structured prompts that fit the current domain and case scenario, addressing the problem of poor adaptability and low inference accuracy of general prompts in specialized domains. By integrating domain knowledge, similar examples, and rule-layer prior knowledge into the prompts, the large model is guided to perform accurate inference, improving the accuracy of cross-domain text-image relevance detection.

[0044] The prompt generation process relies on a pre-defined domain knowledge base and example library. It combines key features from the contextual summary information to perform operations such as domain knowledge injection, similar example retrieval, rule-based prompt conversion, and template filling. The specific execution logic includes loading the domain knowledge base and example library, retrieving similar examples, converting matching rules into natural language prompts, selecting appropriate prompt templates, and filling in relevant information. Detailed procedures will be discussed later. Figure 3 The corresponding embodiments are described in detail.

[0045] The generated prompts are not in a fixed format, but are dynamically adjusted according to the domain type and context features of the current case to ensure that they can provide clear reasoning guidance for the large model, give full play to the cross-modal understanding capabilities of the large model, and avoid invalid reasoning and redundant computation.

[0046] Step 104: Input the image context information, document structure data and prompt words into the large language model, and output the image-text relevance detection results.

[0047] In this embodiment, the aforementioned execution entity can input image context information, document structure data, and prompt words into the large language model and output the image-text relevance detection results.

[0048] This step is the precise reasoning stage for uncertain cases. By calling the multimodal large model API and combining it with the generated domain-adaptive prompts, cross-modal fusion reasoning of image and text information is achieved, and finally, accurate and interpretable detection results are output.

[0049] During execution, the LLM API configuration must first be initialized. It supports access to various mainstream multimodal LLM APIs and can be flexibly switched according to business needs. Then, the prompt words, image URLs (Uniform Resource Locators), and inference configurations (such as inference depth and whether self-verification is enabled) are input into the LLM API. The large model will be guided by the prompt words, synchronously parsing image content and text information to perform structured inference.

[0050] After inference is complete, the system will perform fault-tolerant parsing on the output of the large model, handling anomalies such as non-standard JSON (JavaScript Object Notation) output, and extracting and outputting structured detection results containing matching or non-matching conclusions, confidence levels, inference processes, and key evidence. The core purpose of outputting the inference process and key evidence is to improve the interpretability of the detection results, facilitate error tracing and compliance review, and solve the problem of the black box nature of traditional large model inference.

[0051] Furthermore, for complex cases, a self-verification mechanism can be enabled. This involves asking the large model to re-infer the data through reverse questioning, adjusting the confidence level based on the verification results, further reducing high-confidence errors, and improving the reliability of the detection results. Detailed execution logic, inference parameter configuration, and self-verification process for this step will be discussed later. Figure 4 The corresponding embodiments are described in detail.

[0052] The image-text relevance detection method provided in this disclosure achieves low-cost, high-accuracy, high-efficiency, interpretable, and self-optimizing image-text relevance detection through a four-level collaborative architecture of "rule pre-screening - meta-hint adaptation - dynamic CoT inference - adaptive feedback optimization". The model-train-free design reduces deployment and iteration costs, with most cases solved at zero cost through the rule layer, reducing API call costs. Domain-adaptive hints and dynamic inference depth configuration enhance cross-domain adaptability and improve overall detection accuracy. The structured inference process and key evidence output meet compliance requirements, and errors are traceable. The self-optimization mechanism based on human feedback ensures continuous system performance improvement, increasing weekly accuracy and demonstrating strong long-term stability. This method can be widely applied to various scenarios such as academic paper image-text verification, technical document compliance review, e-commerce product image-text consistency detection, news media content quality inspection, enterprise knowledge base image-text association management, and medical document image-text matching verification, demonstrating strong practicality.

[0053] Figure 2 A flowchart 200 illustrates an embodiment of a multi-dimensional correlation detection step. This multi-dimensional correlation detection step may include the following sub-steps: Step 201: Perform correlation detection on image context information and document structure data in at least two dimensions, and generate similarity scores for at least two dimensions.

[0054] In this embodiment, the aforementioned execution entity can perform correlation detection on image context information and document structure data in at least two dimensions, and generate similarity scores corresponding to at least two dimensions.

[0055] The core of this sub-step is to collect evidence of image-text relevance from different angles through multi-dimensional parallel detection, avoiding the one-sidedness of single-dimensional judgment, and providing comprehensive and objective basic data for subsequent comprehensive score calculation. The detection of each dimension is independent yet complementary, together forming a multi-dimensional evaluation system for image-text relevance.

[0056] In some embodiments, relevance detection may include, but is not limited to, at least two of the following dimensions: positional proximity, explicit citation detection, chapter consistency detection, semantic density detection, and caption information detection. Specifically, positional proximity similarity scores are determined based on the distance between the image and preceding and following paragraphs, as well as the paragraph length; explicit citation similarity scores are determined by detecting preset citation patterns in the text using regular expressions; chapter consistency similarity scores are determined by detecting preset keywords in chapter titles; the number of preset technical terms per preset number of characters in the image context information is counted to determine the similarity score for semantic density detection; and caption information similarity scores are determined based on caption length and valid information within the caption.

[0057] The specific detection logic and scoring criteria for each dimension are as follows: Location proximity detection: This method determines the relevance of images and text within the document layout based on the distance between an image and the preceding and following paragraphs, as well as the paragraph length. For example, paragraph length is defined as 100 characters; paragraphs longer than 100 characters are considered long paragraphs. The specific scoring criteria are: an image immediately preceding a long paragraph (i.e., the long paragraph is immediately before the image) receives 0.9 points; a long paragraph immediately following the image receives 0.8 points; an image with clear preceding text description (not a long paragraph but containing related text) receives 0.6 points; and an image without any preceding or following paragraphs, considered an isolated image, receives 0.3 points. This rule is based on the principle that images adjacent to core descriptive paragraphs in a document generally have a stronger relevance to the text.

[0058] Explicit Reference Detection: This feature scans the text surrounding an image using regular expressions to detect the presence of reference patterns that directly point to the image. Preset reference patterns include common expressions such as "as shown," "see Figure X," "below," "above," and "this image." The scoring criteria are as follows: 1.0 point for matching a precise reference pattern like "see Figure X"; 0.9 points for matching explicit reference patterns like "as shown" or "below"; 0.7 points for matching indirect reference patterns like "this image" or "the aforementioned image"; and 0 points for no reference patterns pointing to the image in the text. This rule directly verifies the relevance between text and image by capturing explicit associations between them.

[0059] Chapter Consistency Detection: This determines whether the theme of the chapter containing the image is consistent with the image's purpose. For example, it checks whether the chapter title contains core keywords such as technology, principle, design, parameters, function, and structure. The specific scoring criteria are: 0.8 points for a chapter title containing these core keywords and matching the image type (e.g., technical illustration, parameter chart, structural diagram); and 0.3-0.5 points (adjusted based on the degree of correlation) for a chapter title that does not contain these core keywords but whose content is indirectly related to the image. This rule helps determine the relevance of the text and images at the document structure level, ensuring that the image belongs to the core content category of the current chapter.

[0060] Semantic density detection: This involves counting the number of predefined technical terms per 100 characters in the image context. The term density reflects the degree of matching between the text's professional attributes and the image. The predefined technical terms are configured according to the current domain (e.g., model number, material, power consumption in e-commerce; lesion, indicator, image features, etc. in the medical field). The specific processing flow is as follows: First, count the total number of predefined technical terms in the context text, calculate the number of terms per 100 characters, then normalize this value and output a score of 0-1. Higher term density indicates stronger text professional attributes, a greater probability of relevance to the image in the corresponding professional domain, and a higher score.

[0061] Caption Information Detection: The relevance of captions to the text theme is determined based on their length and effective information content. Effective information refers to information that reflects the core content of the image, such as numbers, technical terms, and descriptions of core functions. Specific scoring criteria are as follows: Captions ≥ 10 characters long and containing effective information such as numbers and technical terms receive 0.8 points; captions 5-9 characters long and containing some effective information receive 0.5 points; captions < 5 characters long but containing key information (such as core terms) receive 0.3 points; no caption or a caption containing no effective information (e.g., only Image 1, decorative image, etc.) receives 0 points. As direct explanatory information about the image, the effectiveness of captions directly reflects the relevance between the image and text.

[0062] The execution order of each dimension detection can be flexibly adjusted, supporting parallel or serial processing. Ultimately, all outputs a similarity score in the range of 0-1, ensuring that the scores of each dimension are comparable and providing a unified standard for subsequent weighted fusion.

[0063] Step 202: Perform a weighted summation of the similarity scores corresponding to at least two dimensions to generate a text-image relevance score.

[0064] In this embodiment, the aforementioned execution entity can perform a weighted summation of the similarity scores corresponding to at least two dimensions to generate a text-image relevance score.

[0065] The core of this sub-step is to integrate multi-dimensional similarity scores into a single image-text relevance score through weighted fusion, thereby achieving a quantitative summary of image-text relevance. The weighted fusion design is based on the fact that each dimension contributes differently to the judgment of image-text relevance, and by configuring weights, the influence of key dimensions is highlighted, thus improving the accuracy of the comprehensive score.

[0066] For example, the specific weight configuration is as follows: positional proximity weight 0.25, explicit citation detection weight 0.35, chapter consistency detection weight 0.15, semantic density detection weight 0.15, and image caption information detection weight 0.10. This weight configuration is optimized based on a large amount of cross-domain test data. Among them, explicit citation detection has the highest weight because explicit citations of images by text are the strongest direct evidence of image-text relevance. Positional proximity, semantic density, and image caption information detection have the next highest weights, supplementing evidence from the perspectives of layout, content professionalism, and image description, respectively. Chapter consistency detection has a relatively low weight, serving as an auxiliary verification dimension.

[0067] Meanwhile, the weights support dynamic adjustment and can be optimized according to the characteristics of different domains (e.g., in the medical field, the semantic density of documents and figure caption information have a greater impact on relevance judgment, so their weights can be appropriately increased; in the e-commerce field, explicit citations and positional proximity are more important, so their corresponding weights can be maintained or fine-tuned). Dynamic weight configuration enhances the domain adaptability of the rule layer, ensuring that the relevance of text and images can be accurately quantified in different scenarios.

[0068] The weighted summation is calculated as follows: Image-text relevance score = Location proximity score × 0.25 + Explicit citation detection score × 0.35 + Chapter consistency detection score × 0.15 + Semantic density detection score × 0.15 + Caption information detection score × 0.10. The image-text relevance score calculated by this formula ranges from 0 to 1. A higher score indicates a stronger image-text relevance, providing a clear quantitative basis for subsequent case type classification (high-confidence match, high-confidence mismatch, uncertain case).

[0069] The rule pre-screening layer is implemented in a lightweight manner using regular expressions, string processing, and weighted calculations. It does not rely on large models and model training processes, significantly reducing deployment costs. Moreover, the rule weights can be dynamically adjusted according to different domain scenarios, allowing for rapid adaptation to various application needs. At the same time, this layer can directly solve most cases, achieving efficient filtering with zero API cost. This reduces the processing pressure on subsequent modules, improves the overall detection speed, and generates structured context summaries, providing simplified input for subsequent LLM inference. It effectively reduces token consumption and inference difficulty, achieving simultaneous optimization of cost, efficiency, and inference adaptability.

[0070] It should be noted that there is an alternative to the rule pre-screening layer that can achieve the same inventive purpose: replacing weighted fusion with a machine learning model and training the rule weights with a small amount of labeled data. This alternative may slightly improve accuracy, but it increases labeling costs and training steps, losing the advantage of no training required. It is suitable for scenarios where the accuracy of the rule layer is extremely high.

[0071] Figure 3 A flowchart 300 of one embodiment of the prompt word generation step is shown. This prompt word generation step may include the following sub-steps: Step 301: Load the domain knowledge base and example library.

[0072] In this embodiment, the aforementioned execution entity can load a domain knowledge base and a sample library.

[0073] Domain knowledge base and example library are the core data support for generating domain-adaptive prompts. The loading process needs to accurately locate and load the corresponding domain resources according to the domain configuration of the current detection task (such as e-commerce, medical, electronics, etc.) to ensure that the prompts can incorporate targeted domain information.

[0074] Domain-specific knowledge bases store key concepts, common image types, and matching rules categorized by domain, supporting dynamic expansion. For example, the e-commerce knowledge base includes key concepts such as product model, material, function, size, and power consumption; common image types such as product images, functional diagrams, usage scenario diagrams, and parameter comparison diagrams; and matching rules such as ensuring that the image content matches the function, material, and model in the product description, and that usage scenario diagrams match the applicable scenarios for the product. The medical knowledge base includes key concepts such as lesions, imaging features, diagnostic indicators, and treatment plans; common image types such as CT images, pathological slide images, and indicator trend diagrams; and matching rules such as ensuring that the image matches the location and features of the lesion in the diagnostic description.

[0075] The example library contains few-shot examples of image-text relevance detection in various fields. Each example includes image-text context, judgment result, and reasoning process, and supports adding new examples online. The examples are designed according to the principle of being concise and precise, covering typical scenarios in the field (such as product main image matching, functional illustration matching, and decorative image mismatch in e-commerce, and lesion image and diagnostic text matching, and irrelevant image mismatch in the medical field), ensuring that they can provide effective reasoning references for large models.

[0076] During loading, the system verifies the completeness and validity of the knowledge base and example library. If missing or invalid data is found, the system will automatically activate the default domain resources to avoid affecting the prompt word generation process. Simultaneously, it supports batch loading of resources from multiple related domains (e.g., simultaneously loading resources from the electronics and automotive domains for new energy vehicle-related document detection), improving cross-domain adaptability.

[0077] Step 302: Based on the context summary information, retrieve similar examples from the example library.

[0078] In this embodiment, the aforementioned execution entity can retrieve similar examples from the example library based on context summary information.

[0079] The core of this sub-step is to match the most relevant reference examples for the current uncertain cases. These examples guide the large model to understand the text-image matching logic of the current domain, improving inference accuracy. The retrieval process employs a lightweight keyword similarity matching algorithm, eliminating the need for a vector database and balancing retrieval efficiency and accuracy.

[0080] The specific retrieval process is as follows: First, extract core keywords (such as domain terms, image types, chapter topics, etc.) from the context summary information. For example, the core keywords for e-commerce cases may be refrigerator, functional diagram, cooling power, and product parameter chapters. Then, scan the examples in the example library based on these core keywords and calculate the keyword matching degree between the current case and each example. Finally, select the Top-K (K value is configurable, for example, K=3) most relevant examples as similar examples.

[0081] During the search process, the system supports adjusting the matching priority based on keyword weight (e.g., the matching weight of core domain terms is higher than that of ordinary keywords), ensuring that the retrieved examples are highly consistent with the current case in terms of core features. Simultaneously, if there are not enough similar examples in the example library (e.g., the matching degree is lower than a preset threshold), the system will automatically select general examples in the domain to avoid a decrease in the effectiveness of suggestion words due to a lack of relevant examples.

[0082] Step 303: Convert the matching rule information in the multi-dimensional correlation detection into natural language prompts.

[0083] In this embodiment, the aforementioned execution entity can convert the matching rule information in multi-dimensional correlation detection into natural language prompts.

[0084] The purpose of this sub-step is to transfer the prior knowledge from the rule layer to the large model, enabling cross-layer information collaboration. This allows the large model to understand the detection status of the current case at the rule layer, assisting it in making accurate inferences. The conversion process needs to transform the structured rule matching results into intuitive and easy-to-understand natural language, avoiding the use of technical jargon, to ensure that the large model can accurately understand them.

[0085] The specific conversion logic is as follows: First, extract the rules with high scores (e.g., ≥0.5 points) and their corresponding scores from the multi-dimensional relevance detection. For example, the explicit citation detection score is 0.9 points (text contains the image as shown in the figure), and the positional proximity score is 0.8 points (closely following a long paragraph). Then, convert these rule information into natural language descriptions. For example, the context text of the image contains explicit citations as shown in the figure, and the image is closely followed by a long paragraph containing the core description, which initially indicates a strong correlation with the text. Finally, integrate the conversion results of multiple rules to form coherent natural language prompts, avoiding information fragmentation.

[0086] If a rule score for a certain dimension is extremely low (e.g., ≤0.2 points), it can be selectively included in the prompt information. For example, if the image caption has no effective information (score of 0 points), it is necessary to focus on verifying the correlation between the image content and the text, and to prompt the large model with potential contradictions that need to be addressed.

[0087] Step 304: Fill the prompt word template with domain knowledge, similar examples, natural language prompt information and context summary information from the domain knowledge base to generate prompt words.

[0088] In this embodiment, the aforementioned execution entity can fill the prompt word template with domain knowledge, similar examples, natural language prompt information, and context summary information from the domain knowledge base to generate prompt words.

[0089] This sub-step is the final stage of prompt word generation. It achieves structured integration of information through template filling, ensuring that the prompt words are logically clear and the information is complete, guiding the large model to reason according to fixed logic.

[0090] The prompt templates support two types: basic templates and CoT templates, which can be selected based on configuration. Basic templates are suitable for simple, uncertain cases, featuring a concise structure and focusing on conveying core information. CoT templates include step-by-step reasoning guidance (e.g., first step: understanding the core content of the text → second step: analyzing the core features of the image → third step: comparing the correlation points between the text and the image → fourth step: comprehensive judgment), suitable for complex cases, and effectively improving the reasoning logic and accuracy of large models. Templates support A / B testing, allowing you to select the optimal template based on performance feedback from different templates and continuously optimize the prompt effect.

[0091] The filling process incorporates various types of information sequentially according to the pre-defined structure of the template: First, the task description is filled in (clearly requiring the large model to judge the relevance of images and text and output the reasoning process and key evidence); then, domain knowledge is filled in (key concepts, image types, and matching rules of the current domain); next, similar examples are filled in (Top-K search results, including the image and text context of the examples, judgment results, and reasoning processes); then, natural language prompts are filled in (rule-level prior knowledge); and finally, contextual summary information (core information of the current case) is filled in.

[0092] After the information is filled in, the system optimizes the format of the prompts (e.g., segmentation, bolding key information, using list presentation examples) to improve readability, while controlling the length of the prompts within a reasonable range (estimated token consumption based on the depth of subsequent inference to avoid exceeding the token limit of the large model). The final generated domain-adaptive structured prompts contain specific information about the current case, as well as domain knowledge, reference examples, and prior knowledge from the rule layer, effectively guiding the large model to perform accurate and efficient inference.

[0093] The meta-hint adaptation layer utilizes a categorized storage and dynamic expansion design of domain knowledge bases and example libraries, combined with a lightweight dynamic example retrieval mechanism based on keyword similarity matching, enabling efficient operation without relying on vector databases and model training. Its rule-based hint generation function transforms prior knowledge from the rule layer into natural language hints. Coupled with basic templates, CoT templates with step-by-step reasoning guidance, and A / B testing optimization capabilities, it not only significantly improves the accuracy of professional domain detection through domain knowledge injection and similar example guidance, but also reduces the understanding cost of LLM and minimizes invalid reasoning with structured hint words. Simultaneously, it supports the dynamic addition of domains and examples and cross-layer knowledge collaboration, achieving rapid domain adaptation and inter-layer information complementarity without modifying the core code, balancing detection accuracy, reasoning efficiency, and scalability.

[0094] It should be noted that there is an alternative solution for the meta-hint adaptation layer that can achieve the same inventive purpose: semantic retrieval of examples based on a vector database, replacing keyword matching. This alternative offers more accurate example matching, but increases the deployment and maintenance costs of the vector database, reduces its lightweight nature, and is suitable for scenarios with a large example library.

[0095] In some embodiments, human feedback data corresponding to the image-text relevance detection results is obtained; based on the human feedback data, the sample library and prompt word templates are updated.

[0096] Human feedback data is the core input for the system's self-optimization. The acquisition process supports multiple methods (such as manual review and annotation, user feedback submission, etc.), specifically including model prediction results, true labels (match / non-match), the prompt word template used, contextual summary information, and error reason descriptions. The system receives this data through a dedicated feedback collection interface and stores it in the feedback buffer.

[0097] The feedback buffer is implemented using a double-ended queue, with a maximum cache size of 1000 records. It supports filtering data by time, domain, error type, and other dimensions. The caching strategy follows a first-in, first-out (FIFO) principle; when the cache size reaches its limit, the oldest feedback data is automatically deleted to ensure that the buffer stores the latest feedback information. Additionally, it supports manually exporting feedback data for offline analysis, providing a reference for adjusting optimization strategies.

[0098] Updating the example library and prompt word templates based on human feedback data is key to achieving continuous iteration of system performance. It eliminates the need to retrain the model; detection accuracy can be improved simply through data updates, demonstrating the advantages of a training-free architecture. The update process can be triggered automatically by the system or manually; the specific update logic will be explained in detail in subsequent embodiments.

[0099] In some embodiments, the accuracy of different prompt word templates is calculated based on human feedback data; a target prompt word template is selected based on the accuracy; in response to the amount of human feedback data reaching a preset data volume threshold, the error types corresponding to the human feedback data are analyzed; samples are extracted from error cases based on the error types to construct new examples; and the new examples are added to the example library.

[0100] This embodiment details the specific logic for updating the example library and prompt word templates based on human feedback data, ensuring that the update process is scientific and effective, and achieving precise optimization of system performance.

[0101] Specifically, the accuracy of different prompt word templates is first calculated based on human feedback data: for each prompt word template, the number of cases in the corresponding feedback data whose predicted results match the actual labels is counted, and this number is divided by the total number of cases corresponding to that template to obtain the accuracy of that template. The system tracks the accuracy changes of each template in real time, supports statistics by domain and time period (such as weekly or monthly), and clearly presents the performance of each template.

[0102] Then, the target prompt word template is selected based on accuracy: the system sets an accuracy threshold (e.g., 90%). When the accuracy of a template is higher than the threshold and ranks first among all templates, it is automatically set as the target prompt word template, and subsequent similar cases will prioritize using this template to generate prompt words. If the accuracy of all templates is lower than the threshold, the system will trigger a template optimization process (e.g., adjusting the template structure, adding domain information placeholders, etc.), generating a new template and adding it to the template library to participate in accuracy competition. Simultaneously, manually specifying the target template is supported to meet the needs of special scenarios.

[0103] When the amount of manually submitted data reaches a preset threshold (e.g., 50 entries), the system automatically initiates error mode analysis. Error types primarily include high-confidence errors (errors with confidence > 0.8), boundary errors (errors with confidence between 0.4 and 0.6), missing context errors (errors caused by insufficient contextual summary information), and domain mismatch errors (errors caused by missing domain knowledge). During the analysis, the system will calculate the percentage of each type of error and identify the main weaknesses of the current system (e.g., if high-confidence errors have a high percentage, the focus will be on optimizing the self-verification mechanism; if domain mismatch errors have a high percentage, the focus will be on supplementing the domain knowledge base and example library).

[0104] Based on error type, samples are extracted from error cases to construct new examples: For each major error type, typical cases (such as representative high-confidence error cases and frequently occurring domain adaptation error cases) are selected, and their contextual summary information, true labels, and reasoning logic (added manually or generated by the system based on true labels) are extracted. New examples are then constructed according to the unified format of the example library. The new examples must undergo validity verification (such as manual review to confirm their representativeness) to avoid invalid examples from entering the example library.

[0105] Finally, the new examples are added to the example library: new examples are stored by domain, with no more than 20 examples in a single domain. When this limit is exceeded, the oldest or worst-performing examples (such as those with the lowest accuracy in guiding large model inference) are automatically deleted to ensure the timeliness and effectiveness of the example library. After addition, the system updates the index of the example library to ensure that subsequent searches can quickly locate the new examples.

[0106] Through the above update process, the example library can be continuously supplemented with typical cases for error patterns, the prompt word template can always maintain optimal performance, and the system can achieve continuous optimization effect of weekly accuracy improvement based on human feedback, with strong long-term stability.

[0107] The adaptive feedback optimization layer uses a double-ended queue to cache the latest feedback data and supports multi-dimensional filtering. Combined with lightweight error pattern analysis and example generation logic, it can achieve system self-optimization without training any model. It tracks the accuracy of different prompt versions with a performance statistician and supports A / B testing. Through example library management, it automatically maintains library capacity and example validity. It can improve weekly accuracy based on human feedback and ensure long-term stability. At the same time, it can supplement examples in a targeted manner and automatically select the optimal prompt version through error pattern analysis. While reducing the model retraining and maintenance costs, it enhances the ability to handle complex scenarios, reduces human intervention, and achieves intelligent iterative optimization.

[0108] It should be noted that there is an alternative to the adaptive feedback optimization layer that can achieve the same inventive purpose: optimizing prompt words based on reinforcement learning, replacing the example library update. This alternative may be faster, but it requires a complex reinforcement learning framework and a large amount of feedback data, losing its lightweight advantage, and is suitable for large-scale application scenarios with sufficient feedback data.

[0109] Figure 4 A flow 400 illustrating an embodiment of a correlation inference step is provided. This correlation inference step may include the following sub-steps: Step 401: Initialize the configuration of the Large Language Model Application Programming Interface (LLM API).

[0110] In this embodiment, the aforementioned execution entity can initialize the configuration of the Large Language Model Application Programming Interface (LLM API).

[0111] The core of this sub-step is to complete the integration preparation for the LLM API, ensuring flexible access to multimodal large models from different vendors and adapting to different deployment needs and performance requirements. The initialization process supports the integration of multiple mainstream multimodal LLM APIs.

[0112] The specific initialization process is as follows: First, receive the API type, access key, request timeout, and other parameters configured by the user; then, load the corresponding API adaptation module to complete the interface protocol adaptation (such as RESTful API protocol, GraphQL (Graph Query Language) protocol, etc.); next, verify the availability of the API (confirm that the interface can respond normally by sending test requests); finally, configure the default parameters for API calls (such as maximum concurrency, number of retries, etc.) to avoid interruption of the inference process due to interface abnormalities.

[0113] During initialization, the system supports dynamic switching of API types. Users can select the appropriate API based on factors such as cost, performance, and privacy requirements (e.g., choosing an API that supports local deployment when data privacy is critical, or choosing an API with fast response times when inference speed is critical). Simultaneously, the system records the API call status and performance data (such as response time, success rate, and error type) to provide a basis for subsequent API selection and optimization. If API verification fails (e.g., incorrect key, network failure), the system will provide a clear prompt and allow the user to reconfigure, ensuring the inference process can start normally.

[0114] Step 402: Input the prompt word, image Uniform Resource Locator, and inference depth into the LLM API.

[0115] In this embodiment, the aforementioned execution entity can input prompt words, image Uniform Resource Locators, and inference depth into the LLM API.

[0116] The image URL is the network address pointing to the image file to be detected. The system uses this URL to obtain the raw image data for large models to parse the image content. The URL supports HTTP (Hypertext Transfer Protocol) / HTTPS (Hypertext Transfer Protocol Secure) protocols, ensuring compatibility with mainstream image storage services (such as cloud storage, local server storage, etc.). If the image cannot be obtained via the URL (e.g., the URL is invalid, insufficient permissions), the system will prompt the user to provide a valid image resource again (e.g., uploading a local image file, with the system temporarily generating a URL).

[0117] Inference depth is a core parameter controlling the inference complexity of large models. It supports three configuration levels: simple, medium, and deep. Users can choose according to their business needs (such as accuracy requirements and cost budgets), or the system can automatically allocate it based on the complexity of the context summary information. Different inference depths correspond to different application scenarios: simple mode is suitable for uncertain cases with clear context information and little controversy (such as matching simple functional diagrams in the e-commerce field); medium mode is suitable for most common uncertain cases; and deep mode is suitable for cases with complex context and high level of expertise (such as matching complex lesion images in the medical field and matching technical principle diagrams in academic papers).

[0118] During input, the system performs validity checks on prompts, URLs, and inference depth: prompts must conform to the format requirements of the corresponding LLM (e.g., avoiding special characters and controlling length); URLs must be accessible and point to image files; and inference depths must be one of the preset three levels. After successful validation, this information is encapsulated into request data according to the requirements of the corresponding API, ensuring that the large model can correctly parse the input content.

[0119] Step 403: Configure the maximum number of tokens and temperature coefficient based on the inference depth.

[0120] In this embodiment, the execution entity can configure the maximum number of tokens and the temperature coefficient according to the inference depth.

[0121] Maximum number of tokens and temperature coefficient are key parameters affecting the performance and cost of large model inference. Different parameter configurations are required for different inference depths to achieve a balance between accuracy and cost.

[0122] For example, the specific configurations are as follows: Simple mode is configured with 500 tokens / temperature coefficient 0.3; Medium mode is configured with 1000 tokens / temperature coefficient 0.5; and Deep mode is configured with 1500 tokens / temperature coefficient 0.7. This configuration is optimized based on a large amount of test data: the maximum number of tokens increases with the inference depth to ensure that complex cases have enough tokens to support complete inference; the temperature coefficient increases with the inference depth, maintaining a low value (0.3) in Simple mode to ensure the stability and determinism of the inference results, and appropriately increasing it (0.7) in Deep mode to enhance the flexibility and creativity of inference and better handle complex scenarios.

[0123] Parameter configuration supports dynamic adjustment, allowing users to modify it based on actual usage (e.g., if insufficient inference is found in simple mode, the maximum token count can be adjusted to 600 tokens; if excessive fluctuations are found in deep mode, the temperature coefficient can be adjusted to 0.6). The system records inference performance (accuracy, response time, cost) under different parameter configurations, providing users with parameter optimization suggestions. Furthermore, it supports dynamic parameter adjustment based on case complexity (e.g., at the same inference depth, cases with longer context summary information automatically increase the maximum token count by 10%), further improving adaptability.

[0124] Step 404: Call the LLM API to infer the image context information, document structure data, and prompt words to generate image-text relevance detection results.

[0125] In this embodiment, the aforementioned execution entity can call the LLM API to infer the image context information, document structure data, and prompt words to generate image-text relevance detection results.

[0126] This sub-step is the core of accurate detection of uncertain cases. By leveraging the cross-modal understanding capabilities of the large model, it integrates image and text information for structured reasoning, and outputs accurate and interpretable detection results.

[0127] When calling the LLM API, the system sends the encapsulated request data (prompt words, image URLs, inference parameters, etc.) to the corresponding large model server, while recording information such as the call time and request ID for later traceability. A timeout retry mechanism is supported during the call (the number of retries is configurable, e.g., 3 times). If the call fails due to network errors, API service unavailability, or other reasons, it will automatically retry. Upon failure of the retry, an error message will be output and the fault information will be logged to ensure timely handling by the user.

[0128] After receiving the request, the large model will perform inference according to the logic guided by the prompt words: First, it will parse the image content (identify image type, core features, key information, etc.); then, it will understand the contextual summary information and document structure data (grasp the core text content, chapter themes, and the relationship between text and images); next, it will combine domain knowledge and similar examples to analyze the correlation between text and images (such as whether the function displayed by the image is consistent with the function described by the text, whether the parameters in the image match the parameters in the text, etc.); finally, it will comprehensively judge the relevance between text and images, generate a matching / non-matching conclusion, and provide a detailed reasoning process and key evidence.

[0129] After receiving the response from the large model, the system performs fault-tolerant parsing. For anomalies such as non-standard JSON output, incomplete reasoning processes, and missing key evidence, it uses regular expressions to extract core information and supplement the default reasoning process to ensure the uniformity and completeness of the output format. The final generated image-text relevance detection result includes four core parts: first, the decision result (match / non-match), clarifying the image-text relevance; second, the confidence level (0-1 range), reflecting the reliability of the decision result; third, the reasoning process, detailing the reasoning logic of the large model (e.g., the core text content is that the refrigerator's cooling power is 300W, and the image is a functional diagram of the refrigerator, showing the working process of the cooling system, which is highly relevant to the cooling function described in the text, therefore it is judged as a match); and fourth, key evidence, listing the core evidence supporting the decision (e.g., the cooling system components in the image are consistent with the cooling power-related parameters described in the text).

[0130] The detection results support structured output formats (such as JSON, XML (Extensible Markup Language)) and visual report output (such as HTML, Excel), making it easy for users to integrate into downstream systems or view them directly. Simultaneously, the system will associate and store the detection results with rule-level information, prompts, and other data, providing comprehensive data support for subsequent human feedback and system optimization.

[0131] The dynamic CoT inference engine is implemented based on the existing multimodal LLM API, eliminating the need to train custom models. Its dynamic configuration of inference depth allows for flexible adaptation to different business needs. Its self-verification mechanism can be enabled or disabled via secondary API calls, and it also features cost estimation to help users control API call budgets. This engine not only adapts to cases of varying complexity through dynamic inference depth and reduces high-confidence errors through its self-verification mechanism, significantly improving detection accuracy, but it also rationally adjusts inference resource allocation based on case complexity to avoid overcomputation. Furthermore, it ensures error traceability through a structured inference process and key evidence, and supports multiple API switching to adapt to different deployment requirements, balancing detection accuracy, cost control, interpretability, and deployment flexibility.

[0132] It should be noted that there are alternatives to the dynamic CoT inference engine that can achieve the same inventive purpose: deploying local inference using an open-source multimodal model instead of API calls. This alternative has lower API costs and better data privacy in the long run, but it requires GPU (Graphics Processing Unit) computing power, resulting in high deployment and maintenance costs. It is suitable for scenarios with high data privacy requirements and sufficient computing resources.

[0133] In some embodiments, reverse verification prompts corresponding to the prompt words are constructed; image context information, document structure data and reverse verification prompts are input into a large language model, and reverse verification results are output; based on the reverse verification results, the confidence level of the image-text relevance detection results is adjusted.

[0134] This embodiment is a self-verification mechanism of the dynamic CoT inference engine, which is only enabled when the inference depth is in deep mode. Its core purpose is to reduce high confidence errors and improve the reliability of detection results.

[0135] Specifically, the first step is to construct reverse verification prompts: based on the core conclusions and key evidence of the initial reasoning, design prompts to ask reverse questions, guiding the large model to reason from the opposite perspective. For example, for cases where the initial conclusion is a match, the reverse verification prompt might be: "Please assume the image and text do not match, analyze whether there is reasonable evidence, and explain the reasoning process in detail." For cases where the initial conclusion is a mismatch, the reverse verification prompt might be: "Please re-examine the connection points between the image and the text, whether there is any missing matching evidence, and if so, explain and correct the judgment result." The design of the reverse verification prompts must maintain a consistent structure and domain information with the initial prompts to ensure the comparability of the verification results.

[0136] Then, the reverse verification prompts, image URLs, and current inference configuration (same as the initial inference) are input into the multimodal LLM, the LLM API is called to perform reverse inference, and the reverse verification results are output. The reverse verification results also include the decision result, confidence level, inference process, and key evidence.

[0137] Finally, the confidence level of the initial image-text relevance detection result is adjusted based on the reverse verification results: If the reverse verification result is consistent with the initial conclusion, and the reverse verification confidence level is greater than or equal to the initial confidence level, the initial confidence level is increased by a certain percentage (e.g., 10%); if the reverse verification result is consistent with the initial conclusion, but the reverse verification confidence level is less than the initial confidence level, the initial confidence level remains unchanged; if the reverse verification result is inconsistent with the initial conclusion, the difference in confidence levels between the two is calculated; if the difference is less than or equal to a preset threshold (e.g., 0.3), the initial confidence level is decreased by a certain percentage (e.g., 20%), and a dispute is noted during the reasoning process; if the difference is greater than the preset threshold, the initial confidence level is significantly reduced (e.g., 50%), and the case is marked as high-risk, with manual review recommended.

[0138] The self-verification mechanism reduces high-confidence errors, significantly improving the reliability of detection results. It is especially suitable for scenarios with extremely high accuracy requirements (such as medical document detection and academic paper publication detection).

[0139] Taking the consistency detection of product images and text on e-commerce platforms as an example, the application process is as follows: Step 1: Input generation conditions.

[0140] Users upload HTML-formatted product detail page documents containing product images and descriptive text, and configure system parameters; API type: Natural Language Processing Model; Domain: E-commerce; Inference depth: medium; Enable CoT inference: Yes; Enable self-verification: No.

[0141] Step 2: Document parsing.

[0142] The system parses HTML documents and extracts contextual information for each product image (product description text before and after the image, image alt attribute caption, chapter title, image position in the document, etc.) to generate structured document data.

[0143] Step 3: Rule pre-screening.

[0144] The system performs five-dimensional rule detection on each image: positional proximity (whether the image is adjacent to the product description paragraph), explicit reference detection (whether the text contains references such as "as shown in the figure" or "the following images"), chapter consistency (whether the chapter is the core description chapter of the product), semantic density (whether the context contains professional terms such as product model and material), and caption information (whether the alt attribute contains key product information). The weighted fusion rule scores are as follows: the overall score of a product main image is 0.92 (high confidence match), so the matching result is output directly and the subsequent process is skipped; the overall score of a decorative image is 0.18 (high confidence mismatch), so the mismatch result is output directly; the overall score of a functional illustration is 0.65 (uncertain), so proceed to the next step.

[0145] Step 4: Meta prompt adaptation.

[0146] The system loads an e-commerce knowledge base (key concepts: product model, material, function, size, etc.; image types: actual product images, functional diagrams, usage scenario images, etc.; matching rules: the image display content matches the product description's function, material, model, etc.). Search for similar examples: Match relevant examples from the e-commerce example library that match the functional diagrams with the product functional descriptions; Generate rule suggestions: Extract information on chapter consistency and semantic density rule matching from the rule layer and convert it into suggestion words; Select a CoT template, populate it with domain knowledge, similar examples, rule hints, and contextual summaries, and generate adaptive prompt words.

[0147] Step 5: Dynamic CoT inference.

[0148] The system calls the Natural Language Processing Model API, passing in the generated prompt words and the URL of the function diagram, and performs inference at a medium inference depth (1000 tokens, temperature coefficient 0.5); LLM reasoning follows the steps guided by the CoT template: understanding the core text (the working principle of a certain function of the product) → analyzing the image content (the structure and process shown in the functional diagram) → matching degree analysis (the structure of the diagram is consistent with the functional principle described in the text) → comprehensive judgment; The system analyzes the LLM response and extracts the decision result matching, confidence level 0.91, reasoning process, and key evidence (the components in the diagram are consistent with the functional components described in the text).

[0149] Step 6: Results integration and output.

[0150] The system collects the detection results of all images and generates a structured detection report, which includes the ID, URL, decision result, confidence level, processing method (rule / LLM), inference process (LLM processing case only), and API call cost for each image. The report shows that a total of 10 images were tested. The rule layer solved 7 images (70%), and the LLM called 3 images (30%), with an overall accuracy of 94% and a total API cost of $0.012. Output an exportable Excel report, highlighting mismatched decorative images for operations staff to review and modify.

[0151] Step 7: Feedback collection and optimization.

[0152] The operations staff checked the test results, labeled a misclassified image with the correct label, and submitted feedback. The system collects feedback data, and when the cumulative feedback reaches 100, it automatically analyzes error patterns (such as mismatched functional illustrations for some special products). New examples were generated and updated to the e-commerce domain example library, and the performance of prompt words was optimized, resulting in improved accuracy when detecting similar images next time.

[0153] Through the collaborative work of the above four-level architecture, efficient consistency detection of images and text on e-commerce product detail pages is achieved. While ensuring high accuracy, it significantly reduces API call costs and manual review workload, and significantly improves content quality inspection efficiency.

[0154] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a text-image relevance detection device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0155] like Figure 5 As shown, the image-text relevance detection device 500 of this embodiment may include: a first detection module 501, a first generation module 502, a second generation module 503, and a second detection module 504. The first detection module 501 is configured to perform multi-dimensional relevance detection on image context information and document structure data to generate an image-text relevance score; the first generation module 502 is configured to generate context summary information based on image context information and document structure data in response to determining that the image-text relevance score is not greater than a first preset score threshold and not less than a second preset score threshold; the second generation module 503 is configured to generate prompt words based on domain knowledge, similar examples, and context summary information; and the second detection module 504 is configured to input the image context information, document structure data, and prompt words into a large language model and output the image-text relevance detection result.

[0156] In this embodiment, the specific processing of the first detection module 501, the first generation module 502, the second generation module 503, and the second detection module 504 in the image-text correlation detection device 500, and the resulting technical effects, can be found in the following references: Figure 1 The relevant descriptions of steps 101-104 in the corresponding embodiments will not be repeated here.

[0157] In some optional implementations of this embodiment, the first detection module 501 is further configured to: perform relevance detection on image context information and document structure data in at least two dimensions, and generate similarity scores corresponding to at least two dimensions. The relevance detection includes the following at least two dimensions: positional proximity, explicit reference detection, chapter consistency detection, semantic density detection, and caption information detection; and perform weighted summation on the similarity scores corresponding to the at least two dimensions to generate a text-image relevance score.

[0158] In some optional implementations of this embodiment, the first detection module 501 is further configured to: determine the similarity score corresponding to positional proximity based on the distance between the image and the preceding and following paragraphs and the paragraph length; determine the similarity score corresponding to explicit citation detection by detecting preset citation patterns in the text using regular expressions; determine the similarity score corresponding to chapter consistency detection by detecting preset keywords in the chapter titles; determine the similarity score corresponding to semantic density detection by counting the number of preset technical terms per preset number of characters in the image context information; and determine the similarity score corresponding to caption information detection based on the caption length and the effective information in the caption.

[0159] In some optional implementations of this embodiment, the second generation module 503 is further configured to: load a domain knowledge base and an example library; retrieve similar examples from the example library based on context summary information; convert the matching rule information in multi-dimensional relevance detection into natural language prompt information; and fill the prompt word template with domain knowledge, similar examples, natural language prompt information, and context summary information from the domain knowledge base to generate prompt words.

[0160] In some optional implementations of this embodiment, the second detection module 504 is further configured to: initialize the configuration of the Large Language Model Application Programming Interface (LLM API); input the prompt words, image Uniform Resource Locators (URLs), and inference depth into the LLM API; configure the maximum number of tokens and temperature coefficient based on the inference depth; and call the LLM API to infer the image context information, document structure data, and prompt words to generate image-text relevance detection results.

[0161] In some optional implementations of this embodiment, the second detection module 504 is further configured to: construct a reverse verification prompt corresponding to the prompt word; input the image context information, document structure data and the reverse verification prompt into the large language model, and output the reverse verification result; and adjust the confidence of the image-text relevance detection result based on the reverse verification result.

[0162] In some optional implementations of this embodiment, the image-text relevance detection device 500 further includes: an update module, configured to acquire human feedback data corresponding to the image-text relevance detection results; and update the example library and prompt word templates based on the human feedback data.

[0163] In some optional implementations of this embodiment, the update module is further configured to: calculate the accuracy of different prompt word templates based on human feedback data; select a target prompt word template based on the accuracy; analyze the error type corresponding to the human feedback data in response to the data volume of human feedback data reaching a preset data volume threshold; extract samples from error cases based on the error type to construct a new example; and add the new example to the example library.

[0164] In some optional implementations of this embodiment, the image-text relevance detection device 500 further includes: an output module configured to output an image-text matching result in response to determining that the image-text relevance score is greater than a first preset score threshold; and to output an image-text mismatch result in response to determining that the image-text relevance score is less than a second preset score threshold.

[0165] The collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.

[0166] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0167] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0168] like Figure 6As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded into random access memory (RAM) 603 from storage unit 608. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0169] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0170] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the image-text relevance detection method. For example, in some embodiments, the image-text relevance detection method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the image-text relevance detection method described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the image-text relevance detection method by any other suitable means (e.g., by means of firmware).

[0171] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0172] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0173] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0174] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0175] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0176] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0177] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0178] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for detecting the relevance of images and text, comprising: Perform multi-dimensional relevance detection on image context information and document structure data to generate image-text relevance scores; In response to determining that the image-text relevance score is not greater than a first preset score threshold and not less than a second preset score threshold, context summary information is generated based on the image context information and the document structure data; Based on domain knowledge, similar examples, and the contextual summary information, prompt words are generated; The image context information, the document structure data, and the prompt words are input into the large language model, and the image-text relevance detection results are output.

2. The method according to claim 1, wherein, The step of performing multi-dimensional relevance detection on image context information and document structure data to generate an image-text relevance score includes: The image context information and the document structure data are subjected to relevance detection in at least two dimensions to generate similarity scores corresponding to the at least two dimensions. The relevance detection includes the following at least two dimensions: positional proximity, explicit reference detection, chapter consistency detection, semantic density detection, and caption information detection. The similarity scores corresponding to the at least two dimensions are weighted and summed to generate the image-text relevance score.

3. The method according to claim 2, wherein, The step of performing relevance detection on the image context information and the document structure data in at least two dimensions, and generating similarity scores corresponding to the at least two dimensions, includes at least two of the following: Based on the distance between the image and the preceding and following paragraphs, as well as the paragraph length, the similarity score corresponding to the positional proximity is determined; By using regular expressions to detect preset citation patterns in the text, the similarity score corresponding to explicit citation detection is determined. Detect preset keywords in chapter titles and determine the similarity score corresponding to the chapter consistency test; The number of preset technical terms in each preset number of characters in the image context information is counted to determine the similarity score corresponding to semantic density detection; Based on the length of the image caption and the effective information in the image caption, the similarity score corresponding to the image caption information detection is determined.

4. The method according to claim 1, wherein, The generation of prompt words based on domain knowledge, similar examples, and the contextual summary information includes: Load the domain knowledge base and example libraries; Based on the context summary information, retrieve the similar examples from the example library; The matching rule information in the multi-dimensional correlation detection is converted into natural language prompts. The prompt word is generated by filling the prompt word template with the domain knowledge from the domain knowledge base, the similar examples, the natural language prompt information, and the context summary information.

5. The method according to claim 1, wherein, The step of inputting the image context information, the document structure data, and the prompt words into a large language model and outputting image-text relevance detection results includes: Initialize the configuration of the Large Language Model Application Programming Interface (LLM API); Input the prompt words, image Uniform Resource Locator, and inference depth into the LLM API; Configure the maximum number of tokens and temperature coefficient based on the inference depth; The LLM API is invoked to infer the image context information, the document structure data, and the prompt words to generate the image-text relevance detection result.

6. The method according to claim 5, wherein, The step of inputting the image context information, the document structure data, and the prompt words into a large language model and outputting image-text relevance detection results also includes: Construct the reverse verification prompt corresponding to the prompt word; The image context information, the document structure data, and the reverse verification prompts are input into the large language model, and the reverse verification result is output. Based on the reverse verification results, the confidence level of the image-text correlation detection results is adjusted.

7. The method according to claim 4, wherein, The method further includes: Obtain the human feedback data corresponding to the image-text relevance detection results; Based on the human feedback data, update the example library and the prompt word template.

8. The method according to claim 7, wherein, The step of updating the example library and the prompt word template based on the human feedback data includes: Based on the aforementioned human feedback data, the accuracy rate of different prompt word templates is calculated; Based on the accuracy rate, a target prompt word template is selected; When the amount of manually fed data reaches a preset data volume threshold, the error type corresponding to the manually fed data is analyzed. Based on the error type, samples are extracted from error cases to construct new examples; Add the new example to the example library.

9. The method according to any one of claims 1-8, wherein, The method further includes: In response to determining that the image-text relevance score is greater than the first preset score threshold, an image-text matching result is output; In response to determining that the image-text relevance score is less than the second preset score threshold, an image-text mismatch result is output.

10. A device for detecting the correlation between images and text, comprising: The first detection module is configured to perform multi-dimensional relevance detection on image context information and document structure data, and generate an image-text relevance score. The first generation module is configured to generate context summary information based on the image context information and the document structure data in response to determining that the image-text relevance score is not greater than a first preset score threshold and not less than a second preset score threshold. The second generation module is configured to generate prompt words based on domain knowledge, similar examples, and the context summary information; The second detection module is configured to input the image context information, the document structure data, and the prompt words into the large language model and output the image-text relevance detection result.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

12. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-9.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images