Image-text detection model training method and device, electronic equipment and storage medium

Abstract

The invention provides an image-text detection model training method and device, electronic equipment and a storage medium, relates to the technical field of data processing, in particular to the fields of computer vision, natural language processing and the like, and can be used for application scenes such as harmful image-text detection and classification. According to the specific scheme, according to a thinking chain data set, a visual agent, a text agent and a fusion agent in a to-be-trained basic model are trained, and the basic model is obtained; inputting the original data set into the basic model, screening an output initial result through a triple confidence threshold, and constructing a deep thinking training data set; performing data simplification processing on the deep thinking training data set to generate a quick answer training data set; and using the deep thinking training data set and the fast answer training data set to train a deep thinking channel and a fast answer channel of the to-be-trained image-text detection model in parallel to obtain the image-text detection model. According to the scheme, the detection capability of harmful image-text contents can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, particularly to the fields of computer vision and natural language processing, and can be used in application scenarios such as harmful image and text detection and classification. Specifically, it relates to image and text detection model training methods, devices, electronic devices, and storage media. Background Technology

[0002] Existing research on harmful meme detection primarily relies on the representation and fusion of multimodal features, including visual and text extraction methods. Simultaneously, some methods utilize visual language models, such as visual question answering and image captioning techniques, to extract deeper semantic information. However, existing methods still have certain limitations: they cannot accurately and efficiently detect harmful memes, lack a deep understanding of the specific context of the meme, resulting in poor performance in subtle or ambiguous situations, and limited interpretability. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, and storage medium for training an image and text detection model.

[0004] According to a first aspect of this disclosure, a method for training an image-text detection model is provided, comprising: training a visual agent, a text agent, and a fusion agent in a base model to be trained based on a thought chain dataset to obtain a base model; inputting the original dataset into the base model and filtering its initial output results through a triple-reset reliability threshold to construct a deep thinking training dataset; performing data simplification processing on the deep thinking training dataset to generate a quick response training dataset; and using the deep thinking training dataset and the quick response training dataset, training the deep thinking channel and the quick response channel of the image-text detection model to be trained in parallel to obtain the image-text detection model.

[0005] According to a second aspect of this disclosure, a text-image detection method is provided, comprising: acquiring text-image data to be detected; analyzing the text-image data to be detected and determining the recognition difficulty; generating prompt words according to a prompt word template based on the recognition difficulty; inputting the text-image data to be detected and the prompt words into a text-image detection model, so that the text-image detection model outputs the detection result; the text-image detection model is trained by the method of the first aspect.

[0006] According to a third aspect of this disclosure, a training apparatus for an image-text detection model is provided, comprising: a basic model training module, used to train a visual agent, a text agent, and a fusion agent in the basic model to be trained based on a thought chain dataset, to obtain a basic model; a first training data construction module, used to input the original dataset into the basic model, filter its initial output results through a triple-reset reliability threshold, and construct a deep thinking training dataset; a second training data construction module, used to perform data simplification processing on the deep thinking training dataset to generate a fast response training dataset; and a detection model training module, used to train the deep thinking channel and the fast response channel of the image-text detection model to be trained in parallel using the deep thinking training dataset and the fast response training dataset, to obtain the image-text detection model.

[0007] According to a fourth aspect of this disclosure, an image-text detection device is provided, comprising: a data acquisition module for acquiring image-text data to be detected; a data analysis module for analyzing the image-text data to be detected and determining the recognition difficulty; a prompt word generation module for generating prompt words according to a prompt word template based on the recognition difficulty; and a model inference module for inputting the image-text data to be detected and the prompt words into an image-text detection model, so that the image-text detection model outputs a detection result; the image-text detection model is trained using the method of the first aspect.

[0008] According to a fifth aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the methods described in the embodiments of this disclosure.

[0009] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0010] According to a seventh aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0011] The scheme disclosed herein integrates image and text information, designs an end-to-end deep thinking network and a dual-channel training mechanism, significantly improves the detection capability of harmful image and text content, achieves more efficient, accurate and flexible multimodal contextual understanding, and enhances the model's versatility and generalization ability, providing solid technical support for content security systems.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0014] Figure 1 This is a flowchart illustrating the image and text detection model training method according to an embodiment of the present disclosure;

[0015] Figure 2 This is a schematic diagram of the structure of a multi-agent system according to an embodiment of the present disclosure;

[0016] Figure 3 This is a schematic diagram of the data verification and sampling process according to an embodiment of this disclosure;

[0017] Figure 4 This is a schematic diagram of the parallel training process according to an embodiment of the present disclosure;

[0018] Figure 5 This is a schematic flowchart of a text and image detection method according to an embodiment of the present disclosure;

[0019] Figure 6 This is another schematic flowchart of the image and text detection method according to an embodiment of the present disclosure;

[0020] Figure 7 This is a schematic diagram of the structure of a text detection model training device according to an embodiment of the present disclosure;

[0021] Figure 8 This is a schematic diagram of the structure of a graphic detection device according to an embodiment of the present disclosure;

[0022] Figure 9 This is a schematic diagram of a scenario for training a text detection model according to an embodiment of the present disclosure;

[0023] Figure 10 This is a schematic diagram of a scene according to the image and text detection method of the present disclosure;

[0024] Figure 11 This is a structural diagram of an electronic device used to implement the image and text detection model training method and / or image and text detection method of the embodiments of this disclosure. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0026] In this document, the term "and / or" merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The term "at least one" in this document indicates any combination of at least two of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this document refer to and distinguish between multiple similar technical terms, not to restrict the order or to limit there to only two. For example, "first feature" and "second feature" refer to two categories / two features; the first feature can be one or more, and the second feature can also be one or more.

[0027] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0028] Before introducing the technical solutions of the embodiments of this disclosure, the technical terms that may be used in this disclosure will be further explained:

[0029] A meme, a concept originating from Richard Dawkins, refers to a unit of cultural information that can be disseminated, similar to the role of genes in biological evolution. A meme can be any cultural symbol, idea, habit, or concept that can be disseminated, such as emojis and other graphic data, capable of widespread dissemination online and influencing people's thoughts and behaviors. In harmful meme detection, researchers focus on memes with potentially harmful effects, requiring algorithms and technologies to identify and monitor these memes to reduce their negative impact on users and society.

[0030] Harmful meme detection is a challenging task, and existing research primarily focuses on the representation and fusion of multimodal features. These methods include visual object detection, text extraction based on optical character recognition, and simple visual-text fusion techniques. However, research shows that methods relying solely on low-level feature representations have limitations in accuracy and efficiency. In recent years, with the rise of visual language models, techniques such as visual question answering and image captioning have been introduced into harmful meme detection to extract deeper semantic information. The development of these methods has brought new ideas and opportunities to harmful meme detection. However, despite recent progress, current methods still face some key challenges. Meme propagation often relies on implicit context, background knowledge, and cultural references to convey harmful meanings, and most existing methods lack a deep understanding of the specific context of memes. This leads to poor performance in detecting harmful memes in subtle or ambiguous situations and provides limited interpretability, potentially undermining user trust and understanding of their decisions. Against this backdrop, the development of multimodal large language models offers new possibilities for solving this problem. By deeply considering the thought chain of the model, it is expected to improve the accuracy and interpretability of harmful detection.

[0031] To at least partially address one or more of the aforementioned problems and other potential issues, this disclosure proposes a training method for an image and text detection model. This method integrates image and text information, designs an end-to-end deep thinking network and a dual-channel training mechanism, significantly improves the detection capability of harmful image and text content, achieves more efficient, accurate and flexible multimodal contextual understanding, and enhances the model's versatility and generalization ability, providing solid technical support for content security systems.

[0032] This disclosure provides a method for training an image-text detection model. Figure 1 This is a flowchart illustrating a text detection model training method according to an embodiment of the present disclosure. This method can be applied to a text detection model training device. The text detection model training device is located in an electronic device. This electronic device includes, but is not limited to, fixed devices and / or mobile devices. For example, fixed devices include, but are not limited to, servers, which can be cloud servers or ordinary servers. Mobile devices include, but are not limited to, harmful text detection and classification devices, which can be mobile phones, tablets, vehicle terminals, etc. In some possible implementations, the text detection model training method can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 As shown, the training method for this image-text detection model includes:

[0033] S101. Based on the thought chain dataset, train the visual agent, text agent, and fusion agent in the base model to be trained to obtain the base model.

[0034] S102. Input the original dataset into the base model, filter the initial output results by setting three confidence thresholds, and construct the deep thinking training dataset.

[0035] S103. Simplify the deep thinking training dataset to generate a quick response training dataset.

[0036] S104. Using the Deep Thinking training dataset and the Quick Response training dataset, train the Deep Thinking channel and the Quick Response channel of the image and text detection model to be trained in parallel to obtain the image and text detection model.

[0037] Here, the thought chain dataset refers to a collection of samples containing complex reasoning processes and related explanations. In this embodiment of the disclosure, each sample in the thought chain dataset not only provides input and output, but also records the intermediate reasoning chain in detail, allowing the model to not only learn the correspondence between input and output, but also learn how to think.

[0038] Here, the visual agent refers to the model part that focuses on processing and understanding image information. In this embodiment of the disclosure, the visual agent can be a model such as a convolutional neural network (CNN) or a vision transformer (ViT).

[0039] Here, the text agent refers to the model part that focuses on processing and understanding textual information. In this embodiment of the disclosure, the text agent can be a natural language processing model such as Bi-directional Encoder Representations from Transformers (BERT).

[0040] Here, the fusion agent refers to the model part that simultaneously processes and understands image and text information. In this embodiment of the disclosure, the fusion agent can be a multimodal attention mechanism or a fusion alignment network, etc.

[0041] Here, the base model refers to a multimodal model that has undergone preliminary training and possesses a certain level of image and text understanding and reasoning capabilities. In this embodiment of the disclosure, the base model includes a visual agent, a text agent, and a fusion agent.

[0042] In this embodiment, a thought chain dataset containing rich reasoning processes can be prepared first, with each sample associated with a clear visual input, text input, and detailed reasoning chain output. Then, after inputting the thought chain dataset into the base model to be trained, the three agent parts of the model can be trained separately. For example, the visual agent can learn visual features based on image data; the text agent can learn language features based on text data; and the fusion agent can learn how to jointly understand image and text information and reasoning chains through multimodal interaction. Specifically, the training process can employ multi-task learning and an end-to-end approach, enabling the three agents to collaboratively optimize and ultimately obtain a base model capable of handling complex image and text reasoning tasks. The above is merely an illustrative example and does not constitute a limitation on all possible scenarios for obtaining the base model; it is simply not exhaustive.

[0043] Here, the original dataset refers to a widely collected collection of image and text samples that has not undergone in-depth screening. In this embodiment of the disclosure, the original dataset may contain noisy or unevenly challenging samples.

[0044] Here, the confidence threshold refers to the screening criterion set for the confidence score of the model output. In this embodiment of the disclosure, the confidence threshold can be used to filter the output results and retain samples with appropriate confidence levels.

[0045] Here, the initial result refers to the output of the base model after processing the original dataset samples. In this embodiment of the disclosure, the initial result may include detection results, inference chains, confidence scores, etc.

[0046] Here, the deep thinking training dataset refers to a dataset constructed using selected high-quality, high-complexity output results. In this embodiment of the disclosure, the deep thinking training dataset can be used to subsequently train the deep inference capabilities of the model.

[0047] In this embodiment, the original dataset can be input into a pre-trained base model. The model then outputs a preliminary inference result for each sample and determines a confidence score for each result. By setting three or more confidence thresholds, different levels of model output results are filtered. Finally, samples with confidence scores falling within the ideal range can be selected to constitute a high-quality deep thinking training dataset. In particular, the data in the deep thinking training dataset better represents real-world tasks requiring deep reasoning and complex understanding, which helps in training the model's capabilities. The above is merely an illustrative example and does not limit the possibilities for constructing a deep thinking training dataset; it is simply not exhaustive.

[0048] Here, the fast-response training dataset refers to a dataset focused on the model's efficiency and rapid response capabilities, generated by simplifying and optimizing the deep thinking training dataset. In this embodiment, the fast-response training dataset contains structured and concise input and output samples. Specifically, each sample in the fast-response training dataset corresponds to a specific sample in the deep thinking training dataset.

[0049] In this embodiment, the deep thinking training dataset can first be simplified by removing redundant and complex inference chains, retaining only the key inputs and target outputs. Specifically, methods such as automatic summarization and information extraction can be used to make the data more concise and facilitate efficient model learning. Finally, the simplified data is used to construct a fast-response training dataset, specifically for training the model's rapid response capabilities. The above is merely an illustrative example and does not represent all possible scenarios for generating a fast-response training dataset; it is simply not exhaustive.

[0050] Here, the deep thinking channel refers to a training branch of the model that is specifically designed for complex reasoning, relational explanation, and other challenging tasks, enabling interpretable output.

[0051] Here, the fast response channel refers to another training branch of the model, which is specifically designed for fast tasks such as efficient and rapid response, enabling instant discrimination and output.

[0052] Here, the image and text detection model refers to the final model obtained after multi-channel training, which is a multimodal detection model that combines deep reasoning and fast response capabilities.

[0053] In this embodiment, the deep thinking training dataset and the fast response training dataset can be used in parallel to train the deep thinking channel and the fast response channel of the image-text detection model to be trained. Exemplarily, multi-task learning, branching structures, or collaborative optimization methods can be employed to allow the two channels to jointly improve the model's overall capability on different datasets. The final result is an image-text detection model with multi-channel decision-making capabilities. The above is merely an illustrative example and does not constitute a limitation on all possible scenarios for obtaining the image-text detection model; it is simply not exhaustive.

[0054] The technical solution of this disclosure, trained using a thought chain dataset, enables the model to perform not only basic recognition but also deeper reasoning and interpretation capabilities. The multi-agent structure allows the model to efficiently extract visual and textual information separately, and by fusing agents, it achieves stronger cross-modal understanding, improving overall performance and better filtering and generating high-difficulty, high-quality data. A three-stage reliability threshold strategy accurately selects the most representative complex samples, generating a high-quality deep thinking training dataset that helps improve the model's performance on more challenging tasks. Generating a fast-response training dataset allows the model to not only possess deep reasoning but also make efficient and accurate judgments quickly in common scenarios. Simplified data training accelerates the model's learning process, improving its response speed and stability in practical applications. Parallel training of the deep thinking and fast-response channels of the image-text detection model makes the model structure more flexible, enabling better performance in different types of tasks. The model can intelligently select channels based on task difficulty, balancing complex reasoning and rapid response, thus improving adaptability in real-world scenarios.

[0055] In some embodiments, training the visual agent, text agent, and fusion agent in the base model to be trained to obtain the base model includes: performing optical character recognition on image and text data to obtain image data and text data; optimizing the parameters of the visual agent using the image data as input and through a triple reward function; optimizing the parameters of the text agent using the text data as input and through a triple reward function; and optimizing the parameters of the fusion agent using the fusion features of the image data and text data as input and through a triple reward function.

[0056] Here, text data refers to text information extracted from image and text data using the Optical Character Recognition (OCR) method. In this embodiment of the disclosure, the text data is the text content separated from the original image, which can be used for subsequent text feature processing and analysis.

[0057] Here, image data refers to pure image information removed from or independent of the text portion of image-text data. In this embodiment of the disclosure, image data may include the pixels, colors, structure, visual content, etc., of an image, used by a visual intelligent agent for visual feature extraction and modeling.

[0058] In this embodiment, the original image and text data can first undergo OCR processing to extract text content from the image, obtaining text data, and the remaining image portion can be used as image data. The above is merely an illustrative example and is not intended to limit all possible scenarios for obtaining image and text data; it is simply not exhaustive.

[0059] Here, the reward function refers to a mathematical formula or rule used to quantify the performance of an agent in its environment. In this embodiment, the triple reward function sets three different reward criteria for each optimization step, enabling the model to comprehensively optimize multiple capabilities during training.

[0060] In this embodiment of the disclosure, the triple reward function includes an accuracy reward function, a precision reward function, and a format conformity reward function; wherein, the accuracy reward function is used to calculate the binary classification matching degree between the agent's output and the judgment result; the precision reward function is used to calculate the F1 score of fine-grained classification; and the format conformity reward function is used to verify the consistency between the output and the logical framework of the thinking process.

[0061] In this embodiment, image data can be input into a visual agent such as CNN or ViT, and then trained using a triple reward function. For example, the first reward may focus on recognition accuracy, the second on scene understanding, and the third on content relevance. This is merely an illustrative example and does not represent all possible scenarios for optimizing the parameters of the visual agent; it is simply not exhaustive.

[0062] In this embodiment, the text data obtained from OCR can be input into a text agent such as BERT, and then trained using a triple reward function. For example, the first reward may focus on semantic accuracy, the second on text integrity, and the third on consistency with image content. This is merely an illustrative example and does not represent all possible scenarios for optimizing the parameters of the text agent; it is simply not exhaustive.

[0063] In this embodiment, features can first be extracted from the visual agent and the text agent, and then fused using a fusion mechanism. Exemplarily, the fusion mechanism can be implemented using features concatenation, cross-modal attention, or other methods. Subsequently, the fused features can be input into the fused agent, and then trained using a triple reward function. Exemplarily, the first reward can focus on the fusion effect, the second reward on cross-modal consistency, and the third reward on dimensions such as reasoning ability. The above is merely an illustrative example and does not represent all possible scenarios for optimizing the parameters of the fused agent; it is simply not exhaustive.

[0064] In this way, each agent is trained using multiple reward functions, which can balance multiple objectives such as accuracy, relevance, and higher-order reasoning, improving the overall performance of the model and achieving end-to-end optimization. The visual, textual, and fusion agents are optimized independently and then merged as a whole, which helps the model perform well in both single-modal and multi-modal tasks, enhancing its multi-dimensional capabilities. The triple reward function ensures that model training no longer focuses solely on a single metric but considers multiple aspects, improving generalization ability and practical application effectiveness. The reward mechanism of the fusion agent enables the model to better combine visual and textual information, achieving stronger image-text linkage analysis and reasoning capabilities.

[0065] In some embodiments, before inputting the original dataset into the base model, the method further includes: based on a dynamic cultural lexicon, performing localized expression annotation on each original data in the original dataset to obtain a gold label corresponding to each original data.

[0066] Here, a dynamic cultural lexicon refers to a collection of localized vocabulary and expressions that can be continuously updated based on the cultural background of the target region.

[0067] Here, the "gold standard" label refers to the most authentic and accurate reference label assigned to each piece of data by experts or authoritative labelers according to strict standards. In this embodiment of the disclosure, the gold standard label includes: a binary label of harmfulness, a sub-category label, and a set of evidence fragments.

[0068] The harmfulness binary label refers to a binary classification label used to determine whether each piece of data belongs to harmful content. In this embodiment of the disclosure, the harmfulness binary label can be represented as "harmful / harmless" or "1 / 0".

[0069] Here, the subcategories refer to category tags that further subdivide harmful content. In this embodiment of the disclosure, each piece of data may carry one or more more specific category identifiers.

[0070] In this context, the set of evidence fragments refers to image regions, text fragments, or a combination of both used to support subcategorical labeling. In this embodiment, the set of evidence fragments is the specific content indicated by the labeler that can serve as a basis for judgment.

[0071] In this embodiment, each piece of raw data can be localized semantically and standardized in its annotation, incorporating local slang and sensitive words, to ensure the tags are adaptable to the local context. Furthermore, professional annotators / expert teams can assign authoritative "harmful binary tags" and "sub-category tags" to each piece of data based on the localized content. Simultaneously, annotators can select / mark content fragments in images or text as evidence, forming a set of evidence fragments. Finally, each piece of raw data possesses a closely corresponding, authoritative, accurate, and locally applicable "golden tag," greatly enhancing the reliability of training and evaluation. The above is merely an illustrative example and does not constitute a limitation on all possible scenarios for obtaining the golden tag; it is simply not exhaustive.

[0072] In some implementations, reward strategies can be optimized based on accuracy, precision, and format conformity. Visual agents, text agents, and fusion agents can be trained using image data, text data, thought processes, and judgment results, respectively. For example, reward strategies can be optimized based on accuracy, precision, and format conformity, using image data and judgment results for supervised training of the visual agent, while simultaneously using thought processes for association enhancement. Similarly, reward strategies can be optimized based on accuracy, precision, and format conformity, using text data and judgment results for supervised training of the text agent, while simultaneously using thought processes for association enhancement. Likewise, reward strategies can be optimized based on accuracy, precision, and format conformity, using image data, text data, and judgment results for supervised training of the fusion agent, while simultaneously using thought processes for association enhancement.

[0073] Thus, the gold label, based on a local thesaurus and expert annotations, greatly improves the accuracy and practicality of data annotation, reducing misjudgments caused by cultural differences in the model. It focuses not only on accuracy but also on precision and formatting standards, helping the model output high-standard, highly usable detection results. Information about the thought process is incorporated into the training of all three agents, helping the model not only output results but also reasonable explanations, enhancing the model's interpretability and reliability. The collaborative training of the visual, textual, and fusion agents effectively improves the comprehensive capabilities in both single-modal and cross-modal scenarios. Multi-dimensional reward function optimization not only improves model performance but also flexibly adapts to various needs in real-world scenarios.

[0074] In some embodiments, the original dataset is input into the base model, and the initial results of its output are filtered by a triple-reset reliability threshold to construct a deep thinking training dataset, including: inputting samples from the original dataset into the base model to generate initial results, the initial results including harmfulness judgment conclusions, sub-classification probability distributions and evidence chains of reasoning process; performing triple-reset reliability verification on the initial results based on the gold label to construct a deep thinking training dataset that meets the triple-reset reliability threshold.

[0075] Here, the initial results include the harmfulness judgment conclusion, the sub-categorization probability distribution, and the evidence chain of the reasoning process.

[0076] The harmfulness judgment conclusion refers to the model's final determination of whether the input sample is harmful. In this embodiment of the disclosure, the harmfulness judgment conclusion is a binary classification used to determine whether the content contains illegal, sensitive, or harmful elements.

[0077] Here, the sub-category probability distribution refers to the model's probability output for each sub-category. In this embodiment of the disclosure, the sub-category probability distribution is a distribution of a set of probability values, reflecting the likelihood that the model considers the sample to belong to each sub-category.

[0078] The evidence chain in the reasoning process refers to the chain of reasoning or evidence automatically generated or extracted by the model when making a judgment, used to support the judgment conclusion. In this embodiment of the disclosure, the evidence chain in the reasoning process may include image blocks, text fragments, reasoning steps, etc., used to explain why the model made a certain judgment, thereby improving the interpretability of the output.

[0079] In this embodiment, each sample in the original dataset can be sequentially input into a pre-trained base model, which then outputs an initial result for each sample. This is merely an illustrative example and does not represent all possible scenarios for generating initial results; it is simply not exhaustive.

[0080] Here, confidence level refers to the assessment of the credibility of the output result. In this embodiment of the disclosure, confidence level can be expressed in the form of probability or score.

[0081] In this embodiment, confidence tests can first be performed on each initial result output based on different dimensions and the golden label. Only samples whose three-fold reliability meets the preset criteria are selected for the deep thinking training dataset. For example, the three-fold reliability verification process can first determine whether the confidence of the harmfulness judgment reaches a set threshold, then determine whether the credibility of a specific sub-category in the sub-class probability distribution reaches a threshold, and finally determine whether the confidence of the evidence chain in the reasoning process reaches a threshold. Finally, samples that meet the threshold conditions in each aspect and have a clear reasoning chain can be collected as high-quality training samples for subsequent deep reasoning training of the model. The above is merely an illustrative example and does not constitute a limitation on all possible scenarios for constructing the deep thinking training dataset; it is simply not exhaustive.

[0082] Thus, multi-layered confidence screening ensures that the samples in the training set are real, clear, and reliable across multiple dimensions, significantly improving the effectiveness of the training data. Simultaneously, the confidence test of the evidence chain in the reasoning process promotes model transparency and traceability. Through triple screening of harmfulness, fine classification, and reasoning chain, samples with weak model confidence or ambiguous reasoning chains can be effectively eliminated, reducing the impact of noise during training. This also helps the model learn higher-quality and more complex reasoning processes, improving its performance in challenging application scenarios.

[0083] In some embodiments, a triple-reset reliability verification is performed on the initial result based on the gold label, including: when the harmfulness judgment conclusion is consistent with the harmfulness binary label in the gold label, the logistic function probability value of the harmfulness judgment conclusion is extracted as the detection confidence; if they are inconsistent, the detection confidence is recorded as an invalid value; when the category with the highest probability in the sub-category probability distribution is the same as the sub-category label in the gold label, the maximum probability value is extracted as the sub-category confidence; if the categories are different, the sub-category confidence is recorded as an invalid value; the intersection-union ratio (IUGR) similarity between the evidence chain of the reasoning process and the evidence fragment set in the gold label is calculated, and the IUGR similarity is used as the evidence overlap, which is equal to the number of evidence fragments shared by both parties divided by the total number of all evidence fragments of both parties; the sample is adopted only when the following conditions are met simultaneously: the detection confidence is a valid value and is greater than a preset detection threshold; the sub-category confidence is a valid value and is greater than a preset classification threshold; and the evidence overlap is greater than a preset overlap threshold.

[0084] Here, the logistic function probability value refers to the probability value output by the model processed by the logistic function in a binary classification problem. In this embodiment of the disclosure, the logistic function probability value can be represented by the sigmoid function probability value.

[0085] In this embodiment, the harmfulness judgment conclusion output by the model can first be compared with the harmfulness binary label in the gold label. If they match, the sigmoid probability value output by the model is extracted as the detection confidence level; otherwise, if they do not match, the detection confidence level is directly recorded as invalid. For example, the detection confidence level can be recorded as null or -1 to indicate that the detection confidence level is invalid. The above is only an illustrative example and is not intended to limit all possible cases for determining the detection confidence level; it is simply not exhaustive.

[0086] In this embodiment, we can first check whether the category with the highest probability in the model's predicted sub-category probability distribution matches the sub-category label in the gold label. If they match, this highest probability is extracted as the sub-category confidence score; otherwise, if they do not match, the sub-category confidence score is directly recorded as invalid. For example, the sub-category confidence score can be recorded as Null or -1 to indicate that the sub-category confidence score is invalid. The above is only an illustrative example and is not intended to limit all possible cases for determining the sub-category confidence score; it is simply not exhaustive.

[0087] Here, the intersection-union ratio (CIU) similarity refers to the degree of similarity between two sets. In this embodiment of the disclosure, the CIU similarity can be represented by Jaccard similarity.

[0088] In this embodiment of the disclosure, the evidence chain of the reasoning process output by the model can be compared with the set of evidence fragments of the gold label, and the Jaccard similarity between the two can be calculated. This similarity is used as the degree of evidence overlap. The above is only an illustrative example and is not intended to limit all possible situations for determining the degree of evidence overlap; it is simply not exhaustive.

[0089] For example, the process of calculating the Jaccard similarity between the chain of evidence and the set of evidence fragments in the reasoning process can be represented by the following formula:

[0090]

[0091] In the formula, J(E) Golden E Reasoning ) represents Jaccard similarity; E Reasoning E represents the chain of evidence in the reasoning process. Golden This represents a set of evidence fragments.

[0092] In this embodiment of the disclosure, a sample is adopted into the Deep Thinking training dataset only when all three of the following conditions are met: the detection confidence is a valid value and is greater than a preset detection threshold; the sub-classification confidence is a valid value and is greater than a preset classification threshold; and the evidence overlap is greater than a preset overlap threshold. The above is merely an illustrative example and is not intended to limit all possible scenarios for performing triple-reset confidence verification; it is simply not exhaustive.

[0093] Thus, through triple verification of detection confidence, fine-classification confidence, and evidence overlap, only samples that are highly reliable in terms of model judgment, classification, and interpretation of evidence are adopted. This greatly improves the accuracy and practicality of the Deep Thinking training dataset, achieving multi-dimensional, high-standard filtering. Incorporating evidence overlap as a threshold effectively drives the model not only to output correct results but also to provide reasonable and verifiable explanatory evidence, enhancing the model's interpretability and oversight. If the model's conclusion or classification does not match the gold label, or if the evidence chain has low overlap with the label, the sample will be excluded, preventing erroneous samples from entering the training set and reducing the model's learning of low-quality or incorrect patterns.

[0094] In some embodiments, the image-text detection model training method further includes performing data augmentation on each sample that meets the triple confidence threshold.

[0095] Here, data augmentation refers to the process of performing certain transformations, perturbations, synthesis, expansions, etc., on existing training data samples to generate more new samples. In this embodiment of the disclosure, the data augmentation methods include at least: text recognition adversarial perturbation and text injection.

[0096] In this embodiment of the disclosure, data augmentation processing can be performed on each sample. Exemplarily, various types of perturbations can be introduced into the original image or text, such as adding noise or replacing some characters, to achieve data augmentation processing for text recognition against perturbations. Specifically, it is ensured that the original labels remain applicable after perturbation. Similarly, new text content can be injected into the image or text of the original sample, such as adding phrases, slogans, or advertising slogans to the image, or inserting harmless or harmful additional statements into the original image area. Specifically, the injected content can be randomly generated, templated, or selected based on domain knowledge to ensure that the labels remain unchanged or are specifically enhanced. The above are merely illustrative examples and are not intended to limit all possible cases of data augmentation processing; they are simply not exhaustive.

[0097] Thus, the diverse samples obtained through perturbation and injection enable the model to learn richer input features, cope with more real-world scenarios, and improve its generalization ability. Furthermore, adversarial perturbation in text recognition can significantly improve the model's tolerance to input noise, OCR errors, malicious attacks, etc., reducing the false positive rate in practical applications. Text injection allows the model to adapt to diverse forms of text content, without losing its discriminative ability due to partial text changes, enhancing its ability to recognize the same semantics in different forms. Diverse data augmentation can increase the diversity of training, which is beneficial for generalizing to unseen new data.

[0098] In some embodiments, data simplification processing is performed on the deep thinking training dataset, including: removing intermediate inference steps from the deep thinking training data; and compressing the multimodal fusion analysis conclusions into a single label.

[0099] Here, intermediate reasoning steps refer to the intermediate links in the specific reasoning process or reasoning chain generated by the model in the process of judging the harmfulness or subcategorization of image and text samples.

[0100] In this embodiment, each sample in the deep thinking training dataset can be traversed first. Then, the recorded reasoning processes at each level, explanations, and reasoning branches can be deleted, retaining only the final output, such as harmful conclusions, sub-classification conclusions, and evidence fragments. The above is merely an illustrative example and is not intended to limit the removal of all possible intermediate reasoning steps; it is simply not exhaustive.

[0101] In this embodiment of the disclosure, for the analysis conclusions output by the fusion agent (which may contain multiple elements, such as multiple labels, multiple probabilities, multiple reasons, etc.), only one explicit main label is retained, or the most important label is selected by rules or manually. The above is only an illustrative example and is not intended to limit all possible cases of compressing multimodal fusion analysis conclusions; it is simply not exhaustive.

[0102] Thus, by removing redundant reasoning steps and multiple labels, the amount of data is reduced, and the complexity of the model input is lowered, making it suitable for applications requiring efficient training or inference. The simplified data only needs to learn the mapping between the input and the final conclusion, eliminating the need for step-by-step reasoning and reducing the difficulty of model design and training.

[0103] In some embodiments, the deep thinking training dataset and the quick response training dataset are used to train the deep thinking channel and the quick response channel of the image and text detection model to be trained in parallel to obtain the image and text detection model. This includes: merging the deep thinking training dataset and the quick response training dataset according to a preset mixing ratio to generate an image and text detection training dataset; inputting the image and text detection training dataset into the image and text detection model to be trained, and jointly training the deep thinking channel and the quick response channel to obtain the image and text detection model.

[0104] Here, the image and text detection training dataset refers to a dataset that integrates multiple samples and is prepared for training the image and text detection model. In this embodiment of the disclosure, the image and text detection training dataset is a set obtained by merging the deep thinking training dataset and the quick response training dataset according to a certain mixing ratio.

[0105] In this embodiment of the disclosure, samples can be drawn from two types of datasets in a proportional manner and mixed to generate an image-text detection training dataset. For example, a mixing ratio of 7:3 can be selected, or other mixing ratios can be chosen according to the actual situation; this disclosure does not limit this. In particular, during the merging process, it is necessary to ensure that the sample labels, formats, etc., are uniform and standardized to facilitate subsequent model training. The above is merely an illustrative example and does not constitute a limitation on all possible situations for generating an image-text detection training dataset; it is simply not exhaustive.

[0106] In this embodiment, a synthesized image-text detection training dataset is used as input and fed into the image-text detection model to be trained. The model internally includes a deep thinking channel and a quick response channel, both of which participate in model optimization simultaneously. These two channels can be complementaryly improved through multi-task loss, channel weights, etc. The above is merely an illustrative example and is not intended to limit all possible outcomes of the image-text detection model; it is simply not exhaustive.

[0107] In this way, the model can handle both complex reasoning and deep thinking tasks with multiple chains of evidence, as well as efficient daily rapid screening tasks, meeting the needs of different practical applications. Hybrid training allows the model to fully absorb diverse sample features under limited computing power, improving generalization ability and reducing overfitting.

[0108] In some embodiments, the image and text detection training dataset is input into the image and text detection model to be trained, and the deep thinking channel and the quick response channel are jointly trained to obtain the image and text detection model. This includes: constructing a composite loss function that includes interpretability loss, dual-channel consistency loss and inference efficiency loss; and updating the parameters of the deep thinking channel and the quick response channel based on the image and text detection training dataset and the composite loss function through backpropagation to obtain the image and text detection model.

[0109] Here, a composite loss function refers to a function that combines multiple losses with different emphases in a weighted manner, and can be uniformly used for optimizing the model training objective. In this embodiment, the composite loss function may include interpretability loss, dual-channel consistency loss, and inference efficiency loss.

[0110] In this embodiment, interpretability loss can be designed first. For example, the overlap between the evidence chain / intermediate inference results generated by the model and the evidence chain of the gold label can be compared to measure the accuracy of the inference explanation. Subsequently, dual-channel consistency loss can be designed. For example, the output labels, feature representations, probability distributions, etc., of the deep thinking channel and the quick response channel under the same input can be made as close as possible, so that the consistency of the two channel outputs can be measured using methods such as mean squared error and contrastive loss. Next, inference efficiency loss can be designed. For example, for the deep thinking channel, the number of steps, time consumption, and computational resources of each inference can be statistically analyzed to penalize excessively long inference chains and excessive computational overhead; the loss can be proportional to the number of inference steps and latency. Finally, weights can be assigned to each sub-item of the loss according to the task focus. The above is only an illustrative example and does not constitute a limitation on all possible cases for constructing a composite loss function; it is simply not exhaustive here.

[0111] For example, the composite loss function can be expressed by the following formula:

[0112] L total =λ t ·L explain +μ·L consistency +η t ·L efficiency

[0113] In the formula, μ is a fixed weighting coefficient, and λ t η t L is a phased adjustable coefficient. explain L represents the interpretability loss. consistency L represents the dual-channel consistency loss. efficiency This indicates a loss of reasoning efficiency.

[0114] Specifically, during training, λ can be adjusted in stages. t and η t For example, in the first stage, λ can be set... t =0.9, η t =0.1, achieving strong constraint on the precision of the thought chain; the second stage can let λ t =0.5, η t =0.5, achieving balanced dual-channel performance; the third stage can set λ to 0.5. t =0.3, η t =0.7, achieving optimized high-speed channel efficiency.

[0115] In this embodiment, the image-text detection training dataset can first be input into the model to be trained. Then, forward propagation is performed, with the deep thinking channel processing the samples and outputting the inference chain, labels, and interpretation results; simultaneously, the quick response channel processes the samples and outputs direct labels and probabilities. Next, interpretability loss, consistency loss, and inference efficiency loss can be calculated separately and summed according to set weights to obtain the total loss. Then, backpropagation can be performed based on the total loss to update the model parameters of the deep thinking channel and the quick response channel. Finally, the above process can be repeated until the model converges or reaches a preset effect, and the final model is used as the image-text detection model. The above is merely an illustrative example and is not intended to limit all possible cases for obtaining the image-text detection model; it is simply not exhaustive.

[0116] In this way, by explicitly optimizing the alignment between the reasoning explanation and the true labels, the model can not only output conclusions but also human-understandable reasoning chains, thus improving its interpretability. Consistency loss promotes the alignment of knowledge between deep thinking and rapid responses, achieving knowledge complementarity, enhancing the model's comprehensive capabilities and stability, and integrating the advantages of deep and shallow channels.

[0117] In some implementations, a text and image detection model with basic cross-modal reasoning capabilities needs to be built in the early stages of model training. To this end, a multi-agent system is used for cold-start training of the model, combined with supervised fine-tuning (SFT) to improve its basic capabilities. Specifically, a multi-agent system can be built first, comprising an image harmfulness detection agent (visual agent), a text harmfulness detection agent (text agent), and a cross-modal fusion agent (fusion agent). Then, through the collaboration of these agents, the input images, text, and their combinations are judged and reasoned about separately. Subsequently, the qwen-v1 model can be used as the base model, and step-by-step reasoning prompts can be designed to guide the model to sequentially judge the harmfulness of images, text, and image-text combinations, helping the model form a clear reasoning chain. Finally, supervised fine-tuning can be performed on the model using manually labeled training samples, enabling it to master the basic capabilities of cross-modal feature extraction and reasoning. By constructing data offline, the reasoning during pre-training is completely end-to-end, without the addition of additional pipelines, making it more lightweight. For example, Figure 2 A schematic diagram of the structure of a multi-agent system is shown, such as... Figure 2 As shown, it includes a master intelligent agent 201, a visual intelligent agent 202, a text intelligent agent 203, and a fusion intelligent agent 204. Among them, the master intelligent agent 201 is used to connect multiple model intelligent agents with different functions.

[0118] Furthermore, to enhance the rationality and refinement of the model's decisions, a reinforcement learning framework employing a triple reward function (Generalized Relative Preference Optimization, GRPO) is used to train the base model. Specifically, rewards are initially based on classification accuracy, subcategorical F1 score, and inference chain fluency, comprehensively considering both the accuracy and interpretability of the reasoning. Subsequently, the Deep Thinking training dataset is used to simulate environmental feedback rewards, guiding the model to iteratively update its decision-making strategy and optimize inference chain generation and final output. Finally, after reinforcement learning training, a base model with superior reasoning and classification capabilities is obtained.

[0119] For example, the triple reward function of the GRPO framework can be expressed by the following formula:

[0120] R=ρ1·1(y det =y ture )+ρ2·F1 multiclass +ρ3·(1-Perliexity think )

[0121] In the formula, R represents the triple reward function; 1(y det =y ture () represents the accuracy reward function; F1 multiclass Represents the precision reward function; Perliexity think This represents the format compliance reward function; ρ1, ρ2, and ρ3 represent adjustable preset coefficients.

[0122] The accuracy reward function represents the score for exact match and perfect match, with 1 for a perfect match and 0 otherwise. The format conformity reward function is represented by detecting whether the output contains the formatted output, with 1 for a formatted output and 0 otherwise. Preferably, ρ1 can be 0.5, ρ2 can be 0.3, and ρ3 can be 0.2. ρ1, ρ2, and ρ3 can also be adaptively adjusted according to the actual situation, and this disclosure does not limit them.

[0123] Furthermore, to ensure high-quality training samples, a model output confidence verification and rejection sampling strategy is employed to purify and filter the training data. Specifically, the base model can first be used to infer from the initial dataset, filtering samples with high confidence. Subsequently, samples with low confidence or inference chains inconsistent with the labels can be removed, retaining only high-quality, high-confidence training samples for inclusion in the Deep Thinking training dataset. For example, Figure 3 A flowchart illustrating the data validation and sampling process is shown, such as... Figure 3 As shown, it includes:

[0124] S301. Obtain the original dataset.

[0125] S302. Input any raw data from the original dataset into the base model so that the base model can output the initial result.

[0126] S303. Filter the initial results using detection confidence, sub-classification confidence, and evidence overlap to determine whether the detection confidence is a valid value and greater than the preset detection threshold, the sub-classification confidence is a valid value and greater than the preset classification threshold, and the evidence overlap is greater than the preset overlap threshold. If the conditions are met, proceed to S304; otherwise, proceed to S306.

[0127] S304. Keep this sample.

[0128] S305. Traverse each original data point in the original dataset to construct the Deep Thinking training dataset.

[0129] S306, Discard this sample.

[0130] Preferably, the preset detection threshold can be set to 0.9, the preset classification threshold can be set to 0.85, and the preset overlap threshold can be set to 80%. The preset detection threshold, preset classification threshold, and preset overlap threshold can also be adaptively adjusted according to the actual situation, and this disclosure does not limit them.

[0131] Furthermore, to balance the depth and efficiency of the model's inference capabilities, a dual-channel fusion mechanism is employed for the deep thinking channel and the fast response channel of the image-text detection model under training. Specifically, data augmentation can first be performed on the deep thinking training dataset, which has undergone rejection sampling and purification. For example, OCR adversarial perturbations with rotation ±30° and transparency 40%-70% can be applied to the retained samples. Alternatively, insecure text can be injected into image-safe images based on the gold labels of the images and the gold labels of the text in the current training set. In particular, the injection methods can be: text covering areas without key image and text information; text overlapping with the original text in the image; text overlapping, rotating, and having similar colors to the original text. Subsequently, the deep thinking training dataset can be simplified by removing the thought chains to construct a fast response training dataset. Then, the deep thinking training dataset, which has undergone rejection sampling and purification, and the fast response training dataset, which has had its inference chains removed and labels clearly defined, can be combined according to a preset mixing ratio to generate the image-text detection training dataset. Finally, guiding reasoning prompts and direct conclusion prompts can be used for different types of samples to achieve complementarity and synergy between the two channels.

[0132] Finally, in the final training phase, dual-channel joint reinforcement learning can be used to fuse and synergistically improve the capabilities of the image and text detection model. Specifically, interpretability loss, dual-channel consistency loss, and inference efficiency loss can be integrated into a composite loss function, balancing inference accuracy, channel consistency, and inference efficiency. Then, the fused training dataset can be input into the image and text detection model, with the deep thinking channel and the quick response channel updating their parameters synchronously during the same training process. Next, backpropagation can be performed based on the composite loss function to dynamically adjust the parameters of the two channels, enabling the model to accurately handle complex inference scenarios while efficiently handling routine detection tasks. Finally, after training, the model possesses diverse capabilities such as inference chain generation, evidence output, and efficient classification, adapting to different harmful image and text detection application scenarios. For example, Figure 4 A schematic diagram of the parallel training process is shown, such as... Figure 4 As shown, it includes:

[0133] S401a. Train the deep thinking channel to generate a deep thinking supervision signal.

[0134] S401b: Train the rapid response channel to generate a rapid response supervision signal.

[0135] S402. Use deep thinking supervision signals and fast response supervision signals to update the policy and optimize the policy parameters of the model.

[0136] S403. Synchronize the final trained parameters to the parameter sharing layer, so that the two channels can share the underlying features and capabilities to a certain extent.

[0137] In particular, the composite loss function, which integrates interpretability loss, dual-channel consistency loss, and inference efficiency loss, can also be optimized in conjunction with the GRPO loss function.

[0138] For example, the GRPO loss function can be expressed by the following formula:

[0139]

[0140] In the formula, L GRPO Represents the GRPO loss function; The expression represents the average of all samples in the original dataset D; x represents the input sample; y represents the average of all samples in the original dataset D. w Indicates the target label of the sample; y l The inference chain is represented by logσ(·); logσ(·) represents the logarithm of the sigmoid function value, i.e., the log-likelihood loss for binary classification tasks; β represents the scaling factor, used to control the weight of different loss terms on the objective function; λ represents the weighting coefficient, used to balance the consistency between the model output and the reference policy; KL[π θ (·∣x)||πref [·∣x] represents the Kullback-Leibler (KL) divergence; π θ (·|x) represents the output probability distribution of the target model with current parameter θ under input x; π ref (·∣x) represents the output probability distribution in the gold label under input x.

[0141] It should be understood that Figures 2 to 4 The schematic diagrams shown are merely illustrative and not limiting, and are scalable; those skilled in the art can use them as a basis. Figures 2 to 4 Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.

[0142] This disclosure provides a method for image and text detection. Figure 5 This is a schematic flowchart of an image and text detection method according to an embodiment of the present disclosure. This method can be applied to an image and text detection device. The image and text detection device is located in an electronic device. The electronic device includes, but is not limited to, fixed devices and / or mobile devices. For example, fixed devices include, but are not limited to, servers, which can be cloud servers or ordinary servers. For example, mobile devices include, but are not limited to, harmful image and text detection and classification devices, which can be mobile phones, tablets, vehicle terminals, etc. In some possible implementations, the image and text detection method can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 5 As shown, the image and text detection method includes:

[0143] S501. Obtain the image and text data to be detected.

[0144] S502. Analyze the image and text data to be detected and determine the recognition difficulty.

[0145] S503. Generate prompt words according to the prompt word template based on the recognition difficulty.

[0146] S504. Input the image and text data to be detected and the prompt words into the image and text detection model so that the image and text detection model can output the detection results.

[0147] Here, the image and text data to be detected refers to mixed image and text content that has not yet undergone harmful content detection processing and is about to enter the analysis process.

[0148] In this embodiment of the disclosure, text and image content can be crawled in real time from social platforms, forums, comment sections, etc., via an interface or web crawler. Specifically, the acquired content can be standardized, and the resulting standardized data can be used as the text and image data to be detected. The above is merely an illustrative example and is not intended to limit all possible scenarios for obtaining the text and image data to be detected; it is simply not exhaustive.

[0149] Here, recognition difficulty refers to the complexity that the model may encounter when determining the harmfulness of the image and text data. In this embodiment of the disclosure, factors affecting recognition difficulty may include indicators such as whether the image has adversarial perturbations and OCR recognition rate.

[0150] In this embodiment, the image can first undergo OCR recognition to analyze text length, semantic complexity, and whether it contains high-frequency sensitive words or cryptic expressions. Subsequently, it can automatically detect whether the image has been manipulated by rotation, occlusion, mosaic effects, low resolution, or other adversarial processing. Next, the recognition difficulty can be determined by combining manually set rules and the output of an automated lightweight model. The above is merely an illustrative example and does not constitute a limitation on all possible scenarios for determining recognition difficulty; it is simply not exhaustive.

[0151] Here, the prompt word template refers to the parameterized instruction expression that is preset based on the known detection task and sample recognition difficulty, and is used to guide the image and text detection model to understand the task and output style.

[0152] In this embodiment, different types of prompt word templates can be pre-designed to suit different recognition difficulties. Subsequently, based on the recognition difficulty result, the corresponding prompt word template can be automatically selected, and the specific sample content can be filled into the template variable position to obtain the prompt word. The above is merely an illustrative example and is not intended to limit all possible cases of prompt word generation; it is simply not exhaustive.

[0153] Here, the detection result refers to the judgment output of the image and text detection model on the input sample.

[0154] In this embodiment, the image and text data to be detected and the prompt words are combined as standard input and fed into the image and text detection model. Subsequently, the model, guided by the prompt words, automatically outputs classification labels and optional inference chains, criterion explanations, etc. The above is merely an illustrative example and is not intended to limit all possible scenarios for obtaining detection results; it is simply not exhaustive.

[0155] The technical solution of this disclosure, by judging the recognition difficulty, can effectively improve the overall detection efficiency and accuracy, reduce the risk of false positives and false negatives, and optimize the allocation of computing power and costs. By generating prompt words, it significantly improves the targeting and accuracy of model reasoning, allowing the model to perform adaptive reasoning based on task difficulty. Simultaneously, it standardizes the input format, improves the model's understanding ability and generation consistency, and facilitates subsequent output analysis and tracking. By utilizing the image and text detection model to output detection results, it achieves fully automated detection, greatly improving detection efficiency and reliability, and supporting highly interpretable output, thus enhancing the model's generalization ability in various complex scenarios.

[0156] In some embodiments, after generating the prompt words according to the prompt word template, the method further includes: reconstructing the prompt words into a multiple-choice question format; wherein, reconstructing the multiple-choice question format includes: adding a structured option set to the prompt words, the option set including: a hazard detection option: composed of binary mutually exclusive options {harmless, harmful}; and a sub-category option: composed of a preset sub-category set {C1, C2, ..., C...}. K The output is composed of K, where K is the total number of categories; constraints are added to the output format to force the model's output to simultaneously include: text reasoning conclusion: a complete natural language reasoning text containing the chain of evidence; selected option identifier: an option code selected from a set of structured options.

[0157] Here, multiple-choice question format reconstruction refers to rewriting the original open-ended prompts into a structured multiple-choice question format. In this embodiment of the disclosure, the process of multiple-choice question format reconstruction involves explicitly providing several options to guide the model to answer within a fixed set of options.

[0158] Here, the subcategories refer to options that supplement the basic harmful / harmless dichotomy with more detailed hazard type classifications. In this embodiment, each subcategory is designated as C1, C2, ..., C K To express.

[0159] Here, the harmfulness detection options refer to mutually exclusive structured binary options, requiring the model to choose one of them. In this embodiment of the disclosure, the harmfulness detection options are represented by harmless and harmful, respectively.

[0160] In this embodiment, a preliminary prompt word template can be selected first based on the sample identification difficulty and detection task to guide the model to output multi-step reasoning or direct conclusions. Subsequently, harmfulness detection options can be explicitly listed in the prompt words, while supplementing the subdivided category set. The above is only an illustrative example and is not intended to limit all possible cases of adding a structured option set; it is simply not exhaustive.

[0161] Here, constraints refer to strict specifications for the output format of the model, requiring that the output must contain specific types of information.

[0162] Here, the text reasoning conclusion refers to the complete natural language conclusion that the model infers based on the chain of evidence, explaining why the corresponding judgment was made.

[0163] Here, the selected option identifier refers to the option code that is ultimately selected by the model from the structured option set.

[0164] In this embodiment of the disclosure, the prompt can be explicitly required to include the textual reasoning conclusion and the selected option identifier in the model output. The above is merely an illustrative example and is not intended to limit all possible cases of adding constraints to the output format; it is simply not exhaustive.

[0165] In this way, the automated post-processing system can easily identify the option codes output by the model without complex natural language parsing, greatly improving the stability and maintainability of the overall process and increasing the overall accuracy. The detailed category options allow the system not only to determine whether an option is harmful but also to quickly locate the specific risk type, and explicitly require the output of reasoning and evidence chains, ensuring that every judgment result has traceability and explanation.

[0166] In some embodiments, generating prompt words according to a prompt word template based on the recognition difficulty includes: generating high-speed classification prompt words according to a high-speed classification prompt word template when the recognition difficulty is easy; generating deep thinking prompt words according to a deep thinking prompt word template when the recognition difficulty is difficult; and using the high-speed classification prompt words or the deep thinking prompt words as prompt words.

[0167] Here, the high-speed classification prompt word template refers to a prompt word design specifically used for quickly and directly determining the harmfulness or category of a sample, suitable for samples with low identification difficulty and clear content. In this embodiment of the disclosure, the high-speed classification prompt word template requires the model to output conclusions in a concise and efficient manner, simply returning a harmful / harmless label or option classification identifier.

[0168] In this embodiment, if the recognition difficulty is "easy," a high-speed classification prompt word template is automatically selected. Subsequently, the text and image content to be detected can be filled into the high-speed classification prompt word template, and prompt words for that sample are automatically generated. The above is merely an illustrative example and is not intended to limit all possible scenarios for generating high-speed classification prompt words; it is simply not exhaustive.

[0169] Here, the deep thinking prompt template refers to samples that are difficult to identify, have obscure content, and require comprehensive reasoning. In this embodiment of the disclosure, the deep thinking prompt template can guide the model to perform multi-step logical reasoning, generate a complete natural language reasoning chain, and output a harmfulness classification and reasoning.

[0170] In this embodiment of the disclosure, if the recognition difficulty is "difficult," a deep thinking prompt template is automatically selected. Subsequently, the text and image content to be detected can be filled into the deep thinking prompt template, and prompts for that sample are automatically generated. The above is merely an illustrative example and is not intended to limit all possible scenarios for generating deep thinking prompts; it is simply not exhaustive.

[0171] Thus, simple samples utilize high-speed classification templates, significantly reducing inference computation and generation time, achieving high throughput and real-time response. Difficult samples employ deep thinking templates, guiding the model to fully utilize multimodal information and context, outputting high-quality inference chains and classification reasons, reducing the risk of misclassification and missed classification. Dynamically scheduling model computing power allows for rapid processing of simple samples and the allocation of more computing power and inference resources to complex samples, effectively optimizing overall computing power and system cost.

[0172] In some embodiments, the image-text detection model is configured to perform the following steps: in response to a prompt word being a high-speed classification prompt word, invoking the fast response channel of the image-text detection model to perform reasoning and obtain a detection result; in response to a prompt word being a deep thinking prompt word, invoking the deep thinking channel of the image-text detection model to perform reasoning and obtain a detection result.

[0173] In this embodiment of the disclosure, after receiving the input prompt word, the image and text detection model automatically detects whether it belongs to the high-speed classification prompt word or the deep thinking prompt word type. For example, the two types can be automatically distinguished by the structure, keywords, and tags of the prompt word template, or relevant identifiers can be directly included when generating the prompt word, or other methods in the prior art can be selected to distinguish the two types. This disclosure does not limit this approach.

[0174] In this embodiment of the disclosure, when the image-text detection model recognizes the input as a high-speed classification prompt word, it automatically calls the internal fast response channel, which outputs a concise label, and uses this label as the detection result. The above is merely an illustrative example and is not intended to limit all possible scenarios for obtaining the detection result; it is simply not exhaustive.

[0175] In this embodiment of the disclosure, when the image and text detection model recognizes the input as a deep thinking prompt word, it automatically invokes the internal deep thinking channel, which outputs a detailed reasoning chain, evidence explanation, and classification result. The combination of the detailed reasoning chain, evidence explanation, and classification result is used as the detection result. The above is only an illustrative example and is not intended to limit all possible situations for obtaining the detection result; it is simply not exhaustive.

[0176] Thus, a fast channel is used for simple samples, achieving sub-second or second-level response times, significantly improving the overall system's concurrent processing capabilities and real-time performance. Simultaneously, resources can be concentrated on complex samples, optimizing computational investment and costs. For complex or obscure samples, the system automatically routes them to a deep thinking channel, where multi-step reasoning and comprehensive analysis by the model effectively improves accuracy and reduces false positives and false negatives, while also outputting the reasoning chain and evidence chain. The reasoning path can be dynamically adapted based on sample difficulty, accommodating various business scenarios.

[0177] In some implementations, using multiple-choice questions for reinforcement learning may lead to the risk of guessing correctly during the thinking process; therefore, we use text results as rewards. However, changing the reasoning format to multiple-choice questions during the final reasoning stage also increases the overall accuracy. Specifically, the original samples in the Deep Thinking and Quick Response training datasets typically contain diverse information such as input content, reasoning chains, intermediate reasoning steps, and final labels. Therefore, to adapt to multiple-choice recombination, complex reasoning chain samples need to be transformed into structured options first. Subsequently, each reasoning chain, reasoning conclusion, and possible incorrect reasoning, common misunderstandings, etc., are compiled into multiple-choice options, so that the reasoning chain and final label of each sample are mapped to a set of structured options. Finally, the input content and the task to be reasoned can be clearly described in the prompts, while the options include multi-dimensional content such as correct reasoning chains, common incorrect chains, superficial judgments, and missing reasoning, which facilitates model differentiation.

[0178] For example, the prompts for the restructured multiple-choice question format could be:

[0179] Original input: "This image and text belong to: [A] harmless [B] specifically harmful [C] generally offensive [D] seriously insulting"

[0180] Output specifications:

[0181]

[0182] Figure 6 Another flowchart of the image and text detection method is shown, such as... Figure 6 As shown, the method includes:

[0183] S601. Obtain the image and text data to be detected.

[0184] S602. Analyze the image and text data to be detected and determine the recognition difficulty; if the recognition difficulty is determined to be easy, proceed to S603; if the recognition difficulty is determined to be difficult, proceed to S605.

[0185] S603. Generate high-speed classification prompts according to the high-speed classification prompt template, and then execute S604.

[0186] S604. Call the fast response channel of the image and text detection model to perform end-to-end inference, then proceed to S608.

[0187] S605. Generate deep thinking prompts according to the deep thinking prompt template.

[0188] S606a. The visual agent of the image detection model performs inference analysis on the image data in the image data to be detected and obtains the first result.

[0189] S606b: The text agent of the image-text detection model performs reasoning analysis on the text data in the image-text data to be detected, and obtains the second result.

[0190] S606c: The fusion agent of the image and text detection model performs reasoning analysis on the image and text data in the image and text data to be detected, and obtains a third result.

[0191] S607. Integrate the first result, the second result, and the third result, and then execute S608.

[0192] S608. Format the inference results as detection results and then output them.

[0193] S609. Determine whether the test result is high-risk. If yes, proceed to S610a; if no, proceed to S610b.

[0194] S610a. Transfer the image and text data to be inspected to the review queue for manual or other review.

[0195] S610b, Mark the image and text data to be detected as passable.

[0196] It should be understood that Figure 6 The schematic diagrams shown are merely illustrative and not limiting, and are scalable; those skilled in the art can use them as a basis. Figure 6 Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.

[0197] This disclosure provides an image-text detection model training device, such as... Figure 7 As shown, the device may include: a basic model training module 701, used to train the visual agent, text agent, and fusion agent in the basic model to be trained based on the thought chain dataset, to obtain a basic model; a first training data construction module 702, used to input the original dataset into the basic model, filter its initial output results through a three-fold reliability threshold, and construct a deep thinking training dataset; a second training data construction module 703, used to perform data simplification processing on the deep thinking training dataset to generate a quick response training dataset; and a detection model training module 704, used to train the deep thinking channel and quick response channel of the image and text detection model to be trained in parallel using the deep thinking training dataset and the quick response training dataset, to obtain an image and text detection model.

[0198] In some embodiments, the thought chain dataset includes: image and text data, thought process, and judgment result; the basic model training module 701 includes: a character recognition submodule, used to perform optical character recognition on the image and text data to obtain image data and text data; a visual optimization submodule, used to optimize the parameters of the visual agent using the image data as input and through a triple reward function; a text optimization submodule, used to optimize the parameters of the text agent using the text data as input and through a triple reward function; and a fusion optimization submodule, used to optimize the parameters of the fusion agent using the fusion features of the image data and text data as input and through a triple reward function. The triple reward function includes an accuracy reward function, a precision reward function, and a format conformance reward function; the accuracy reward function is used to calculate the binary classification matching degree between the agent's output and the judgment result; the precision reward function is used to calculate the F1 score for fine-grained classification; and the format conformance reward function is used to verify the consistency between the output and the logical framework of the thought process.

[0199] In some embodiments, the image and text detection model training apparatus further includes: a localization annotation module 705. Figure 7 (Not shown in the image) is used to perform localized expression annotation on each original data in the original dataset based on a dynamic cultural lexicon, and obtain the corresponding gold label for each original data. The gold label includes: harmfulness binary label, sub-category label, and evidence fragment set.

[0200] In some embodiments, the first training data construction module 702 includes: an initial result generation submodule, used to input samples from the original dataset into the base model to generate initial results, the initial results including harmfulness judgment conclusions, sub-classification probability distributions and reasoning process evidence chains; and a confidence verification submodule, used to perform triple-reset confidence verification on the initial results based on the gold label to construct a deep thinking training dataset that meets the triple-reset confidence threshold.

[0201] In some embodiments, the confidence verification submodule is configured to: extract the logistic function probability value of the harmfulness judgment conclusion as the detection confidence when the harmfulness judgment conclusion is consistent with the harmfulness binary label in the gold label; if inconsistent, the detection confidence is recorded as an invalid value; extract the maximum probability value as the sub-category confidence when the category with the highest probability in the sub-category probability distribution is the same as the sub-category label in the gold label; if the categories are different, the sub-category confidence is recorded as an invalid value; calculate the intersection-union ratio (IUGR) similarity between the evidence chain of the reasoning process and the evidence fragment set in the gold label, and use the IUGR similarity as the evidence overlap, where the evidence overlap is equal to the number of evidence fragments shared by both parties divided by the total number of all evidence fragments of both parties; and adopt a sample only when the following conditions are met simultaneously: the detection confidence is a valid value and greater than a preset detection threshold; the sub-category confidence is a valid value and greater than a preset classification threshold; and the evidence overlap is greater than a preset overlap threshold.

[0202] In some embodiments, the first training data construction module 702 further includes: a data augmentation submodule for performing data augmentation processing on each sample that meets the triple-reset confidence threshold; wherein the data augmentation processing methods include at least: text recognition adversarial perturbation and text injection.

[0203] In some embodiments, the second training data construction module 703 includes: an information removal submodule for removing intermediate inference steps from the deep thinking training data; and a conclusion compression submodule for compressing the multimodal fusion analysis conclusion into a single label.

[0204] In some embodiments, the detection model training module 704 includes: a training data generation submodule, used to merge data from the deep thinking training dataset and the quick response training dataset according to a preset mixing ratio to generate an image and text detection training dataset; and a joint training submodule, used to input the image and text detection training dataset into the image and text detection model to be trained, and to jointly train the deep thinking channel and the quick response channel to obtain the image and text detection model.

[0205] In some embodiments, the joint training submodule is used to: construct a composite loss function comprising interpretability loss, dual-channel consistency loss, and inference efficiency loss; the formula for the composite loss function is L. total =λ t ·L explain +μ·L consistency +η t ·L efficiency Where μ is a fixed weighting coefficient, and λ t η t L is a phased adjustable coefficient. explain L represents the interpretability loss. consistency L represents the dual-channel consistency loss. efficiency The inference efficiency loss is represented by the image and text detection training dataset and the composite loss function. The parameters of the deep thinking channel and the fast response channel are updated by backpropagation to obtain the image and text detection model.

[0206] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0207] The image-text detection model training device in this embodiment is trained using a thought chain dataset. The model can not only perform basic recognition but also learn deeper reasoning and interpretation capabilities. The multi-agent structure allows the model to efficiently extract visual and textual information separately, and then achieve stronger cross-modal understanding through agent fusion, improving overall performance and enabling better selection and generation of high-difficulty, high-quality data. A three-level reliability threshold strategy accurately selects the most representative complex samples, generating a high-quality deep thinking training dataset that helps improve the model's performance on more challenging tasks. Generating a fast-response training dataset allows the model to not only possess deep reasoning but also make efficient and accurate judgments quickly in common scenarios. Simplified data training accelerates the model's learning process, improving its response speed and stability in practical applications. Parallel training of the deep thinking and fast-response channels of the image-text detection model makes the model structure more flexible, enabling better performance in different types of tasks. The model can intelligently select channels based on task difficulty, balancing complex reasoning and rapid response, thus improving adaptability in real-world scenarios.

[0208] This disclosure provides an image and text detection device, such as... Figure 8 As shown, the device may include: a data acquisition module 801 for acquiring the image and text data to be detected; a data analysis module 802 for analyzing the image and text data to be detected and determining the recognition difficulty; a prompt word generation module 803 for generating prompt words according to the prompt word template based on the recognition difficulty; and a model inference module 804 for inputting the image and text data to be detected and the prompt words into the image and text detection model so that the image and text detection model can output the detection result; the image and text detection model is trained using the method described above.

[0209] In some embodiments, the data analysis module 802 includes: a high-speed classification prompt word submodule, used to generate high-speed classification prompt words according to the high-speed classification prompt word template when the recognition difficulty is easy; a deep thinking prompt word submodule, used to generate deep thinking prompt words according to the deep thinking prompt word template when the recognition difficulty is difficult; and a prompt word determination submodule, used to use the high-speed classification prompt words or the deep thinking prompt words as prompt words.

[0210] In some embodiments, the image and text detection device further includes: a format reconstruction module 803. Figure 8 (Not shown in the image), used to restructure the prompt word into a multiple-choice question format. The format restructuring module 803 includes: an option adding submodule, used to add a structured option set to the prompt word, which includes: a harmlessness detection option: composed of binary mutually exclusive options {harmless, harmful}; and subcategorization options: composed of a preset subcategories set {C1, C2, ..., C...}. KThe model consists of a set of submodules, where K is the total number of categories; the constraint addition submodule is used to add constraints to the output format, forcing the model output to simultaneously include: text reasoning conclusion: a complete natural language reasoning text containing the chain of evidence; and selected option identifier: an option code selected from a set of structured options.

[0211] In some embodiments, the image-text detection model is configured to perform the following steps: in response to a prompt word being a high-speed classification prompt word, invoking the fast response channel of the image-text detection model to perform reasoning and obtain a detection result; in response to a prompt word being a deep thinking prompt word, invoking the deep thinking channel of the image-text detection model to perform reasoning and obtain a detection result.

[0212] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0213] The image and text detection device in this embodiment effectively improves overall detection efficiency and accuracy by judging the recognition difficulty, reducing the risk of false positives and false negatives, and optimizing the allocation of computing power and costs. By generating prompt words, it significantly improves the targeting and accuracy of model reasoning, allowing the model to adaptively reason according to task difficulty. Simultaneously, it standardizes the input format, improving model understanding and generation consistency, facilitating subsequent output analysis and tracking. By utilizing the image and text detection model to output detection results, it achieves fully automated detection, greatly improving detection efficiency and reliability, and supporting highly interpretable output, enhancing the model's generalization ability in various complex scenarios.

[0214] This disclosure provides a scenario illustration of a text detection model training method, such as... Figure 9 As shown.

[0215] As previously described, the image and text detection model training method provided in this disclosure is applied to electronic devices. These electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.

[0216] Specifically, the electronic device may perform the following operations:

[0217] Based on the known thought chain dataset, the visual agent, text agent, and fusion agent in the base model to be trained are trained to obtain the base model; the original dataset is input into the base model, and its initial output results are filtered by three resetting confidence thresholds to construct the deep thinking training dataset; the deep thinking training dataset is simplified to generate the fast response training dataset; using the deep thinking training dataset and the fast response training dataset, the deep thinking channel and fast response channel of the image and text detection model to be trained are trained in parallel to obtain the image and text detection model.

[0218] It should be understood that Figure 9 The scene diagrams shown are merely illustrative and not restrictive; those skilled in the art can interpret them based on... Figure 9 Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.

[0219] This disclosure provides a scenario illustration of an image and text detection method, such as... Figure 10 As shown.

[0220] As previously described, the image and text detection method provided in this disclosure is applied to electronic devices. These electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers.

[0221] Specifically, the electronic device may perform the following operations:

[0222] Acquire the image and text data to be detected; analyze the image and text data to be detected to determine the recognition difficulty; generate prompt words according to the prompt word template based on the recognition difficulty; input the image and text data to be detected and the prompt words into the image and text detection model, so that the image and text detection model can output the detection results.

[0223] It should be understood that Figure 10 The scene diagrams shown are merely illustrative and not restrictive; those skilled in the art can interpret them based on... Figure 10 Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.

[0224] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0225] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0226] Figure 11A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0227] like Figure 11 As shown, device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 1102 or a computer program loaded from storage unit 1108 into random access memory (RAM) 1103. The RAM 1103 may also store various programs and data required for the operation of device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. An input / output (I / O) interface 1105 is also connected to bus 1104.

[0228] Multiple components in device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0229] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as image and text detection model training methods and / or image and text detection methods. For example, in some embodiments, the image and text detection model training methods and / or image and text detection methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by computing unit 1101, one or more steps of the image-text detection model training method and / or image-text detection method described above can be performed. Alternatively, in other embodiments, computing unit 1101 can be configured to perform the image-text detection model training method and / or image-text detection method by any other suitable means (e.g., by means of firmware).

[0230] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0231] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0232] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0233] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0234] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0235] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0236] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0237] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training an image-text detection model, comprising: Based on the MindChain dataset, the visual agent, text agent, and fusion agent in the base model to be trained are trained to obtain the base model; The original dataset is input into the base model, and the initial output results are filtered by three sets of confidence thresholds to construct the deep thinking training dataset. The deep thinking training dataset is simplified to generate a quick response training dataset; Using the deep thinking training dataset and the fast response training dataset, the deep thinking channel and fast response channel of the image and text detection model to be trained are trained in parallel to obtain the image and text detection model.

2. The method according to claim 1, wherein, The thought chain dataset includes: text and image data, thought process, and judgment result; The basic model is obtained by training the visual agent, text agent, and fusion agent in the base model to be trained based on the thought chain dataset, including: Optical character recognition is performed on the image and text data to obtain image data and text data; Using the image data as input, the parameters of the visual agent are optimized through a triple reward function; Using the text data as input, the parameters of the text agent are optimized through the triple reward function; Using the fusion features of the image data and the text data as input, the parameters of the fusion agent are optimized through the triple reward function; The triple reward function includes an accuracy reward function, a precision reward function, and a format compliance reward function; wherein, The accuracy reward function is used to calculate the binary classification matching degree between the agent's output and the judgment result; The accuracy reward function is used to calculate the F1 score for fine-grained classification; The format compliance reward function is used to verify the consistency between the output and the logical framework of the thought process.

3. The method according to claim 1, wherein, Before inputting the original dataset into the base model, the following steps are also included: Based on a dynamic cultural lexicon, each original data point in the original dataset is localized and labeled to obtain a gold label corresponding to each original data point. The gold label includes: a harmfulness binary label, a sub-category label, and a set of evidence fragments.

4. The method according to claim 3, wherein, The process of inputting the original dataset into the base model, filtering its initial output results through three sets of confidence thresholds, and constructing a deep thinking training dataset includes: The samples in the original dataset are input into the base model to generate the initial results, which include harmfulness judgment conclusions, sub-class probability distributions, and evidence chains for the reasoning process. Based on the gold label, a triple-reset reliability verification is performed on the initial results to construct a deep thinking training dataset that meets the triple-reset reliability threshold.

5. The method according to claim 4, wherein, The process of performing a triple-reset reliability verification on the initial result based on the gold label includes: When the harmfulness judgment conclusion is consistent with the harmfulness binary label in the gold label, the logical function probability value of the harmfulness judgment conclusion is extracted as the detection confidence level; if they are inconsistent, the detection confidence level is recorded as an invalid value. When the category with the highest probability in the sub-category probability distribution is the same as the sub-category label in the gold label, the highest probability value is extracted as the sub-category confidence score; if the categories are different, the sub-category confidence score is recorded as an invalid value. Calculate the intersection-union ratio similarity between the evidence chain in the reasoning process and the evidence fragment set in the gold tag, and use the intersection-union ratio similarity as the evidence overlap degree. The evidence overlap degree is equal to the number of evidence fragments shared by both parties divided by the total number of all evidence fragments of both parties. A sample is adopted only if all of the following conditions are met: The detection confidence level is a valid value and greater than the preset detection threshold; The confidence level of the sub-classification is a valid value and greater than the preset classification threshold; The degree of overlap of the evidence is greater than a preset overlap threshold.

6. The method according to claim 4, wherein, The method further includes: Data augmentation is performed on each sample that meets the three reset confidence thresholds; wherein the data augmentation methods include at least: text recognition adversarial perturbation and text injection.

7. The method according to claim 1, wherein, The data simplification process for the deep thinking training dataset includes: Remove intermediate inference steps from deep thinking training data; The conclusion of the compressed multimodal fusion analysis is a single label.

8. The method according to claim 1, wherein, The process of using the deep thinking training dataset and the quick response training dataset to train the deep thinking channel and quick response channel of the image-text detection model to be trained in parallel, thereby obtaining the image-text detection model, includes: Based on the deep thinking training dataset and the quick response training dataset, the data are merged according to a preset mixing ratio to generate an image and text detection training dataset. The image and text detection training dataset is input into the image and text detection model to be trained, and the deep thinking channel and the quick response channel are jointly trained to obtain the image and text detection model.

9. The method according to claim 8, wherein, The step of inputting the image-text detection training dataset into the image-text detection model to be trained, and jointly training the deep thinking channel and the quick response channel to obtain the image-text detection model includes: Construct a composite loss function that includes interpretability loss, dual-channel consistency loss, and inference efficiency loss; the formula for the composite loss function is L. total =λ t ·L explain +μ·L consistency +η t ·L efficiency Where μ is a fixed weighting coefficient, and λ t η t L is a phased adjustable coefficient. explain L represents the interpretability loss. consistency L represents the dual-channel consistency loss. efficiency This indicates a loss of reasoning efficiency; The image and text detection model is obtained by backpropagating the parameters of the deep thinking channel and the fast response channel based on the image and text detection training dataset and the composite loss function.

10. A method for detecting text in an image, comprising: Obtain the image and text data to be detected; Analyze the image and text data to be detected to determine the recognition difficulty; Based on the recognition difficulty, generate prompt words according to the prompt word template; The image and text data to be detected and the prompt words are input into the image and text detection model, and the image and text detection model outputs the detection results. The image and text detection model is obtained by training using the method described in any one of claims 1-9.

11. The method according to claim 10, wherein, The step of generating prompt words according to the prompt word template based on the recognition difficulty includes: When the recognition difficulty is set to "easy", high-speed classification prompt words are generated according to the high-speed classification prompt word template. When the recognition difficulty is "difficult", deep thinking prompts are generated according to the deep thinking prompt template; The high-speed classification prompt or the deep thinking prompt is used as the prompt.

12. The method according to claim 11, wherein, After generating the prompts according to the prompt template, it also includes: Restructure the prompts into a multiple-choice question format; The restructuring of the multiple-choice question format includes: Add a structured options set to the prompt word, which includes: Hazard detection options: consist of two mutually exclusive options {harmless, harmful}; Subcategories options: These are defined by a preset set of subcategories {C1, C2, ..., C}. K The system is composed of K, where K is the total number of categories. Add constraints to the output format to force the model's output to include: Textual reasoning conclusion: A complete natural language reasoning text containing a chain of evidence; Selected option identifier: The option code selected from the structured option set.

13. The method according to claim 12, wherein, The image and text detection model is configured to perform the following steps: In response to the prompt word being the high-speed classification prompt word, the fast response channel of the image and text detection model is invoked for reasoning to obtain the detection result; In response to the prompt word being the deep thinking prompt word, the deep thinking channel of the image and text detection model is invoked to perform reasoning and obtain the detection result.

14. A training device for an image-text detection model, comprising: The basic model training module is used to train the visual agent, text agent, and fusion agent in the basic model to be trained based on the thought chain dataset, so as to obtain the basic model. The first training data construction module is used to input the original dataset into the base model, filter its initial output results through three sets of confidence thresholds, and construct the deep thinking training dataset. The second training data construction module is used to simplify the data of the deep thinking training dataset and generate a fast response training dataset. The detection model training module is used to train the deep thinking channel and the quick response channel of the image and text detection model to be trained in parallel using the deep thinking training dataset and the quick response training dataset, so as to obtain the image and text detection model.

15. A text and image detection device, comprising: The data acquisition module is used to acquire the image and text data to be detected; The data analysis module is used to analyze the image and text data to be detected and determine the recognition difficulty; The prompt word generation module is used to generate prompt words according to the prompt word template based on the recognition difficulty. The model inference module is used to input the image and text data to be detected and the prompt words into the image and text detection model, so that the image and text detection model can output the detection result; The image and text detection model is obtained by training using the method described in any one of claims 1-9.

16. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1-13.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein, Computer instructions are used to cause a computer to perform the method according to any one of claims 1-13.

18. A computer program product comprising a computer program stored on a storage medium, wherein the computer program, when executed by a processor, implements the method according to any one of claims 1-13.

Images