Image detection method and device

Abstract

The embodiment of the specification provides an image detection method and device, comprising: acquiring a first image and a first text indicating authenticity detection of the first image; inputting the first image and the first text into a large language model to make the large language model perform first detection on the first image to obtain a first detection result, the large language model being trained based on sample images and label data thereof, the label data comprising reasoning label data for guiding the large language model to perform multi-step reasoning and conclusion label data indicating authenticity of a corresponding image, the reasoning label data comprising first label data for guiding the large language model to perform first detection on the image based on the whole image and second label data for guiding the large language model to perform second detection on the image based on details of the image; determining a confidence degree based on the first detection result; and if the confidence degree is lower than a specified threshold, triggering the large language model to perform second detection on the first image to realize accurate detection on authenticity of the image.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to an image detection method and apparatus. Background Technology

[0002] With the continuous development of deep learning technology, images produced by generative models such as Generative Adversarial Networks (GANs) and Latent Diffusion Models (LDMs) (hereinafter referred to as generated images, also known as fake images) are becoming increasingly realistic. These generative models can replace one person's image or video with another person's, thereby creating multimedia content that is indistinguishable from reality. This type of content is usually difficult to identify with the naked eye, making it impossible for users to judge the authenticity of online content, thus posing a threat to the content security and authenticity of the online environment.

[0003] Therefore, there is an urgent need for an image detection method to accurately detect the authenticity of images. Summary of the Invention

[0004] This specification provides one or more embodiments of an image detection method and apparatus to achieve accurate detection of image authenticity.

[0005] According to the first aspect, an image detection method is provided, comprising:

[0006] Acquire a first image to be detected and a first text indicating that the first image should be tested for authenticity;

[0007] The first image and the first text are input into a large language model, so that the large language model performs a first detection on the first image and obtains a first detection result. The large language model is trained based on sample images and their corresponding label data. The label data includes inference label data that guides the large language model to perform multi-step inference and conclusion label data that indicates the authenticity of the corresponding sample images. The inference label data includes: first label data and second label data. The first label data is used to guide the large language model to perform a first detection based on the global image of the corresponding sample images. The second label data is used to guide the large language model to perform a second detection based on the image details of the corresponding sample images.

[0008] Based on the first detection result, its confidence level is determined;

[0009] If the confidence level is lower than a specified threshold, the large language model is triggered to perform the second detection on the first image.

[0010] According to a second aspect, an image detection apparatus is provided, comprising:

[0011] The acquisition module is configured to acquire a first image to be detected and a first text indicating whether the first image is genuine or fake.

[0012] The input module is configured to input the first image and the first text into a large language model, so that the large language model performs a first detection on the first image and obtains a first detection result. The large language model is trained based on sample images and their corresponding label data. The label data includes inference label data that guides the large language model to perform multi-step inference and conclusion label data that indicates the authenticity of the corresponding sample images. The inference label data includes: first label data and second label data. The first label data is used to guide the large language model to perform a first detection based on the global image of the corresponding sample images, and the second label data is used to guide the large language model to perform a second detection based on the image details of the corresponding sample images.

[0013] The determination module is configured to determine its confidence level based on the first detection result;

[0014] The triggering module is configured to trigger the large language model to perform the second detection on the first image when the confidence level is lower than a specified threshold.

[0015] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in the first aspect.

[0016] According to a fourth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in the first aspect.

[0017] According to the method and apparatus provided in the embodiments of this specification, a first image to be detected and a first text indicating the authenticity detection of the first image are obtained; the first image and the first text are input into a large language model, so that the large language model performs a first detection on the first image to obtain a first detection result. The large language model is trained based on sample images and their corresponding label data. The label data includes inference label data guiding the large language model to perform multi-step inference and conclusion label data indicating the authenticity of the corresponding sample image. The inference label data includes first label data and second label data. The first label data is used to guide the large language model to perform a first detection based on the global image of the corresponding sample image, and the second label data is used to guide the large language model to perform a second detection based on image details of the corresponding sample image. Based on the first detection result, its confidence level is determined. If the confidence level is lower than a specified threshold, the large language model is triggered to perform a second detection on the first image.

[0018] In the above process, a large language model is trained using sample images and their corresponding label data (including inference label data guiding the large language model to perform multi-step inference and conclusion label data indicating the authenticity of the corresponding sample images). The inference label data includes: first label data to guide the large language model to perform a first detection based on the overall image, and second label data to guide the large language model to perform a second detection based on image details. This allows the large language model to be injected with anthropomorphic image authenticity analysis and identification capabilities (i.e., the ability to first identify authenticity from the overall image, and then perform more detailed authenticity identification based on local image details), enabling it to perform multi-step inference analysis on image authenticity and thus draw conclusions. Combined with the inherent inference characteristics of the large language model, the above process can, to some extent, solve the generalization problem caused by the rapid iteration of deepfake technology, achieving accurate detection of images generated by various deepfake techniques.

[0019] Furthermore, the above process can dynamically and adaptively decide whether to trigger a second detection based on image details (i.e., long thought chain detection) based on the confidence level of the first detection result (i.e., the credibility assessment of the model's inference results). It not only achieves comprehensive analysis and identification of various deepfake techniques (including new and existing techniques), but also effectively balances the computational cost and accuracy in the detection process. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0021] Figure 1A This is a schematic diagram illustrating the implementation framework of one embodiment disclosed in this specification;

[0022] Figure 1B A schematic diagram of a large language model;

[0023] Figure 2 A flowchart illustrating the training process of a large language model provided for an embodiment;

[0024] Figure 3 A schematic flowchart of an image detection method provided in an embodiment;

[0025] Figure 4 This is a schematic block diagram of an image detection device provided in an embodiment. Detailed Implementation

[0026] In this specification, the Large Language Model (LLM) may also be referred to simply as the Large Model. A Large Language Model is a natural language processing model based on deep learning techniques, typically with billions to hundreds of billions or even more parameters, possessing powerful language understanding and generation capabilities. Large Language Models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. Large Language Models learn the statistical features and semantic relationships of language through pre-training on large-scale corpora, giving them excellent generalization capabilities. The core capabilities of Large Language Models include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Its usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, the user guides the Large Language Model to generate specific outputs by designing prompts. Cue words can be task descriptions or instructions in text form, used to stimulate the semantic understanding and generation capabilities of large language models. In fine-tuning mode, large language models are further trained on small-scale datasets in specific domains to optimize their performance on specific tasks. The powerful generalization ability and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0027] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio data). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0028] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025), and will not be repeated here.

[0029] The technical solutions of the embodiments of this specification will now be described in detail with reference to the accompanying drawings.

[0030] This specification discloses an image detection method and apparatus through its embodiments. The application scenarios and technical concepts of the method are first introduced below:

[0031] Currently, in order to detect the authenticity of online content, related technologies provide an image detection method that can distinguish between genuine and fake images. The process generally involves: first, training a Deepfake detector based on a large number of generated images (also known as fake images) and their corresponding labels indicating that they are generated images, and a large number of real images and their corresponding labels indicating that they are real images, to obtain a detector that can detect the authenticity of images; then, using the trained detector, the authenticity of the images is identified.

[0032] However, in the aforementioned image detection methods, the detector's ability to detect image authenticity relies heavily on specific generative feature traces left by the learned generative model. When the generative model is upgraded or new image generation (i.e., forgery) techniques emerge, these specific generative feature traces (the generative feature traces learned by the detector) may disappear or change, leading to detection failure. Therefore, an improved image detection method is urgently needed to achieve accurate detection of image authenticity.

[0033] In light of this, the inventors propose an image detection method that utilizes a large language model infused with anthropomorphic image authenticity analysis capabilities to perform multi-step reasoning analysis on image authenticity. Combined with the inherent reasoning characteristics of the large language model, this anthropomorphic analysis capability significantly improves the accuracy and generalization of the model's detection. This method can dynamically and adaptively decide whether to trigger a second detection based on image details, based on the confidence level of the first detection result (i.e., the credibility assessment of the model's reasoning results). This method not only achieves comprehensive analysis and identification of various deepfake techniques (including novel and existing techniques) but also effectively balances computational cost and accuracy in the detection process.

[0034] Figure 1A This diagram illustrates an implementation scenario according to an embodiment disclosed in this specification. In this implementation scenario, a large language model trained based on sample images and their corresponding label data is deployed.

[0035] This large language model is a multimodal large language model, capable of processing data in at least two modalities: image and text. For example, the large language model may include an image encoding layer, a text encoding layer, a fusion layer, and multiple processing network layers (such as...). Figure 1B The diagram shows an M-layer structure and a head structure. The image encoding layer processes the image, extracting visual features; the text encoding layer processes the input text, extracting text features; the fusion layer aligns the visual features to the text feature space for fusion with the aforementioned text features (e.g., concatenation or fusion based on cross-attention mechanisms) to obtain a multimodal feature sequence; the multi-layer processing network is used for cross-modal reasoning and context integration of the multimodal feature sequence; and the head structure decodes the output of the last layer of the multi-layer processing network, mapping it to a vocabulary to generate the corresponding output text.

[0036] The label data corresponding to the sample image includes inference label data that guides the large language model to perform multi-step inference and conclusion label data that indicates the authenticity of the corresponding sample image. The inference label data includes: first label data and second label data. The first label data is used to guide the large language model to perform a first detection based on the global image of the corresponding sample image, and the second label data is used to guide the large language model to perform a second detection based on the image details of the corresponding sample image.

[0037] When it is necessary to perform authenticity detection on an image, a first image to be detected and a first text indicating that the first image should be detected are obtained. Then, the first image and the first text are input into a large language model so that the large language model performs a first detection on the first image and obtains a first detection result. Then, based on the first detection result, its confidence level is determined. If the aforementioned confidence level is lower than a specified threshold, the confidence level of the first detection result of the large language model is considered to be low. Accordingly, the large language model is triggered to perform a second detection on the first image, that is, the large language model is triggered to perform a more detailed long thought chain detection on the first image.

[0038] In the above process, a large language model is trained using sample images and their corresponding label data (including inference label data guiding the large language model to perform multi-step inference and conclusion label data indicating the authenticity of the corresponding sample images). The inference label data includes: first label data to guide the large language model to perform a first detection based on the overall image, and second label data to guide the large language model to perform a second detection based on image details. This allows the large language model to be injected with anthropomorphic image authenticity analysis and identification capabilities (i.e., the ability to first identify authenticity from the overall image, and then perform more detailed authenticity identification based on local image details), enabling it to perform multi-step inference analysis on image authenticity and thus draw conclusions. Combined with the inherent inference characteristics of the large language model, the above process can, to some extent, solve the generalization problem caused by the rapid iteration of deepfake technology, achieving accurate detection of images generated by various deepfake techniques.

[0039] Furthermore, the above process can dynamically and adaptively decide whether to trigger a second detection based on image details (i.e., long thought chain detection) based on the confidence level of the first detection result (i.e., the credibility assessment of the model's inference results). It not only achieves comprehensive analysis and identification of various deepfake techniques (including new and existing techniques), but also effectively balances the computational cost and accuracy in the detection process.

[0040] The image detection method provided in this specification will be described in detail below with reference to specific embodiments.

[0041] The training process of the large language model will be introduced below.

[0042] In some examples, such as Figure 2 As shown, a two-stage training approach can be used to train the large language model. The training process for the large language model can include the following steps: Step 01, constructing a first training set for training the large language model; Step 02, performing a first-stage training on the large language model based on the first training set to obtain the large language model trained in the first stage; Step 03, using a reinforcement learning algorithm to perform a second-stage training on the large language model trained in the first stage to obtain the large language model trained in the second stage.

[0043] The construction process of the first training set (step 01) will be described below.

[0044] The process of constructing the first training set includes: obtaining real images and generated images from multiple public data sources as sample images; and then determining the corresponding label data for each sample image to construct the first training set.

[0045] The label data is constructed by simulating the thinking patterns of human experts. It includes: reasoning label data that guides the large language model to perform multi-step reasoning and conclusion label data that indicates the authenticity of the corresponding sample images. The reasoning label data includes: first label data and second label data.

[0046] The first label data is used to guide the large language model to perform a first detection based on the global image of the corresponding sample image. That is, to guide the large language model to perform a first detection on the corresponding sample image based on the overall visual features of the image to determine the authenticity of the corresponding sample image (i.e., whether it is a generated image).

[0047] In some possible examples, the sample images include sample images identified as simple (hereinafter referred to as simple samples) and sample images identified as difficult (hereinafter referred to as difficult samples). Simple samples can include high-quality real images (i.e., images that are not easily misidentified) and generated images identified as low-quality (i.e., generated images that are not easily identified as real images). Difficult samples can include low-quality real images (i.e., real images that may contain misleading features, i.e., features that are prone to misidentification, such as blurriness, sharpness below a specified sharpness threshold, or missing visual details), and generated images identified as high-quality (i.e., generated images that are easily identified as real images).

[0048] For simple samples, the first label data contains the content that expresses the existence of obvious specified features in the image (such as obvious splicing boundaries in the generated image, or the existence of realistic structure in the real image), thus obtaining a relatively certain result of authenticity (e.g., "At first glance, this image seems fake due to obvious fusion boundaries").

[0049] For hard samples, the first label data contains content indicating uncertainty in judging the authenticity of the image, and / or content indicating the need for a second detection process (e.g., "It looks real at first glance (or, it looks fake at first glance)... This makes it difficult to determine its authenticity without further examination").

[0050] Secondary label data is used to guide the large language model to perform secondary detection based on image details for the corresponding sample images. In some possible examples, the secondary label data for each sample image may include: planning label data, evidence reasoning label data, and reflection label data.

[0051] In some other possible examples, for simple samples, the second label data may include: evidence-gathering reasoning label data; for difficult samples, the second label data may include: planning label data, evidence-gathering reasoning label data, and reflection label data.

[0052] The planning label data may include: image detail detection strategy content for the corresponding sample image, such as: strategy content indicating the detection of the image from a specified multi-dimensional perspective, which may include but is not limited to: structural integrity (such as the proportion, symmetry, etc. of targets such as faces, human bodies or other objects), artifact detail recognition (image texture / edge details), and physical feasibility verification, such as illumination direction, shadow consistency, and reflection logic light and shadow features.

[0053] For the evidence-gathering reasoning label data, when the second label data includes the planning label data, it includes: the image evidence acquisition and analysis verification steps required to be performed according to the detection strategy content (determined based on the planning label data), and the results corresponding to the analysis and verification steps. For example, it includes the following: the steps of checking the image one by one according to the detection dimensions (e.g., structure, artifact details, lighting features, background, etc.) indicated in the detection strategy content in the planning label data to find suspicious features (or credible features); and the steps of connecting multiple suspicious features (or credible features) (e.g., skin too smooth + lack of catchlight in the eyes + unnatural background blur) to form a comprehensive argument (including detailed evidence description) supporting the decision to "generate" or "be true".

[0054] The aforementioned process of checking each item includes, for example, the following: instructions for locating artifact details: low-level artifacts: instructions for finding pixel-level anomalies, such as skin texture that is too smooth (lacking pores), blurred or unnatural sharpening of edges, and imperfections in the blending of hair with the background; instructions for locating high-level semantic / physical violations: instructions for checking whether the lighting direction is consistent, whether the shadows conform to geometric logic, whether the reflection content is reasonable (such as reflections from glasses), whether the text is readable and aligned, and whether the clothing texture conforms to the physical material characteristics, etc.

[0055] In the absence of planning label data in the second label data, the evidence reasoning label data may include: the steps of checking the image one by one to find suspicious features (or credible features) by referring to the detection strategy content indicated by the first detection result or the detection dimensions (such as structure, artifact details, lighting features, background, etc.) indicated by the detection strategy content specified according to common sense, and linking the found suspicious features (or credible features) together to form a comprehensive argument that supports the decision to "generate" or "be true".

[0056] Reflective label data can include content that rethinks and explores the authenticity assessment results and related image evidence represented by the primary label data and the evidence reasoning label data. For example, it may include content that indicates how to support, revise, or refute previous assumptions (i.e., the authenticity assessment results represented by the primary label data and the evidence reasoning label data) by introducing new perspectives (relative to the related image evidence in the evidence reasoning label data) and / or combining common sense / physical laws, in order to discover hidden, illogical, and deeply contradictory features in the images. This reflective label data is used to improve the robustness of model decisions, helping the model overcome overfitting to known features when facing new forgery techniques, and better discover unknown and subtle forgery traces.

[0057] The conclusion label data is used to indicate the authenticity of the corresponding sample images. It can include: a comprehensive judgment formed after integrating the intuition of a "quick judgment" (including the detection results indicated by the first label data), a planned detection strategy for the label data (if any), specific evidence from the label data, and in-depth insights from reflecting on the label data (rethinking and uncovering the content) (if any). It includes a coherent logical loop based on the aforementioned scattered evidence points, a clear judgment of the image's authenticity, and a brief summary of the core reasons supporting that judgment.

[0058] The label data corresponding to each sample image can be labeled manually or through a specific labeling process. For example, the label data corresponding to a sample image can be represented as follows: <fast> First tag data< / fast> <planning> Planning label data< / planning> <reasoning> Forensic reasoning tag data< / reasoning> <conclusion>Reflecting on the label data< / conclusion> <answer>Conclusion label data< / answer>. Where < > and<!-- --> Used to denote the beginning and end of a corresponding part, these can be called specified start symbols and specified end symbols indicating the beginning of the corresponding part.

[0059] In some possible examples, the sample images may include real human face images as well as forged human face images (i.e., generated human face images) obtained through various image editing techniques. These image editing techniques may include, but are not limited to: face replacement (e.g., replacing face A with face B), partial face editing, and full face generation.

[0060] The above method yields sample images and their corresponding label data, forming the first training set. Subsequently, the large language model is trained in the first stage based on this first training set. This is understandable. The label data corresponding to each sample image in the first training set consists of labels that accurately and precisely represent the reasoning process (i.e., the content of the first and second label data) and that accurately represent the conclusion / answer (i.e., the conclusion label data), conforming to human preferences.

[0061] The first stage of training (step 02) will be described below.

[0062] like Figure 2 As shown, the first stage of training process includes the following steps 021-024: In step 021, any first sample image and its corresponding label data are obtained from the first training set.

[0063] In step 022, the large language model is trained using the first sample image and its corresponding label data, employing a supervised fine-tuning method. Specifically, step 022 includes: inputting the first sample image and its corresponding first sample text indicating whether the first sample image is genuine or fake into the large language model; the large language model processes the first sample image and the first sample text, performing multi-step inference detection on the first sample image to obtain its output inference detection result; then, based on the label data corresponding to the first sample image and the inference detection result, the large language model is fine-tuned under supervision. The cross-entropy loss function can be used, and the model loss value is determined based on the difference between the label data corresponding to the first sample image and the inference detection result. The goal of supervised fine-tuning of the large language model is to minimize the model loss value (i.e., minimize the difference between the label data corresponding to the first sample image and the inference detection result).

[0064] The above process uses only a first sample image as an example to introduce the supervised fine-tuning process of the large language model. In some possible examples, step 022 can be executed multiple times for different first sample images to perform supervised fine-tuning of the model parameters of the large language model until the large language model meets the preset convergence conditions, thus obtaining the supervised fine-tuned large language model. The preset convergence conditions may include, but are not limited to: determining that the model loss value is lower than a preset loss threshold, the number of iterations to adjust the parameters of the large language model exceeding a specified threshold, and the duration of fine-tuning the parameters of the large language model exceeding a specified duration.

[0065] Through the aforementioned supervised fine-tuning, the large language model can learn to detect the authenticity of images through multi-step reasoning (based on the reasoning steps defined by the label data). This ensures that the large language model can perform structured reasoning as indicated by the label data to obtain accurate image authenticity detection results. Furthermore, by reflecting on the label data, the robustness of the model's decision-making is improved. When faced with new forgery techniques, the model can overcome overfitting to known features and better discover unknown and subtle forgery traces.

[0066] Next, in step 023, a second training set is constructed for further training of the supervised fine-tuned large language model; then, in step 024, based on the second training set, a hybrid preference offline reinforcement learning algorithm is used to further train the supervised fine-tuned large language model to obtain the large language model completed in the first stage of training.

[0067] In the process of continuing to train the large language model after supervised fine-tuning using the hybrid preference offline reinforcement learning algorithm, a hybrid preference training set (i.e., the second training set) was constructed to further promote reasoning that conforms to human thinking.

[0068] In this second training set, the label data corresponding to each sample image includes: preference label data. And unpreference label data, wherein the unpreference label data may include first unpreference label data. and / or second unpreference label data .

[0069] Second training set It can be represented as .in, This refers to the sample image and its corresponding sample text used for authenticity detection. This indicates that the label data includes the first unpreference label data. The number of sample images, This indicates that the label data includes second unpreference label data. The number of sample images.

[0070] Among them, preference label data Label data that is accurate, detailed and precise in the reasoning process described above (i.e., the content of the first and second label data), and whose conclusion / answer (i.e., the conclusion label data) is accurate and conforms to human preferences.

[0071] First unbiased label data This refers to label data that does not conform to human preferences, where the reasoning process (i.e., the content of the first and second label data) is not precise or detailed enough, but the conclusion / answer (i.e., the conclusion label data) is accurate.

[0072] Second unbiased label data Label data that does not conform to human preferences, where the reasoning process (i.e., the content of the first and second label data) is inaccurate and the conclusion / answer (i.e., the conclusion label data) is inaccurate.

[0073] In some possible examples, the preference label data corresponding to each sample image in the second training set. The two types of unbiased labeled data can be obtained by manual annotation of each sample image in the second training set by annotators. The second training set may include sample images that are the same as or different from those in the first training set.

[0074] In some other possible examples, for each sample image in the second training set, its preference label data The two types of unbiased label data can be obtained by filtering the model output results obtained by processing the corresponding sample images through a large language model after supervised fine-tuning.

[0075] Among these, preference label data corresponding to the sample images can be directly obtained from the model output results obtained through supervised fine-tuning of the large language model. And two types of non-preference label data. Additionally, initial preference label data corresponding to the sample images can be obtained from the large language model after supervised fine-tuning. The initial two types of unpreference label data were followed by the annotation of the initial preference label data by the annotators. The initial two types of unbiased labeled data are optimized to obtain the final preferred labeled data corresponding to the sample images. and two types of non-preference label data

[0076] In some other possible examples, for each sample image in the second training set, its preference label data The labels can be obtained by manual annotation of each sample image in the second training set by the annotators; the two types of unbiased label data can be obtained by filtering the model output results obtained by processing the corresponding sample images through a large language model after supervised fine-tuning.

[0077] The second training set, namely a number of sample images and their corresponding data containing preference labels, is obtained through the above method. After obtaining the label data for the two types of unpreference label data, proceed to step 024.

[0078] Step 024 may include the following steps 0241-0243: In step 0241, obtain any second sample image i and its corresponding preference label data from the second training set. and the first unpreference label data Second unpreference label data Tag data.

[0079] In step 0242, the second sample image and its corresponding second sample text, which are used to perform authenticity checks on themselves, are input into the supervised fine-tuned large language model. The supervised fine-tuned large language model processes the second sample image and the second sample text, performs multi-step inference detection on the second sample image, and outputs preference label data from the supervised fine-tuned large language model. The first predicted probability distribution, the first unbiased label data The second predicted probability distribution, and the second unbiased label data The third predicted probability distribution.

[0080] In step 0243, the supervised fine-tuned large language model is further trained with the goal of maximizing the first objective function. The first objective function is determined based on the first, second, and third prediction probability distributions. The first objective function can be expressed by the following formula (1):

[0081] ; (1)

[0082] in, This represents the second sample image and the second sample text; This indicates the calculation of the expected value; This indicates the current strategy, i.e., the current values ​​of the model parameters of the large language model; This represents the first predicted probability distribution. This represents the second and / or third predicted probability distributions, where the second sample image corresponds to the first unbiased label data. ,but This represents the second predicted probability distribution, if the second sample image corresponds to the second unbiased label data. ,but This represents the third prediction probability distribution; and These represent the strength of the deviation of the current policy from its previous policy, and are preset parameters; the previous policy can be the policy preceding the current policy or a reference policy. This reference strategy, for example, is to supervise the values ​​of the model parameters of the large language model after fine-tuning. This represents the sigmoid function.

[0083] The above process uses only a second sample image as an example to illustrate the process of continuing to train the supervised fine-tuned large language model using the hybrid preference offline reinforcement learning algorithm. In some possible examples, steps 0241-0243 can be executed multiple times for different second sample images to update and adjust the model parameters of the supervised fine-tuned large language model multiple times until the large language model after multiple updates and adjustments reaches the specified convergence condition, thus obtaining the large language model after the first stage of training. At this point, the large language model not only possesses the structured reasoning and basic capabilities provided by the SFT stage, but also learns the non-preference label data that needs to be rejected through the hybrid preference offline reinforcement learning algorithm, thereby enabling it to perform more accurate and fine-grained reasoning. Its reasoning process is more precise and detailed, and the accuracy of the detection results is higher.

[0084] The aforementioned specified convergence conditions may include, but are not limited to: determining that the value of the first objective function exceeds a specified threshold, the number of times the parameters of the large language model are iteratively adjusted exceeds a preset threshold, and the duration of fine-tuning the parameters of the large language model exceeds a preset duration.

[0085] Through the above process, pattern-guided cold-start training of large language models can be achieved, enabling the large language models to have basic reasoning ability to determine the authenticity of images.

[0086] Subsequently, in order to further improve the image authenticity detection and reasoning capabilities of the large language model, a second phase of training, namely online reinforcement learning training, was conducted on the large language model that had completed the first phase of training (hereinafter referred to as the first large language model).

[0087] The second phase of training (step 03) will be described below.

[0088] The second-stage training process includes the following steps 031-034: In step 031, any third sample image and its corresponding label data are obtained from the third training set. The label data may include a label indicating whether the corresponding third sample image is a real image (or a generated image). The sample images in the third training set may be sampled from the aforementioned first and / or second training sets, or may be obtained by referring to the method used to obtain real and generated images from the first training set.

[0089] Next, in step 032, the third sample image and its corresponding third sample text, which indicates whether the third sample image is genuine or fake, are input into the first language model, so that the first language model processes the third sample image and the third sample text based on its current first strategy to obtain G different inference detection prediction results.

[0090] In step 033, a preset reward function is used to determine G reward values ​​for the G inference detection prediction results. The preset reward function... It can be expressed by the following formula (2):

[0091] ; (2)

[0092] in, This indicates the pattern-aware reward value. Indicates the quality reward value for reflection. Represents the format of the reward value. and This indicates the parameter for adjusting the percentage of the corresponding reward value, which is a preset value. Let represent the indicator function, where the predicted image authenticity (which can be called the answer) in the inference detection prediction result is correct (i.e., ...). When the value is 1, the indicator function value is 1; otherwise, it is 0.

[0093] in, It is determined based on the correctness C of the answer in the reasoning test prediction results, and the presence and quality of the specified reasoning pattern (i.e., planning prediction data and reflection prediction data) in the reasoning test prediction results.

[0094] Specifically, if the answer in the reasoning test prediction result is correct (C=1), and the reasoning test prediction result includes both planning prediction data and reflection prediction data, then set... Set to the first value (e.g., 2, to incentivize the model to score highly) to encourage the top language model to arrive at the correct answer when faced with difficult images by using complex reasoning paths (planning the predicted data and / or reflecting on the predicted data).

[0095] If the reasoning test prediction result is correct (C=1), but does not include planning prediction data and reflection prediction data, then set... The second value (e.g., 1) is used to allow the first language model to give the correct answer directly when faced with simple images, without forcing excessive thinking.

[0096] If the reasoning test prediction result is incorrect (i.e., C is not equal to 1) and does not include planning prediction data and reflection prediction data, then set... It is the third value (e.g., 0).

[0097] If the reasoning test prediction result is incorrect (i.e., C is not equal to 1), and it includes planning prediction data (but does not include reflection prediction data), then set... The fourth value (e.g., -0.5) is used to penalize the first language model for ineffective planning reasoning (i.e., the first language model thinks a lot but thinks incorrectly).

[0098] If the reasoning detection prediction result is incorrect (i.e., C is not equal to 1), and includes reflective prediction data, then set... The fifth value (e.g., -1) is used to punish the behavior where the first major language model's answer is still wrong after reflection, indicating that the reflection direction is completely incorrect.

[0099] Among them, the first value is greater than the second value, the second value is greater than the third value, the third value is greater than the fourth value, and the fourth value is greater than the fifth value.

[0100] for In other words, it can be obtained by evaluating the quality of reflective prediction data in the inference detection prediction results through an external reward model (e.g., Reward Model, MM). The higher the quality of the reflective prediction data in the inference detection prediction results as evaluated by the external reward model, the better. The higher the value, the better. In some possible examples, this... Alternatively, the reward model, whether manually or externally configured, can be set based on the perspective of "whether the reflective prediction data in the reasoning detection prediction results offers a new perspective compared to the evidence-gathering reasoning prediction data in the reasoning detection prediction results." Specifically, the better the quality of the reflective prediction data in the reasoning detection prediction results, and the more new perspectives the reflective prediction data offers compared to the evidence-gathering reasoning prediction data in the reasoning detection prediction results, the higher the reward. The higher the value, the better.

[0101] for Specifically, the determination is based on whether the reasoning detection prediction result conforms to a predefined valid pattern format combination. This predefined valid pattern format combination is determined based on the label data corresponding to the third sample image. For example, if the label data corresponding to the third sample image includes first label data, planning label data, evidence collection reasoning label data, reflection label data, and conclusion label data, then the reasoning detection prediction result must include first prediction data (corresponding to the first label data), planning prediction data (corresponding to the planning label data), evidence collection reasoning prediction data (corresponding to the evidence collection reasoning label data), reflection prediction data (corresponding to the reflection label data), and conclusion prediction data (corresponding to the conclusion label data). In this case, the format of the reasoning detection prediction result is considered valid. The value is 1, and vice versa. The value is 0.

[0102] After determining the G reward values ​​of the G inference detection prediction results using the aforementioned preset reward function R, in step 034, the GRPO (Group Relative Policy Optimization) algorithm is used to calculate the G advantage values ​​A of the G inference detection prediction results based on the G reward values. Then, with the goal of maximizing the second objective function, the current first strategy of the first language model is adjusted, wherein the second objective function includes at least a strategy benefit term, which is determined based on the G advantage values ​​A of the G inference detection prediction results.

[0103] In some possible examples, the second objective function also includes a KL divergence term, which is determined based on G first prediction probability distributions and G fourth prediction probability distributions for each of the G inference detection prediction results. The first prediction probability distribution is the probability distribution of the corresponding inference detection prediction result output by the large language model based on the current policy, and the fourth prediction probability distribution is the probability of the fourth prediction probability distribution output by the large language model based on the policy of the first large language model.

[0104] Wherein, the second objective function It can be expressed by the following formula (3):

[0105] ; (3)

[0106] in, This represents the aforementioned strategy benefit item. The KL divergence term is represented by the second objective function, which is positively correlated with the policy return term and negatively correlated with the KL divergence term. The strategy for representing the largest language model; The participation strength used to control the KL divergence is a preset value; where J can be expressed by the following formula (4):

[0107] ; (4)

[0108] in, This indicates the calculation of the expected value. , , This represents the advantage value of the j-th inference detection prediction result. This represents the mean of the G dominance values. This represents the standard deviation of the G dominance values. This represents any input third-sample image and third-sample text. This represents the prediction result of the j-th inference detection; This indicates the current first strategy. This indicates the strategy preceding the current first strategy; This represents the shearing parameter, used to limit the step size of policy updates and ensure the stability of policy updates. It is usually set to a small constant. express Less than When, take the value , Greater than When, take the value In this context, the strategy of a large language model can refer to the values ​​of the model parameters.

[0109] The above process uses only a third sample image as an example to introduce the second-stage training process of the first large language model. In some possible examples, steps 031-034 can be executed multiple times for different third sample images to update and adjust the model parameters of the first large language model (i.e., the large language model trained in the first stage) multiple times until the large language model after multiple updates and adjustments reaches the corresponding convergence condition, thus obtaining the large language model trained in the second stage. At this time, the large language model not only possesses the structured reasoning and basic capabilities given by the SFT stage, but also learns the non-preference label data that needs to be rejected through the hybrid preference offline reinforcement learning algorithm. As a result, the large language model can perform more accurate and fine-grained reasoning, resulting in more precise and detailed reasoning processes and better detection results. It also has good adaptive and reflective capabilities, enabling better detection and recognition of unknown generation techniques.

[0110] The aforementioned convergence conditions may include, but are not limited to: determining that the value of the second objective function exceeds a preset threshold, the number of times the parameters of the first language model are iteratively adjusted exceeds a preset threshold, and the duration of fine-tuning the parameters of the first language model exceeds a preset duration.

[0111] The generalization ability of large language models can be transferred to the field of image authenticity (deepfake technology) detection using the methods described above. Labeled data constructed by simulating the thinking patterns of human experts introduces structured multi-step reasoning into image authenticity detection tasks, prompting large language models to perform comprehensive and in-depth analysis of forgery traces. Furthermore, the aforementioned online reinforcement exploration and pattern-based reward mechanism enhance the model's adaptive and reflective capabilities, enabling it to possess deep adaptive planning and reflective abilities when facing challenging and unknown forgery types.

[0112] The large language model trained in the above manner is obtained, and this trained large language model is used to provide image authenticity detection services.

[0113] Figure 3 A flowchart of an image detection method according to one embodiment of this specification is shown. This method is executed by an electronic device, which can be any device, equipment, platform, device cluster, etc., with computing and processing capabilities. During the image detection process, such as... Figure 3 As shown, the method includes the following steps S310-S340:

[0114] In step S310, a first image to be detected and a first text indicating that the first image should be tested for authenticity are obtained. In this step, the first image may be an image input by a user or other device that needs to be tested for authenticity; the first text may be a first text input by a user or other device indicating that the first image should be tested for authenticity, such as including the content "test the authenticity of the input image".

[0115] In some possible examples, the first image could be a human face image.

[0116] Then, in step S320, the first image and the first text are input into the large language model so that the large language model performs a first detection on the first image and obtains a first detection result. The large language model is trained based on the sample image and its corresponding label data. The label data includes inference label data that guides the large language model to perform multi-step inference and conclusion label data that indicates the authenticity of the corresponding sample image. The inference label data includes: first label data and second label data. The first label data is used to guide the large language model to perform a first detection based on the global image of the corresponding sample image, and the second label data is used to guide the large language model to perform a second detection based on the image details of the corresponding sample image.

[0117] In this step, the electronic device inputs the first image and the first text into the large language model, enabling the large language model to process the first image and the first text, perform a first detection on the first image, and obtain a first detection result. At this point, the large language model can be instructed to pause the subsequent second detection. This large language model is the one completed in the aforementioned second training phase.

[0118] In some possible examples, a designated storage space on the electronic device may pre-store a first cue template, which includes content instructing the large language model to perform multi-step inference (i.e., first detection and second detection) on the first image based on the instructions of the first text. It may also include cue content instructing the large language model to pause subsequent second detections after obtaining the first detection result.

[0119] The aforementioned input of the first image and the first text into the large language model can be achieved by adding the first image and the first text to their respective corresponding addition positions in the first prompt template to obtain the first prompt content, and then inputting the first prompt text content into the large language model.

[0120] In some possible examples, as previously described, the second label data may include: planning label data, evidence-gathering reasoning label data, and reflection label data. The planning label data includes: the detection strategy content for image details of the corresponding sample images. The evidence-gathering reasoning label data includes: the image evidence acquisition and analysis verification steps required to be performed according to the detection strategy content, and the results corresponding to the analysis and verification steps. The reflection label data includes: content reflecting a rethinking and exploration of the authenticity assessment results and related image evidence represented by the first label data and the evidence-gathering reasoning label data.

[0121] Next, in step S330, the confidence level is determined based on the first detection result. In this step, the electronic device determines the confidence level based on the first detection result, that is, whether the first detection result (i.e., the rapid judgment result) performed by the large language model on the authenticity of the first image is reliable. Then, based on the confidence level, the subsequent process is dynamically decided.

[0122] In some possible examples, the aforementioned first detection result includes the word sequence corresponding to the first output text and the first probability distribution corresponding to the position of each word; wherein, the first probability distribution corresponding to the position of any word includes the probability of each word in the vocabulary appearing at the corresponding position.

[0123] The first output text may include: the preliminary judgment result of the authenticity of the first image obtained by the large language model through the first detection (based on the overall visual features of the image), and may also include the content of the process determined by the large language model that requires a second detection to obtain an accurate image authenticity detection result. For example, the first output text could be: "At first glance, the clarity is not high enough, but this is most likely caused by camera shake; it looks real." Another example of the first output text could be: "At first glance, it looks very real, but the skin texture of the face is too smooth and flawless, making it difficult to determine its authenticity without further examination."

[0124] Considering the probability distribution of each word at each position determined by the large language model during the generation of the first output text, it can, to some extent, characterize the confidence level of the large language model in its initial detection of image authenticity. For example, if, for a certain position, the probability value of word A in its corresponding first probability distribution is much greater than the probability values ​​of other words (the probability value of word A exceeds 90%, while the sum of the probability values ​​of other words does not exceed 10%), it indicates that the large language model is very confident in its decision to output word A at that position. Conversely, if, for a certain position, the probability values ​​of each word in its corresponding first probability distribution are relatively balanced (for example, the difference between the maximum and minimum probability values ​​is lower than a certain difference threshold, such as 10%), it indicates that the large language model is not very confident in its decision to output word A at that position.

[0125] Based on the above considerations, step S330 may include steps 11-13:

[0126] In step 11, a first lexical unit is determined from the lexical unit sequence, wherein the first lexical unit is a lexical unit that represents the true meaning or the generated meaning.

[0127] In some possible examples, the storage area corresponding to the electronic device may store a set of lexical units representing the real meaning (hereinafter referred to as the first set), such as including but not limited to the following lexical units corresponding to words that can indicate that the image is a real image: {'real', 'authentic', 'genuine', 'camera', 'natural'}, and a set of lexical units representing the generated (i.e. fake) meaning (hereinafter referred to as the second set), such as including but not limited to the following lexical units corresponding to words that can indicate that the image is a generated image (fake image): {'fake', 'synthetic', 'generated', 'machine', 'artificial', 'algorithmical'}.

[0128] The lexical units in both sets are included in the aforementioned vocabulary, which is the vocabulary corresponding to the large language model.

[0129] After receiving the first detection result, the electronic device determines the lexical sequence corresponding to the first output text. Then, based on the first set and the second set, it determines the first lexical unit from the lexical sequence. This first lexical unit is either a lexical unit representing the true meaning or a lexical unit representing the generated meaning. In some examples, if the lexical sequence includes multiple lexical units representing the true meaning or a lexical unit representing the generated meaning, the lexical unit representing the true meaning or a lexical unit that appears first in the lexical sequence can be taken as the first lexical unit.

[0130] Then, in step 12, the first probability value corresponding to the first word element and the second probability value corresponding to the second word element are extracted from the first probability distribution corresponding to the position of the first word element, wherein the second word element has the opposite meaning to the first word element.

[0131] In this step, after the electronic device determines the first lexical unit, it determines the first probability distribution corresponding to the position of the first lexical unit from the first probability distribution corresponding to the positions of each lexical unit in the lexical unit sequence. This first probability distribution includes the probability value of each lexical unit in the vocabulary appearing at the position of the first lexical unit. From the first probability distribution corresponding to the position of the first lexical unit, the first probability value corresponding to the first lexical unit and the second probability value corresponding to the second lexical unit are extracted. The second lexical unit has the opposite meaning to the first lexical unit; that is, if the first lexical unit represents the true meaning, the second lexical unit represents the generated meaning; if the first lexical unit represents the generated meaning, the second lexical unit represents the generated true meaning.

[0132] Next, in step 13, the confidence level is determined based on the first probability value and the second probability value. In this step, the electronic device combines the first probability value and the second probability value to determine the confidence level corresponding to the first detection result.

[0133] In some possible examples, step 13 may include steps 131-132:

[0134] In step 131, the first probability value and the second probability value are normalized to obtain normalized first probability value and normalized second probability value, respectively. In this step, the electronic device can use a specified normalization function to normalize the first probability value and the second probability value, respectively. For example, the normalization function can be, but is not limited to, the Softmax function. The normalized first probability value... Taking this as an example, the normalized first probability value can be expressed by the following formula (5):

[0135] ; (5)

[0136] in, This represents the first probability value corresponding to the first word element mentioned above. This represents the second probability value corresponding to the aforementioned second lexical unit; This represents an exponential function.

[0137] Similarly, referring to the aforementioned formula (5), the normalized second probability value can be obtained. .

[0138] Then, in step 132, the normalized first probability value is... and the normalized second probability value The absolute value of the difference between the two probabilities is determined as the confidence level. In this step, the electronic device calculates the absolute value of the difference between the normalized first probability value and the normalized second probability value, and uses this as the confidence level of the large language model for its first detection result. The larger the absolute value of the difference between the normalized first probability value and the normalized second probability value, the greater the confidence level, which indicates a higher degree of certainty in the large language model's first detection result.

[0139] In some possible examples, the electronic device may also directly calculate the absolute value of the difference between the first probability value and the second probability value, which will be determined as the confidence level.

[0140] In some other possible examples, the first detection result includes the word sequence corresponding to the first output text and the hidden state vector corresponding to each word. The last word in the word sequence is a designated end-of-word symbol indicating the completion of the first output text generation. It can be understood that the last word in the word sequence corresponding to the first output text is the designated end-of-word symbol indicating the completion of the first output text generation, and its corresponding hidden state vector aggregates the visual and textual information accumulated during the first detection (inference) process.

[0141] The corresponding step S330 may include steps 21-23:

[0142] In step 21, the hidden state vector corresponding to the specified terminator is processed through the head structure of the large language model to obtain the second probability distribution of each word in the vocabulary.

[0143] In this step, the electronic device can input the hidden state vector corresponding to the specified terminator into the head structure of the large language model. The head structure of the large language model processes the hidden state vector corresponding to the specified terminator, mapping this hidden state vector onto the probability distribution of the vocabulary (i.e., the logits distribution), thus obtaining the second probability distribution of each word in the vocabulary. The second probability distribution of each word includes the probability value predicted by the head structure based on the hidden state vector corresponding to the specified terminator, which aggregates visual and textual information accumulated during the first detection (inference) process, indicating the probability of the corresponding word appearing at the next position.

[0144] Next, in step 22, the third probability value corresponding to the third word element and the fourth probability value corresponding to the fourth word element are extracted from the second probability distribution. The third word element represents the true meaning, and the fourth word element represents the generated meaning. In this step, the electronic device extracts the third probability value corresponding to the third word element and the fourth probability value corresponding to the fourth word element from the second probability distribution. For example, when both the aforementioned first set and second set include word elements corresponding to multiple words, the electronic device can extract the probability values ​​corresponding to the specified word elements in the first set and the second set from the second probability distribution to obtain the third probability value corresponding to the third word element and the fourth probability value corresponding to the fourth word element, respectively.

[0145] Then, in step 23, the confidence level is determined based on the third probability value and the fourth probability value. The implementation principle of step 23 is similar to that of step 13 mentioned above, and its implementation process can be found in the implementation process of step 13 mentioned above, so it will not be repeated here.

[0146] In the above process, the head structure of the large language model can be considered as a binary word classifier. Based on the hidden state vector corresponding to the specified end symbol of the first output text, the head structure predicts the probability of each word appearing at the next position. Among the probability values ​​of each word in this class, the third probability value corresponding to the third word and the fourth probability value corresponding to the fourth word are equivalent to the large language model's tendency to decide the authenticity of the image based on the current information (i.e., the hidden state vector that aggregates visual and textual information accumulated during the first detection (inference) process). The relative magnitude between the third probability value corresponding to the third word and the fourth probability value corresponding to the fourth word can, to a certain extent, objectively reflect the large language model's preference for these two possible decision answers, that is, it can characterize the certainty of the large language model for its first detection result.

[0147] After obtaining the confidence level of the first detection result through the above process, in step S340, if the confidence level is lower than a specified threshold, the large language model is triggered to perform a second detection on the first image. In this step, the electronic device judges the magnitude of the aforementioned confidence level and the specified threshold. If it is determined that the confidence level is lower than the specified threshold, it can be considered that the large language model has low certainty about its first detection result, that is, it has low confidence in the content of the first detection result. At this time, the large language model can continue to be triggered to perform a second detection on the first image.

[0148] In some possible instances, step S340 may specifically include: inputting the context information corresponding to the first detection result into a large language model, and using the large language model to perform a second detection on the first image based on the context information. In this step, when the electronic device obtains the first detection result generated by the large language model, it can simultaneously obtain the context information corresponding to the first detection result at the end of the generation of the first detection result by the large language model. If the confidence level is determined to be lower than a specified threshold, the context information corresponding to the first detection result is input into the large language model, and the large language model processes the context information to continue performing a second detection on the first image.

[0149] Subsequently, in some examples, the electronic device may display to the user the first output text in the aforementioned first detection result generated by the large language model, and the second output text obtained by performing a second detection on the first image. The second output text may include, but is not limited to, the content of planning prediction data, evidence reasoning prediction data, reflection prediction data, and conclusion prediction data obtained from the first image and the aforementioned first output text.

[0150] In some possible examples, a designated storage space on the electronic device may pre-store a second prompt template, which instructs the large language model to perform a second detection on the first image based on the context information corresponding to the first detection result. In some examples, to better prompt the large language model to continue performing a second detection on the first image, it may also include prompting the large language model that it has now completed the first detection of the image and needs to continue generating planning prediction data, evidence-gathering reasoning prediction data, reflection prediction data, and conclusion prediction data based on the context information corresponding to the first detection result.

[0151] The planning prediction data includes: the image detail prediction detection strategy generated by the large language model based on the context information corresponding to the first detection result; the evidence collection reasoning prediction data includes: the image evidence acquisition and analysis verification steps to be performed in accordance with the aforementioned prediction detection strategy, and the results corresponding to the analysis and verification steps; the reflection prediction data includes: the content of rethinking and mining the authenticity evaluation results and related image evidence represented by the first detection result and the evidence collection reasoning prediction data.

[0152] The conclusion prediction data includes: the comprehensive judgment (i.e. the judgment on the authenticity of the first image) formed after the big language model integrates the first detection result, the content of the prediction detection strategy, the specific evidence of the evidence reasoning prediction data, and the in-depth insight of reflecting on the prediction data (the content of rethinking and mining).

[0153] In some possible examples, such as Figure 3 As shown, the image detection method may further include step S350: In step S350, if the confidence level is not lower than a specified threshold, the large language model is truncated for a second detection. In this step, if the electronic device determines that the confidence level is not lower than the specified threshold, it can determine that the large language model has a high certainty regarding its first detection result. At this time, in order to reduce the model inference cost of image authenticity detection, the large language model can be truncated for the second detection. For example, the context information corresponding to the first detection result is no longer input into the large language model.

[0154] Subsequently, in some examples, the electronic device can display to the user the first output text from the aforementioned first detection result generated by the large language model. In other examples, while displaying the first output text, the electronic device can also display to the user content indicating that the confidence level of the first detection result is not lower than a specified threshold, and can output a prompt message to suggest that the user can manually trigger the large language model to continue performing a second detection on the first image.

[0155] In the above process, a large language model is trained using sample images and their corresponding label data (including inference label data guiding the large language model to perform multi-step inference and conclusion label data indicating the authenticity of the corresponding sample images). The inference label data includes: first label data to guide the large language model to perform a first detection based on the overall image, and second label data to guide the large language model to perform a second detection based on image details. This allows the large language model to be injected with anthropomorphic image authenticity analysis and identification capabilities (i.e., the ability to first identify authenticity from the overall image, and then perform more detailed authenticity identification based on local image details), enabling it to perform multi-step inference analysis on image authenticity and thus draw conclusions. Combined with the inherent inference characteristics of the large language model, the above process can, to some extent, solve the generalization problem caused by the rapid iteration of deepfake technology, achieving accurate detection of images generated by various deepfake techniques.

[0156] Furthermore, the above process can dynamically and adaptively decide whether to trigger a second detection based on image details (i.e., long thought chain detection) based on the confidence level of the first detection result (i.e., the credibility assessment of the model's inference results). It not only achieves comprehensive analysis and identification of various deepfake techniques (including new and existing techniques), but also effectively balances the computational cost and accuracy in the detection process.

[0157] The foregoing description describes specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than those shown in the embodiments, and the desired result may still be achieved. Furthermore, the processes depicted in the drawings do not necessarily need to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0158] Corresponding to the above method embodiments, this specification provides an image detection device 400, the schematic block diagram of which is shown below. Figure 4 As shown, it includes:

[0159] The acquisition module 410 is configured to acquire a first image to be detected and a first text indicating the authenticity detection of the first image;

[0160] Input module 420 is configured to input the first image and the first text into a large language model, so that the large language model performs a first detection on the first image and obtains a first detection result. The large language model is trained based on sample images and their corresponding label data. The label data includes inference label data that guides the large language model to perform multi-step inference and conclusion label data that indicates the authenticity of the corresponding sample images. The inference label data includes: first label data and second label data. The first label data is used to guide the large language model to perform a first detection based on the global image of the corresponding sample images, and the second label data is used to guide the large language model to perform a second detection based on the image details of the corresponding sample images.

[0161] The determination module 430 is configured to determine its confidence level based on the first detection result;

[0162] Trigger module 440 is configured to trigger the large language model to perform the second detection on the first image when the confidence level is lower than a specified threshold.

[0163] In some possible examples, the first detection result includes the word sequence corresponding to the first output text and the first probability distribution corresponding to the position of each word;

[0164] The determining module 430 is specifically configured to determine a first word element from the word element sequence, wherein the first word element is a word element that represents the true meaning or represents the generated meaning;

[0165] From the first probability distribution corresponding to the position of the first word, extract the first probability value corresponding to the first word and the second probability value corresponding to the second word, wherein the second word has the opposite meaning to the first word;

[0166] The confidence level is determined based on the first probability value and the second probability value.

[0167] In some possible examples, the determining module 430 is specifically configured to normalize the first probability value and the second probability value to obtain the normalized first probability value and the normalized second probability value, respectively.

[0168] The absolute value of the difference between the normalized first probability value and the normalized second probability value is determined as the confidence level.

[0169] In some possible examples, the first detection result includes: the word sequence corresponding to the first output text and the hidden state vector corresponding to each word, wherein the last word of the word sequence is a specified end symbol indicating that the first output text has been generated.

[0170] The determining module 430 is specifically configured to process the hidden state vector corresponding to the specified terminator through the head structure of the large language model to obtain the second probability distribution of each word in the vocabulary.

[0171] From the second probability distribution, extract the third probability value corresponding to the third word element and the fourth probability value corresponding to the fourth word element, wherein the third word element is a word element representing the true meaning and the fourth word element is a word element representing the generated meaning;

[0172] The confidence level is determined based on the third probability value and the fourth probability value.

[0173] In some possible examples, a truncation module (not shown in the figure) is also included, configured to truncate the large language model for the second detection if the confidence level is not lower than the specified threshold.

[0174] In some possible examples, the second tag data includes: planning tag data, evidence-gathering reasoning tag data, and reflection tag data;

[0175] The planning label data includes: the content of the image detail detection strategy for the corresponding sample images;

[0176] The evidence collection and reasoning label data includes: the image evidence acquisition and analysis verification steps required to be performed according to the detection strategy, and the results corresponding to the analysis and verification steps;

[0177] The rethinking label data includes: content that rethinks and explores the authenticity assessment results and related image evidence represented by the first label data and the evidence reasoning label data.

[0178] In some possible examples, the triggering module 440 is specifically configured to input the context information corresponding to the first detection result into the large language model, and perform a second detection on the first image based on the context information through the large language model.

[0179] The above-described apparatus embodiments correspond to the method embodiments, and detailed descriptions can be found in the description of the method embodiments section, which will not be repeated here. The apparatus embodiments are derived based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments; detailed descriptions can be found in the corresponding method embodiments.

[0180] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the image detection method provided in this specification.

[0181] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the image detection method provided in this specification.

[0182] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for storage media and computing devices are basically similar to the method embodiments, so they are described more simply; relevant parts can be referred to the descriptions of the method embodiments.

[0183] Those skilled in the art will recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0184] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made based on the technical solutions of the present invention should be included within the scope of protection of the present invention.< / answer> < / conclusion>

Claims

1. An image detection method, comprising: Acquire a first image to be detected and a first text indicating that the first image should be tested for authenticity; The first image and the first text are input into a large language model, so that the large language model performs a first detection on the first image and obtains a first detection result. The large language model is trained based on sample images and their corresponding label data. The label data includes inference label data that guides the large language model to perform multi-step inference and conclusion label data that indicates the authenticity of the corresponding sample images. The inference label data includes: first label data and second label data. The first label data is used to guide the large language model to perform a first detection based on the global image of the corresponding sample images. The second label data is used to guide the large language model to perform a second detection based on the image details of the corresponding sample images. Based on the first detection result, its confidence level is determined; If the confidence level is lower than a specified threshold, the large language model is triggered to perform the second detection on the first image.

2. The method as described in claim 1, wherein, The first detection result includes the word sequence corresponding to the first output text and the first probability distribution corresponding to the position of each word; Determining its confidence level includes: The first word element is determined from the word element sequence, wherein the first word element is a word element that represents the true meaning or the generated meaning; From the first probability distribution corresponding to the position of the first word, extract the first probability value corresponding to the first word and the second probability value corresponding to the second word, wherein the second word has the opposite meaning to the first word; The confidence level is determined based on the first probability value and the second probability value.

3. The method as described in claim 2, wherein, Determining the confidence level based on the first probability value and the second probability value includes: The first probability value and the second probability value are normalized to obtain the normalized first probability value and the normalized second probability value, respectively. The absolute value of the difference between the normalized first probability value and the normalized second probability value is determined as the confidence level.

4. The method of claim 1, wherein, The first detection result includes: the word sequence corresponding to the first output text and the hidden state vector corresponding to each word, wherein the last word of the word sequence is a specified end symbol indicating that the first output text has been generated. Determining its confidence level includes: By processing the hidden state vector corresponding to the specified terminator through the head structure of the large language model, the second probability distribution of each word in the vocabulary is obtained. From the second probability distribution, extract the third probability value corresponding to the third word element and the fourth probability value corresponding to the fourth word element, wherein the third word element is a word element representing the true meaning and the fourth word element is a word element representing the generated meaning; The confidence level is determined based on the third probability value and the fourth probability value.

5. The method of claim 1, further comprising: If the confidence level is not lower than the specified threshold, the large language model is truncated for the second detection.

6. The method of claim 1, wherein, The second set of tag data includes: planning tag data, evidence collection and reasoning tag data, and reflection tag data; The planning label data includes: the content of the image detail detection strategy for the corresponding sample images; The evidence collection and reasoning label data includes: the image evidence acquisition and analysis verification steps required to be performed according to the detection strategy, and the results corresponding to the analysis and verification steps; The rethinking label data includes: content that rethinks and explores the authenticity assessment results and related image evidence represented by the first label data and the evidence reasoning label data.

7. The method of claim 1, wherein, The triggering of the large language model to perform the second detection on the first image includes: The context information corresponding to the first detection result is input into the large language model, and the first image is then subjected to a second detection based on the context information using the large language model.

8. An image detection apparatus, comprising: The acquisition module is configured to acquire a first image to be detected and a first text indicating whether the first image is genuine or fake. The input module is configured to input the first image and the first text into a large language model, so that the large language model performs a first detection on the first image and obtains a first detection result. The large language model is trained based on sample images and their corresponding label data. The label data includes inference label data that guides the large language model to perform multi-step inference and conclusion label data that indicates the authenticity of the corresponding sample images. The inference label data includes: first label data and second label data. The first label data is used to guide the large language model to perform a first detection based on the global image of the corresponding sample images, and the second label data is used to guide the large language model to perform a second detection based on the image details of the corresponding sample images. The determination module is configured to determine its confidence level based on the first detection result; The triggering module is configured to trigger the large language model to perform the second detection on the first image when the confidence level is lower than a specified threshold.

9. The apparatus of claim 8, wherein, The first detection result includes the word sequence corresponding to the first output text and the first probability distribution corresponding to the position of each word; The determining module is specifically configured to determine a first word element from the word element sequence, wherein the first word element is a word element that represents the true meaning or represents the generated meaning; From the first probability distribution corresponding to the position of the first word, extract the first probability value corresponding to the first word and the second probability value corresponding to the second word, wherein the second word has the opposite meaning to the first word; The confidence level is determined based on the first probability value and the second probability value.

10. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-7.

Images