Data authenticity detection method, electronic equipment, storage medium and program product

Abstract

The invention discloses a data authenticity detection method, an electronic device, a storage medium and a program product, and relates to the technical field of data processing. A pre-trained authenticity detection model in the application can identify the authenticity of input data fused by image data and text data; and selecting the target detection result corresponding to the authenticity of the input data from the output options, and outputting the target description data corresponding to the target detection result according to the characteristics of the input data to explain the reason for determining the target detection result, so that the technical problem of low detection precision of the fused data can be solved, and the detection accuracy of the fused data is improved. And the detection result is explained while the precision of the detection result of the image-text fusion data is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data authenticity detection method, electronic equipment, storage medium and program product. Background Art

[0002] With the development of multimedia technology, the processing methods of various modal media data (such as image data, text data, audio data, etc.) are becoming more and more diverse. In practical applications, some users may tamper with the feature information or semantic content of media data and use the complex relationship between modalities to hide the traces of tampering operations. Therefore, users need to identify the authenticity of various modal media data.

[0003] Related detection methods for identifying the authenticity of media data include single-modal detection methods such as image authenticity detection methods, text authenticity detection methods, and audio authenticity detection methods, as well as fused modal data detection methods. Among them, single-modal detection methods can only be used to detect the authenticity of data in a single modality, and cannot be used to identify the authenticity of data in other modalities. The related fused modal data detection methods have low detection accuracy for fused data (such as fused data of image data and text data, and fused data of text data and audio data), and cannot correctly identify the authenticity of fused data. Summary of the invention

[0004] The present application provides a data authenticity detection method, electronic device, storage medium and program product to at least solve the problem of low authenticity detection accuracy of fused data in related technologies.

[0005] This application provides a data authenticity detection method, including:

[0006] Obtaining input data and output options; wherein the input data includes image data and text data, and the output options include at least one detection result;

[0007] Input data and output options are input into a pre-trained authenticity detection model to obtain a target detection result of the input data output by the pre-trained authenticity detection model, and target description data corresponding to the target detection result; wherein the target detection result is used to indicate the authenticity of the input data.

[0008] The present application also provides a data authenticity detection device, comprising:

[0009] An acquisition module, used to acquire input data and output options; wherein the input data includes image data and text data, and the output options include at least one detection result;

[0010] The detection module is used to input input data and output options into a pre-trained authenticity detection model to obtain a target detection result of the input data output by the pre-trained authenticity detection model, and target description data corresponding to the target detection result; wherein the target detection result is used to indicate the authenticity of the input data.

[0011] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned data authenticity detection methods when executing the computer program.

[0012] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned data authenticity detection methods are implemented.

[0013] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data authenticity detection methods when executed by a processor.

[0014] Through this application, since the pre-trained authenticity detection model can identify the authenticity of the input data (the fusion data of image data and text data), and select the target detection result corresponding to the authenticity of the input data from the output options, and output the target description data corresponding to the target detection result according to the characteristics of the input data to explain the reason for determining the target detection result. Therefore, the technical problem of low detection accuracy of fusion data can be solved, and the detection results can be explained while improving the accuracy of the detection results of the image and text fusion data, so as to facilitate user understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 A schematic diagram of a scenario for data authenticity detection provided in an embodiment of the present application;

[0017] Figure 2 One of the flow diagrams of data authenticity detection provided in the embodiment of the present application;

[0018] Figure 3 A schematic diagram of the structure of a pre-trained authenticity detection model provided in an embodiment of the present application;

[0019] Figure 4 The second flowchart of data authenticity detection provided in the embodiment of the present application;

[0020] Figure 5 A schematic diagram of an application for determining an attention score provided in an embodiment of the present application;

[0021] Figure 6 Another structural schematic diagram of a pre-trained judgment module provided in an embodiment of the present application;

[0022] Figure 7 The third flowchart of the data authenticity detection provided in the embodiment of the present application;

[0023] Figure 8 The fourth flowchart of the data authenticity detection provided in the embodiment of the present application;

[0024] Fig. 9 A schematic diagram of a fusion data authenticity detection process provided in an embodiment of the present application;

[0025] Fig.10 A schematic diagram of the structure of a data authenticity detection device provided in an embodiment of the present application;

[0026] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0029] Batch size refers to the uniform resizing of a group of images when processing them in batches.

[0030] Deep Learning (DL) refers to machine learning based on deep neural network models and methods. The characteristic of deep learning is the ability to automatically extract features. Deep neural network is the model basis for deep learning to achieve automatic feature extraction.

[0031] The backbone network refers to the core part of the entire neural network architecture, which is responsible for extracting the feature representation of the input data.

[0032] A fully connected layer (FC) is a layer in which each node is connected to all nodes in the previous layer and is used to integrate the features extracted previously.

[0033] Multi-layer Perceptron (MLP) refers to a feed-forward artificial neural network model that maps multiple input data sets to a single output data set.

[0034] Batch Normalization layer (BN), or simply Batch Normalization, is a technique commonly used in deep learning to accelerate the training of neural networks and improve the convergence speed.

[0035] Convolution (Conv), also known as stacked product, is a mathematical operator that generates a third function through two functions.

[0036] Convolutional neural network (CNN) refers to a feedforward neural network that includes convolution calculations and has a deep structure. It is a deep learning algorithm. Convolutional neural networks have representation learning capabilities and can perform shift-invariant classification on input data according to the hierarchical structure of convolutional neural networks.

[0037] Average pooling refers to a pooling method commonly used in convolutional neural networks (CNNs) to reduce the dimension of feature maps, reduce the amount of calculation, and retain important feature information.

[0038] Max-pooling refers to a pooling operation in deep learning, which is used in convolutional neural networks (CNN) to reduce the dimension of feature maps, extract significant features, and reduce the amount of calculation.

[0039] Layer refers to the basic building block in a neural network model, which consists of a set of learnable parameters and a data conversion method.

[0040] Feature maps are the output of the convolutional layer in a convolutional neural network (CNN). They are two-dimensional arrays that contain the features extracted by the convolution filter from the input image or signal.

[0041] Stochastic gradient descent (SGD) refers to an optimization algorithm widely used in machine learning, mainly for parameter optimization problems.

[0042] Momentum algorithm refers to an optimization method of gradient descent algorithm, which is used to solve the problem that traditional gradient descent is prone to fall into local minimum or oscillate near saddle point.

[0043] Depthwise Separable Convolution refers to an algorithm that improves the standard convolution calculation in convolutional neural networks. It can reduce the number of parameters required for convolution calculation by splitting the correlation between spatial dimensions and channel (depth) dimensions.

[0044] Concat is usually used to combine features and is a common feature fusion method in deep learning. It combines feature information from different branches or different levels by concatenating different feature maps together according to a certain dimension (usually the channel dimension).

[0045] Sampling refers to replacing the original continuous signal in time with a sequence of signal sample values ​​at regular intervals, that is, discretizing the analog signal in time.

[0046] Pre-training refers to a strategy for deep learning model training. Its core is to use large-scale data sets to conduct preliminary training on the model so that the model can learn common feature representations. This process is similar to the basic learning stage of humans before learning new knowledge, accumulating experience through extensive reading and observation.

[0047] A pre-trained large language model generally refers to a large language model training task designed based on a large-scale corpus (including language training materials such as sentences and paragraphs), and a large-scale neural network algorithm structure is trained to learn and implement. The final large-scale neural network algorithm structure and parameters are the pre-trained large language model. Other tasks can be used to extract features or fine-tune tasks based on this model to achieve specific task purposes. The idea of ​​pre-training is to first train a task to obtain a set of model parameters, then use this set of model parameters to initialize the network model parameters, and then use the initialized network model to train other tasks to obtain models adapted to other tasks. By pre-training on a large-scale corpus, the neural language representation model can learn powerful language representation capabilities and can extract rich syntactic and semantic information from the text. The pre-trained large language model can provide tokens and sentence-level features containing rich semantic information for use in downstream tasks, and can also be fine-tuned directly on the pre-trained model for downstream tasks, so as to obtain downstream-specific models conveniently and quickly.

[0048] The neural network algorithm structure of the pre-trained large language model training can be a large language model (LLM), or CNN, RNN, LSTM, etc., or a model built with an attention network, such as transformer, bert, GPT, Clip, etc., which is not limited in this application. An attention network refers to a network model trained using an attention mechanism. The model assigns different weights to each part of the input sequence, thereby extracting more important feature information from the input sequence, so that the model ultimately obtains a more accurate output.

[0049] The activation function layer (Softmax layer) is often used in deep learning to convert real numbers into probability distributions.

[0050] With the development of multimedia technology, the processing methods of various modal media data (such as image data, text data, audio data, etc.) are becoming more and more diverse. In practical applications, some users may tamper with the feature information or semantic content of media data and use the complex relationship between modalities to hide the traces of tampering operations. Therefore, users need to identify the authenticity of various modal media data.

[0051] Related detection methods for identifying the authenticity of media data include single-modal detection methods such as image authenticity detection methods, text authenticity detection methods, and audio authenticity detection methods, as well as fused modal data detection methods. Among them, single-modal detection methods can only be used to detect the authenticity of data in a single modality, and cannot be used to identify the authenticity of data in other modalities. The related fused modal data detection methods have low detection accuracy for fused modal data (such as fused data of image data and text data, and fused data of text data and audio data), and cannot correctly identify the authenticity of the fused data. In addition, users cannot determine the reasons for the detection results of the above-mentioned fused modal data detection methods, and the credibility of the detection results is low.

[0052] In order to solve the above problems, the present application provides a data authenticity detection method, electronic device, storage medium and program product, which obtains input data of image data and text data and output options including at least one detection result, and inputs the input data and output options into a pre-trained authenticity detection model to obtain a target detection result of the authenticity of the input data and target description data corresponding to the target detection result. In this way, the authenticity of the input data can be detected by the pre-trained authenticity detection model, thereby improving the accuracy of the detection result.

[0053] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0054] In combination with the specific application environment architecture or the specific hardware architecture on which the execution of the data authenticity detection method depends, the specific application environment architecture or the specific hardware architecture is described here. Figure 1 , Figure 1 A schematic diagram of a scenario for data authenticity detection provided in an embodiment of the present application.

[0055] Taking the electronic device as a smartphone as an example, see Figure 1 When the smart phone obtains the graphic-text fusion data 101 and the output option 102, the graphic-text fusion data 101 and the output option 102 ( Figure 1 The output options 102 shown in the figure include true 1021, false 1022 and uncertain 1023) are input into the pre-trained authenticity detection model 103, and the target detection result of the above-mentioned image-text fusion data 101 output by the pre-trained authenticity detection model 103 is: the image-text fusion data is false data, and the target description data corresponding to the "false data": the number of clouds in the image-text fusion data is 4, which is different from the number of clouds in the text content "There is a white cloud floating in the sky", and is displayed on the display screen of the smartphone (such as interface 104).

[0056] Figure 2One of the flow diagrams of data authenticity detection provided in the embodiment of the present application is as follows: Figure 2 As shown, the embodiment of the present application provides a data authenticity detection method, and the method is described in detail as follows:

[0057] S201: Obtain input data and output options; wherein the input data includes image data and text data, and the output options include at least one detection result.

[0058] Specifically, the input data is used to indicate multimodal fusion data, which includes image data and text data. The output options include at least one detection result corresponding to the input data, so that the pre-trained authenticity detection model can select one from the at least one detection result as a target detection result of the input data based on the feature information of the input data and output it.

[0059] Exemplarily, the output options include three detection results: true, false, and uncertain. When the input data is true image-text fusion data, the target detection result output by the pre-trained authenticity detection model can be: true. When the input data is false image-text fusion data, the target detection result output by the pre-trained authenticity detection model can be: false. When the pre-trained authenticity detection model cannot identify the authenticity of the input data, the target detection result output by the pre-trained authenticity detection model can be: uncertain.

[0060] It can be understood that the authenticity of the input data includes but is not limited to: the image data is true and the text data is true; the image data is false and the text data is true; the image data is true and the text data is false; the image data is false and the text data is false.

[0061] The data authenticity detection method refers to a method of performing authenticity detection on fused data containing multiple types of information (such as text type, image type, audio type and other modalities) to determine whether it has been forged.

[0062] The characteristics of multimodal fusion data include: complementarity and complexity. Among them, complementarity means that information from different modalities complement each other. For example, image data can provide visual information, text data can describe semantic information, and audio data can convey sound-related information. When describing an event, image data may be used to display scene information, text data is used to explain background information and details, and audio data can be used to supplement environmental sound information. Complexity refers to the complex structure and relationship of multimodal data. There may be multiple associations between the various modes of multimodal data, such as semantics, time and space. For example: the synchronization relationship between image frames and audio data in video data, the semantic matching relationship between image data and related text data, and other associations.

[0063] Methods for falsifying multimodal data include intra-modal falsification and cross-modal falsification. Intra-modal falsification refers to falsification within a single modality. For example, image editing software is used to tamper with image data to modify objects, scenes or characters; audio editing tools are used to edit, mix or add false sound effects to audio data; text generation algorithms are used to generate false articles, etc. Cross-modal falsification refers to falsification using the relationship between different modalities. For example, an image generation model is used to generate matching image data based on false text information, or related text data is generated based on forged image data.

[0064] S202: Inputting input data and output options into a pre-trained authenticity detection model to obtain a target detection result of the input data output by the pre-trained authenticity detection model, and target description data corresponding to the target detection result; wherein the target detection result is used to indicate the authenticity of the input data.

[0065] Specifically, the input data and output options are input into a pre-trained authenticity detection model, the pre-trained authenticity detection model is used to identify the authenticity of the input data, and a target detection result indicating the authenticity of the input data and target description data corresponding to the target detection result are output.

[0066] The pre-trained authenticity detection model refers to a pre-trained neural network model used to detect the authenticity of input data.

[0067] It can be understood that based on the target description data corresponding to the target detection result, the reason why the pre-trained authenticity detection model outputs the target detection result can be determined.

[0068] For example, the target detection result is: true, and the target description data is: the texture features of the image data are consistent with the description of the text content. For another example, the target detection result is false, and the target description data is: the content of the image data is a person, and the content of the text data is an animal, and the image and text are inconsistent.

[0069] In some embodiments, the pre-trained authenticity detection model includes a pre-trained judgment module and a pre-trained large language model, and the pre-trained judgment module is connected to the pre-trained large language model.

[0070] Specifically, the pre-trained authenticity detection model includes a pre-trained judgment module and a pre-trained large language model, and the output end of the pre-trained judgment module is connected to the input end of the pre-trained large language model. Among them, the pre-trained judgment module refers to a pre-trained neural network module that can extract feature information (that is, the initial detection vector) for identifying the authenticity of input data. The pre-trained large language model refers to a pre-trained large language model that can determine the authenticity of input data and the corresponding description data based on the initial detection vector and output options.

[0071] The embodiment of the present application obtains input data that is a fusion of image and text and output options that include at least one detection result, and inputs the above input data and output options into a pre-trained authenticity detection model to obtain a target detection result that indicates the authenticity of the input data and target description data corresponding to the target detection result, thereby achieving rapid identification of the authenticity of the input data through the pre-trained authenticity detection model, thereby improving the accuracy of the authenticity identification results of the input data.

[0072] In some implementations, step S202 includes:

[0073] S2021: Input the input data into the pre-trained judgment module, and use the pre-trained judgment module to extract an initial detection vector of the input data.

[0074] Specifically, the input data is input into a pre-trained judgment module, and the pre-trained judgment module is used to extract feature information of image data and feature information of text data in the input data, and the corresponding initial detection vector is obtained through vector representation, wherein the initial detection vector includes but is not limited to image feature vectors of image data and text feature vectors of text data.

[0075] Optionally, the feature information of the image includes but is not limited to texture features, color features, etc. The above feature information is identified and extracted by a pre-trained judgment module to form an image feature vector of the image data.

[0076] Optionally, the text data may also carry a label, and the corresponding initial detection vector may also include a feature vector of the label.

[0077] Optionally, the initial detection vector It can be expressed by the following formula:

[0078] .

[0079] represents a specific conversion function, and S represents the similarity feature vector between image data and text data.

[0080] The process of using a pre-trained judgment module to determine the initial detection vector can also be called a trace reasoning process, which is used to identify the authenticity of multimodal information. For example, forged image data may leave forgery traces at the pixel level, including discontinuity of the edge of the image data, abnormal color distribution, and specific patterns left by the image compression algorithm. For example, the spliced ​​image data may have color and texture mismatches at the splicing edge, or after scaling, rotating, and other operations on the image data, the image data will have traces of pixel distortion. Forged audio data may have specific patterns such as abnormal background noise, unnatural speech rhythm, and inconsistent audio spectrum features with real speech. For video data, in addition to the traces of image data and audio data, the frame rate stability of video data, the coherence of lens switching, etc. can also be used as specific patterns to determine forgery traces. Through trace reasoning with a pre-trained judgment module, it is possible to accurately determine whether the multimodal fusion data is forged, and to distinguish between real feature information and forged feature information to determine the authenticity of the multimodal fusion data.

[0081] For example, in the image-text fusion data, there may be cases where the image data is tampered with and the text data is fabricated. Through the pre-trained judgment module to perform trace reasoning, we can analyze clues such as the pixel change pattern in the image data, the language style and source of the text data. Forged image data may leave modified feature information in the edges and color distribution of the image. Forged text may be different from the feature information of real text data in terms of vocabulary usage frequency, grammatical structure, etc., and the forged feature information can be identified through trace reasoning of the pre-trained judgment module.

[0082] S2022: Input the initial detection vector and output options into the pre-trained large language model to obtain the target detection results and target description data output by the pre-trained large language model.

[0083] Specifically, the initial detection vector and output options output by the pre-trained judgment module are input into the pre-trained large language model, and the pre-trained large language model is used to identify the authenticity of the input data and the corresponding description data based on the initial detection vector.

[0084] In some embodiments, the pre-trained judgment module includes an image feature extraction module, a text feature extraction module, a similarity calculation module and an adaptor, and the image feature extraction module and the text feature extraction module are both connected to the similarity calculation module; the similarity calculation module is connected to the adaptor.

[0085] Figure 3 A schematic diagram of the structure of a pre-trained authenticity detection model provided in an embodiment of the present application.

[0086] See also Figure 3The pre-trained authenticity detection model 300 includes a pre-trained judgment module 301 and a pre-trained large language model 302. The pre-trained judgment module 301 includes an image feature extraction module 3011, a text feature extraction module 3012, a similarity calculation module 3013 and an adaptor 3014. The input end of the similarity calculation module 3013 is connected to the output end of the image feature extraction module 3011 and the output end of the text feature extraction module 3012, the output end of the similarity calculation module 3013 is connected to the input end of the adaptor 3014, and the output end of the adaptor 3014 is connected to the input end of the pre-trained large language model 302.

[0087] The similarity calculation module 3013 is used to determine the similarity feature vector between the image feature vector of the image data and the text feature vector of the text data. The adaptor 3014 is used to map the similarity feature vector into the input data of the pre-trained large language model 302, so as to facilitate the pre-trained large language model 302 to understand and generate natural language with explanatory power (easy for users to understand), and obtain the target recognition result and the corresponding target description data.

[0088] Figure 4 The second flow chart of data authenticity detection provided in the embodiment of the present application is as follows: Figure 4 As shown, step S2021 of the method includes:

[0089] S20211: dividing the image data into a plurality of sub-image data;

[0090] S20212: Determine the information entropy of each sub-image data, and determine the input image data set based on the information entropy of each sub-image data.

[0091] Specifically, the image data in the input data is divided into multiple sub-image data, and the image feature vector of each sub-image data is extracted according to the image feature extraction module to obtain multiple image feature vectors, which is convenient for the pre-trained authenticity detection model to deeply analyze the feature information of the image data and determine the feature pattern in each sub-image data, thereby determining whether the image data in the input data has traces of forgery.

[0092] Exemplarily, the number of sub-images may be preset.

[0093] For example, pre-set partition sub-image data, the image data is evenly divided into sub-image data of the same size.

[0094] For example, the size of the image data is Pixels, then Among the sub-image data, the size of each sub-image data is Pixel.

[0095] Specifically, the information entropy of each sub-image data in the image data is determined. The information entropy can indicate the complexity of the pixel values ​​in the sub-image data. The higher the information entropy value, the more complex the distribution of the sub-image data. The lower the information entropy, the simpler the distribution of the sub-image data. Based on this, the similarity between different sub-image data can be determined according to the information entropy, and the input image data set of the image data can be determined according to the information entropy of each sub-image data.

[0096] Exemplarily, if the information entropy difference between two sub-image data is small, it means that the similarity between the two sub-image data is high.

[0097] In some embodiments, determining the information entropy of each sub-image data, and determining the input image data set according to the information entropy of each sub-image data, includes:

[0098] Determine the information entropy of each sub-image data, and collect the position information of each sub-image data;

[0099] The information entropy and corresponding position information of each sub-image data are spliced ​​to obtain an input image data set; wherein the input image data set includes multiple input image data arranged in ascending or descending order based on information entropy; each input image data includes at least one sub-image data, information entropy and position information of the sub-image data.

[0100] Specifically, according to each pixel value in each sub-image data, the information entropy of each sub-image data is determined, and the position information of each sub-image data is collected, wherein the position information is used to represent the position coordinates of the sub-image data in the image data.

[0101] Still based on the size of the image data Taking pixels as an example, the coordinates of the center point of the image data are , in the horizontal direction, the width W of the image data is evenly divided into n parts, each with a length of Pixels, get n-1 vertical segmentation lines, the position of each vertical segmentation line in the image data is ,in, In the vertical direction, the height H of the image data is evenly divided into m parts, and the height of each part is pixels, and obtain m-1 horizontal segmentation lines. The position of each horizontal segmentation line in the image data is ,in, The coordinates of the sub-image data in the i-th row and j-th column are: . m represents the number of rows of sub-image data in the image data, and n represents the number of columns of sub-image data in the image data.

[0102] Specifically, each sub-image data is spliced ​​with the position information corresponding to the sub-image data to obtain the corresponding input image data, and each input image data is arranged based on the information entropy in each input image data to obtain an input image data set . , Represents the first Input image data.

[0103] Optionally, the arrangement of the input image data includes an ascending arrangement or a descending arrangement.

[0104] In some implementations, determining the information entropy of each sub-image data includes:

[0105] Acquire a pixel value set of each sub-image data; wherein the pixel value set includes each pixel value in the sub-image data;

[0106] Based on the pixel value set of each sub-image data, the probability of each pixel value in the sub-image data is determined, and according to the probability of each pixel value in the sub-image data, the information entropy of the sub-image data is determined.

[0107] Specifically, the pixel value set of each sub-image data is obtained, and the pixel value set is ,in, Indicates the length of the sub-image data, Indicates the width of the sub-image data, Represents the pixel value of the i-th row and j-th column in the sub-image data.

[0108] Specifically, according to the pixel value set of each sub-image data, the probability of each pixel value e in the pixel value set in the sub-image data is determined: , according to the probability of each pixel value in the pixel value set in the sub-image data Determine the information entropy of sub-image data .

[0109] In some embodiments, inputting the input data into a pre-trained judgment module, and using the pre-trained judgment module to extract an initial detection vector of the input data includes:

[0110] Inputting the image data into the image feature extraction module, and using the image feature extraction module to extract multiple image feature vectors of the image data;

[0111] Inputting the text data into a text feature extraction module, and using the text feature extraction module to extract multiple text feature vectors of the text data;

[0112] Inputting a plurality of image feature vectors and a plurality of text feature vectors into a similarity calculation module, and using the similarity calculation module to determine similarity feature vectors of the plurality of image feature vectors and the plurality of text feature vectors;

[0113] The similarity feature vector is input into the adaptive device, and the adaptive device is used to classify and calculate the similarity feature vector to obtain an initial detection vector.

[0114] Specifically, the image data and text data are extracted from the input data, wherein the input data may include at least one image-text fusion data, and each image-text fusion data includes image data and text data.

[0115] Specifically, the image data is input into the image feature extraction module to extract multiple image feature vectors of the image data to obtain a set of image feature vectors. , B represents the number of image feature vectors. Input the text data into the text feature extraction module to extract multiple text feature vectors of the text data to obtain a set of text feature vectors , c represents the number of text feature vectors.

[0116] Specifically, multiple image feature vectors of image data and multiple text feature vectors of text data are input into the similarity calculation module to determine the similarity feature vectors between the image data and the text data, and the similarity feature vectors are input into the adaptor, and the similarity feature vectors are classified and calculated to obtain the initial detection vector of the input data. .in, represents the first similarity between multiple text feature vectors and corresponding image context data, Indicates the second similarity between the plurality of image feature vectors and the corresponding text context data.

[0117] Optionally, the image-text similarity may be calculated by cosine similarity.

[0118] Correspondingly, the similarity can be expressed by the following formula:

[0119] ;

[0120] in, Represents the set F of text feature vectors and the set of image feature vectors The dot product of and Represent the set of text feature vectors F and the set of image feature vectors respectively Model.

[0121] Optionally, the first similarity can be expressed by the following formula:

[0122] ;

[0123] Optionally, the second similarity can be expressed by the following formula:

[0124] ;

[0125] in, Represents the image context data corresponding to multiple text feature vectors, ; Represents the text context data corresponding to multiple image feature vectors, .

[0126] In some embodiments, a plurality of image feature vectors and a plurality of text feature vectors are input into a similarity calculation module, and the similarity calculation module is used to determine similarity feature vectors of the plurality of image feature vectors and the plurality of text feature vectors, including:

[0127] Inputting the plurality of image feature vectors and the plurality of text feature vectors into a similarity calculation module, and using the similarity calculation module to determine an attention score matrix based on the plurality of text feature vectors and the plurality of image feature vectors;

[0128] Determine, according to the attention score matrix, a first attention score between each text feature vector and any image feature vector, and a second attention score between each image feature vector and any text feature vector;

[0129] Determining image context data corresponding to the plurality of text feature vectors according to each first attention score;

[0130] Determining text context data corresponding to the plurality of image feature vectors according to each second attention score;

[0131] A similarity feature vector is determined based on the image context data and the text context data.

[0132] Specifically, the set of multiple image feature vectors can also be called an image feature sequence , the set of multiple text feature vectors can also be called a text feature sequence F, and the attention score matrix S is calculated based on the text feature sequence and the image feature sequence.

[0133] Optionally, the attention score matrix S can be determined by the dot product attention calculation method:

[0134] in, .

[0135] Among them, T represents the transposition process, Represents a text feature vector With the image feature vector The attention score between .

[0136] Specifically, according to the attention matrix score matrix S, the first attention score ai of each text feature vector and any image feature vector, and the second attention score of each image feature vector and any text feature vector are determined. The first attention score ai is normalized to obtain the attention weight matrix A. Among them, ;

[0137] Figure 5 A schematic diagram of an application for determining an attention score provided in an embodiment of the present application.

[0138] See also Figure 5 , the image feature vector is represented by a hollow square, and the text feature vector is represented by a hollow circle. Represents a text feature vector With the image feature vector The attention score between Represents a text feature vector With the image feature vector The attention score between , and so on. Text feature vector The first attention score ai of each image feature vector is , , , sum.

[0139] Among them, the method of obtaining the attention weight matrix based on normalization processing can be expressed by the following formula:

[0140] ;

[0141] Among them, p represents the text feature vector, Represents the attention score between the text feature vector p and any image feature vector.

[0142] Optionally, the first attention score ai can be normalized by a normalized exponential softmax function.

[0143] Specifically, according to the attention weights in the attention weight matrix A By weighted summing of image feature vectors, we can obtain the image context data corresponding to multiple text feature vectors. And according to the second attention score of each image feature vector and any text feature vector, the text context data corresponding to multiple image feature vectors can be calculated. .

[0144] Optionally, a cosine similarity calculation method is used to calculate a first similarity between multiple text feature vectors and corresponding image context data, and a second similarity between multiple image feature vectors and corresponding text context data, and the first similarity and the second similarity are concatenated to obtain a similarity feature vector of the image data and the text data.

[0145] Optionally, the pre-trained judgment module also includes a multi-layer perceptron (MLP), and the output result of the multi-layer perceptron MLP is the texture reasoning result. .

[0146] It is understandable that when the texture reasoning results of the multi-layer perceptron (MLP) are directly input into the pre-trained large language model, the pre-trained large language model cannot obtain the texture information in the texture reasoning results and output the detection results. A texture adapter (also called an adapter) can be constructed to connect the texture reasoning results of the multi-layer perceptron and the pre-trained large language model.

[0147] Figure 6 Another structural schematic diagram of a pre-trained judgment module provided in an embodiment of the present application.

[0148] See also Figure 6 The pre-trained judgment module 301 includes an image feature extraction module 3011, a text feature extraction module 3012, a similarity calculation module 3013, a multi-layer perceptron 3015 and an adaptor 3014. The image feature extraction module 3011 and the text feature extraction module 3012 are both connected to the similarity calculation module 3013, and the similarity calculation module 3013, the multi-layer perceptron 3015 and the adaptor 3014 are connected in sequence.

[0149] The image feature extraction module 3011 is used to input the image feature vector to the similarity calculation module 3013, and the text feature extraction module 3012 is used to input the text feature vector to the similarity calculation module 3013. The similarity calculation module 3013 determines the similarity feature vector based on the image feature vector and the text feature vector ( Figure 6 The multilayer perceptron 3015 determines the texture inference result based on the similarity feature vector ( Figure 6 ), and is input to the adaptor 3014.

[0150] In some implementations, the similarity feature vector is input into an adaptor, and the adaptor is used to perform classification calculation on the similarity feature vector to obtain an initial detection vector, including:

[0151] The similarity feature vector is input into the adaptive device, and the adaptive device is used to perform a linear transformation process on the similarity feature vector to obtain a first output vector;

[0152] Performing feature enhancement processing on the first output vector to obtain a second output vector;

[0153] Perform dimension adaptation on the second output vector to obtain the initial detection vector

[0154] Optionally, the adaptor includes an input adaptation layer, a feature enhancement layer and an output adaptation layer, which are connected in sequence. The input adaptation layer is used to perform dimension adaptation and linear transformation processing on the input similarity feature vector to obtain a first output vector , to map the similarity feature vector to the internal feature space of the adaptor.

[0155] in, express The first weight matrix of size, ; represents the first bias vector, . , Indicates the range of the first output vector.

[0156] Specifically, the feature enhancement layer is composed of multiple nonlinear transformation units, which are used to improve the ability to explain the similarity feature vector and the accuracy of identifying the traces of false and forged data, that is, to perform feature enhancement processing on the first output vector output by the input adaptation layer to obtain the second output vector. The feature enhancement layer can include a multi-layer perceptron (MLP) structure or multiple convolutional layers to achieve the above functions.

[0157] Taking a multilayer perceptron including N hidden layers as an example, the number of neurons in each hidden layer is Among them, the output data of the first hidden layer is: , represents the activation function (for example, the activation function is ReLU), express The second weight matrix of size, represents the second bias vector, Represents the range of the second bias vector. The subsequent hidden layers can determine the output data of each hidden layer by analogy. The output data of the Nth hidden layer is .

[0158] Specifically, the output adaptation layer is used to perform dimension adaptation processing on the second feature vector output by the feature enhancement layer so that the processed third feature vector can match the detection target (including the recognition result and the corresponding description data).

[0159] Optionally, express The third weight matrix of size, represents the third bias vector, Represents the range of the third bias vector, and the output data of the output adaptation layer is: ,in, , express range, It can also represent the initial detection vector output by the adapter. The initial detection vector, instruction text data, and output options are input into the pre-trained large prediction model, and the pre-trained large prediction model is used for fusion and analysis to determine the target detection result of the input data and the corresponding target description data.

[0160] It should be noted that in the initial authenticity detection model, the parameters of the initial adaptive adapter are close to 0. During the training process, the parameters of the initial adaptive adapter have little effect on the initial authenticity detection model, making the initial authenticity detection model relatively stable during the pre-training process. In this way, the adaptive adapter provides the initial authenticity detection model with the ability to learn and correct to output detection targets (such as detection results and corresponding description data) without significantly changing the structure and parameters of the initial authenticity detection model, making the initial authenticity detection model applicable to a variety of tasks, with the characteristics of flexibility and efficiency.

[0161] The embodiment of the present application adapts and transforms the image feature vector extracted by the image feature network (for example, the image feature network is composed of a basic convolutional neural network) through an adapter, learns more complex feature information in the input data, highlights features related to traces of false and forged data, and suppresses features that are not related to authenticity identification, thereby matching the feature space and task requirements of the pre-trained authenticity detection model, and improving the detection accuracy of the pre-trained authenticity detection model in the authenticity detection task of multimodal data.

[0162] Figure 7 The third flow chart of data authenticity detection provided in the embodiment of the present application is as follows: Figure 7 As shown, step S2022 of the method includes:

[0163] S20221: Convert the instruction text data into a first vector, and convert the output options into a second vector; the instruction text data is used to indicate a detection target corresponding to the input data; the detection target includes a detection result and description data corresponding to the detection result.

[0164] Specifically, the instruction text data (IN) includes a text prompt for informing the pre-trained large language model of the task target (also referred to as a detection target or expected behavior) corresponding to the input data. For example: the instruction text data includes a text prompt for informing the pre-trained large language model which feature information in the input data can be identified to obtain the target detection result of the input data. The detection target includes but is not limited to the detection result and the descriptive data corresponding to the detection result. The output option refers to at least one option of the detection result corresponding to the authenticity of the input data.

[0165] Exemplarily, the instruction text data may include "performing an in-depth analysis on the semantic association between the image data and the text data in the input data to identify the semantic consistency of the image data and the text data".

[0166] As another example, the instruction text data may include "identifying edge discontinuity features of image data, matching the above-mentioned edge discontinuity traces with the semantic logic in the text data to identify the authenticity of the image data, and the logical consistency between the image data and the text data."

[0167] As another example, the instruction text data may include "identifying feature information of input data, and matching it with the semantics and logic of real events to identify the consistency between the input data and the real events."

[0168] For example, the instruction text data includes "Please identify the authenticity of the input data based on real events, select one from the output options as the target detection result based on the identification situation, and explain the reason for determining that the input data is the target detection result", and the output options include "true, false". The data output by the pre-trained large language model includes "The input data is false, and the reason for determining that the input data is false is: the text data describes that the input data is a picture of a dance program for the 2024 Spring Festival Gala. After comparing it with the real pictures of various dance programs in the 2024 Spring Festival Gala, it is determined that the content in the image data is not included, so the input data does not match the real event."

[0169] Optionally, the pre-trained large language model may be an open source neural network model. For example, the pre-trained large language model is llama3.

[0170] Optionally, the pre-trained large language model can identify the target detection result of the input data and the target description data corresponding to the target detection result based on the input instruction text data, output options and input data.

[0171] Optionally, the output option (OP) can be set to a triplet, which is used to provide a clear detection category option for the pre-trained large language model. Each element in the triplet or bigram has a specific semantics and effect on the output detection result. The pre-trained large language model can select an option from the triplet or bigram as the target detection result based on the recognized information. Optionally, the output option OP can also be a bigram.

[0172] For example, the output options OP include <true, false, uncertain>; or, the output options OP include <true, false, uncertain>. Or, the output options OP include <true, false>.

[0173] It is understandable that, during the pre-training process of the initial large language model, the output options can also be used as the detection results corresponding to the training sample data to pre-train the initial large language model. At this time, the output options are used as supervision signals in the pre-training process to guide the output results of the initial large language model.

[0174] Exemplarily, each training sample data (i.e., fusion data of image data and text data) includes a true option corresponding to the training sample data, which is used to indicate the authenticity of the training sample data. The output option is an option determined based on the training sample data and selected from the triple. The output accuracy of the initial large language model can be determined by comparing the true option included in the training sample data with the detection result corresponding to the training sample data output by the initial large language model during the pre-training process.

[0175] For example, the training sample data 001 is determined to be false data after user analysis. During the pre-training process, the detection result corresponding to the training sample data output by the initial large language model is "true", then it is determined that the output accuracy of the initial large language model is low, and the parameters of the initial large language model can be adjusted through the loss function to improve the output accuracy of the initial large language model.

[0176] The initial detection vector output by the pre-trained judgment module can provide feature information for determining authenticity, the instruction text data can inform the model of the task objectives, and the output options can provide the model with judgment category options. The pre-trained large language model performs deep learning and reasoning on this basis, which can accurately identify the forged feature information in the multimodal information and improve the accuracy of the detection results. In addition, the pre-trained large language model has strong semantic understanding and text generation capabilities. Based on the language patterns and logical relationships learned from a large amount of text data, it can understand the feature information output by the pre-trained judgment module. For example, when there is semantically inconsistent feature information between image data and text data, the pre-trained large language model can accurately determine whether there are subtle descriptive differences or forgery intentions in the above feature information, thereby improving the accuracy of the authenticity detection results of the entire multimodal fusion data.

[0177] The embodiment of the present application pre-trains the initial large language model based on output options and performs result-oriented processing on the pre-trained large language model, so that the authenticity information of the input data detected by the pre-trained large language model can be presented in a standardized form.

[0178] For example, when the pre-trained large language model recognizes the feature information of the input data that "the content of the image data is a person, the content of the text data is an animal, and the image and text are inconsistent", the pre-trained large language model can select the "error" option from the triplet of output options as the target recognition result, and convert the above feature information into corresponding target description data.

[0179] It is understandable that when the pre-trained large language model recognizes the feature information of the input data, it can determine that the input data is in line with expectations and true and correct under the conditions of "semantic consistency between image data and text data, edge continuity of image data, logical consistency between image data and text data, and the graphic and text features of the input data are consistent with the semantics and logic of real events", and select the "correct" or "true" option from the output options.

[0180] It is understandable that when any of the following conditions is met: "semantic inconsistency between image data and text data, discontinuous edges of image data, logical inconsistency between image data and text data, and semantic inconsistency or logical inconsistency between the graphic features of input data and real events", the pre-trained large language model selects the "error" or "false" option from the output options.

[0181] For example, the input data is a picture of a historical relic with a graphic description. The age corresponding to the characteristic traces of the historical relic (such as the degree of wear and tear of the historical relic, the craftsmanship characteristics of the historical relic, the structural materials of the historical relic, etc.) does not match the age in the text data. Or, the characteristic information described in the text data does not exist in the image data. The pre-trained large language model selects the "error" or "false" option from the output options.

[0182] Also, for the pre-trained large language model, select the "uncertain" option from the output options in cases where "the feature information extracted from the input data is incomplete, and the image data is unclear", and where it is impossible to determine "the semantic consistency of the image data and text data, the edge continuity of the image data, the logical consistency between the image data and text data, and the semantic and logical consistency between the graphic features of the input data and the real events".

[0183] For example, when the image data has low clarity and resolution, the pre-trained large language model cannot extract the features of the image data; or when there is less information in the text data and the pre-trained large language model cannot identify whether it is consistent with the image data, the pre-trained large language model cannot give a clear "correct" or "wrong" judgment. The pre-trained large language model selects the "uncertain" option from the output options and explains in the output results what information needs to be further supplemented to make accurate detection results.

[0184] Specifically, the instruction text data can be converted into a first vector based on a pre-trained text encoder to obtain semantic information and grammatical information carried in the instruction text data. The output options can be converted into a second vector based on a pre-trained text encoder to make the vocabulary of the instruction text data and the output options consistent, so that the pre-trained large language model can determine the relationship between the above instruction text data and the output options based on the data in vector form.

[0185] S20222: Determine a target input vector based on the first vector, the second vector, and the initial detection vector;

[0186] S20223: Input the target input vector into the pre-trained large language model to obtain the target detection result and description data output by the pre-trained large language model.

[0187] Specifically, the first vector C and the second vector and the initial detection vector Combine to get the target input vector .

[0188] Optionally, the pre-trained large language model includes multiple encoder (Transformer) layers, multi-head self-attention mechanism (Multi-Head Self-Attention) layers and feed-forward neural network (Feed-Forward Neural Network) layers, and the above encoder (Transformer) layers, multi-head self-attention mechanism (Multi-Head Self-Attention) layers, and feed-forward neural network layers are connected in sequence.

[0189] Optionally, the target input vector is input into a pre-trained large language model LLM, which contains many encoder (Transformer) layers. The target input vector can be multiplied by the embedding matrix of the pre-trained large language model and the position encoding can be added to incorporate the position information. In each layer of the encoder, based on the multi-head self-attention mechanism, the pre-trained large language model identifies different parts of the target input vector from different subspaces in parallel, determines the attention distribution, and updates the vector representation by weighted summation to capture the semantic association between the elements in the target input vector. The target input vector is nonlinearly transformed through the feedforward neural network layer to further extract the feature information of the input data. Based on this, the pre-trained large language model can use its learned knowledge, semantics, and logic to deeply reason and integrate the feature information of the instruction text data, output options, and input data, and convert the vector representation inside the pre-trained large language model into a target detection result through the specific mapping and decoding processing of the output layer. , thereby providing users with detection results of the authenticity of multimodal fusion data and understandable and in-depth reasoning descriptions.

[0190] Figure 8 The fourth flow chart of data authenticity detection provided in the embodiment of the present application is as follows: Figure 8 As shown, the method includes:

[0191] S301: Collect image comparison sample data; wherein the image comparison sample data includes at least one first image sample data pair and at least one second image sample data pair; the first image sample data pair includes two first target image sample data whose information entropy is less than or equal to a first threshold; the second image sample data pair includes two second target image sample data whose information entropy is greater than or equal to a second threshold; the second threshold is greater than the first threshold.

[0192] Specifically, the initial authenticity detection model includes an initial judgment module and an initial large language model. The pre-training process of the initial authenticity detection model can be divided into a first-order pre-training process for the initial judgment module and a second-order pre-training process for the initial judgment module.

[0193] It can be understood that the initial authenticity detection model can be pre-trained as a whole model. In the embodiments of the present application, the training process of the initial judgment module in the initial authenticity detection model and the training process of the initial judgment module are described separately, which does not constitute a limitation on the pre-training process of the initial authenticity detection model.

[0194] Specifically, image comparison sample data is collected, and the image comparison sample data is image data including a plurality of sub-image sample data. Two sub-image sample data whose information entropy is less than or equal to a first threshold in the image comparison sample data are both determined as first target image sample data to obtain first image sample data; two sub-image data whose information entropy is greater than or equal to a second threshold are both determined as second target image sample data to obtain a second image sample data pair.

[0195] Specifically, the first threshold and the second threshold can be determined according to the information entropy of each sub-image sample data, and the second threshold is greater than the first threshold. For example, the first threshold is 0.1 and the second threshold is 0.8. Alternatively, the first threshold is 0.2 and the second threshold is 0.82. The embodiments of the present application are not specifically limited here.

[0196] S302: Pre-training the initial judgment module based on the image comparison sample data to obtain a pre-trained judgment module.

[0197] Specifically, the initial judgment module is pre-trained based on the above image comparison sample data to obtain a pre-trained judgment module.

[0198] In some implementations, pre-training the initial judgment module based on the image comparison sample data to obtain a pre-trained judgment module includes:

[0199] Based on the image comparison sample data, the initial judgment module is pre-trained using a loss function to obtain a pre-trained judgment module; the loss function is determined based on a first similarity metric and a second similarity metric, the first similarity metric is used to indicate the similarity metric between the two first target image sample data in each first image sample data pair, and the second similarity metric is used to indicate the similarity metric between the two second target image sample data in each second image sample data pair.

[0200] Optionally, based on the image comparison sample data, the initial judgment module is pre-trained using a loss function so that the distance between two sub-image sample data with greater similarity in the feature space of the initial judgment module is reduced, and the distance between two sub-image sample data with less similarity in the feature space of the initial judgment module is increased.

[0201] Optionally, the loss function is a noise contrast estimation (InfoNCE) loss function, which can be expressed by the following expression:

[0202] ;

[0203] in, Represents two first target image sample data in the first image sample data pair. Represents two second target image sample data in the second image sample data pair. Represents the first target image sample data The characteristic vector of Represents the temperature parameter, which is used to adjust the sensitivity of the similarity measure.

[0204] in, Represents the similarity measure between two feature vectors.

[0205] For example, the first similarity measure between two first target image sample data in the first image sample data pair can be expressed by the following formula:

[0206] .

[0207] Optionally, the image comparison sample data is input into an initial judgment module (for example, the initial judgment module can be a convolutional neural network), and the loss function value is calculated. The parameters of the initial judgment module are updated through a back propagation algorithm to gradually reduce the loss function value. The training is repeated until the loss function value is less than a preset loss function value, or until the number of repeated training reaches a preset number of training times.

[0208] Optionally, during the pre-training process, an optimization algorithm can be used to balance the ratio of the first image sample data pair and the second image sample data pair, and randomly select the same number of real sample data and false sample data to ensure that the performance of the authenticity detection model obtained by pre-training is optimal.

[0209] Optionally, the optimization algorithm is an adaptive learning rate optimization (Adaptive Moment Estimation, Adam) algorithm, and the learning rate of the Adam algorithm is set to 0.001, the exponential decay rate of the first-order moment estimation is set to 0.9, and the exponential decay rate of the second-order moment estimation is set to 0.999.

[0210] The embodiment of the present application pre-trains the initial judgment module based on the loss function and image comparison sample data, so that the initial judgment module network can learn the essential characteristics of the image data (for example, learn the differences and similarities between the texture features of different image data), which can improve the accuracy of extracting the texture features of the image data, thereby improving the accuracy of the authenticity detection results of the multimodal data.

[0211] In practical applications, text data can also be used as the type of training sample data to train the initial judgment module to improve the accuracy of the initial judgment module in extracting the semantic features of text data. The embodiment of the present application only uses image data as the type of training sample data to pre-train the initial judgment module to illustrate the process of improving the accuracy of the initial judgment module in extracting texture features of image data, and does not constitute a limitation on the pre-training process of the initial judgment module.

[0212] In some implementations, after pre-training the initial judgment module based on the image comparison sample data to obtain the pre-trained judgment module, the method includes:

[0213] Collecting a sample data set; wherein the sample data set includes a plurality of training data, each training data includes instruction data, output options, and output data of the initial judgment module during the pre-training of the initial judgment module; the instruction data is used to indicate the format of the output result of the initial large language model;

[0214] Based on the sample data set, the initial large language model is pre-trained to obtain a pre-trained large language model.

[0215] Specifically, a sample data set is collected, and the sample data set includes multiple training data, each of which includes the output data (specifically including image data and feature information), instruction data, and output options of the initial judgment module during the pre-training of the initial judgment module. The instruction data is used to indicate the format of the output result of the initial large language model. The specific format indicated by the above instruction data may include a description of the truth or falsehood reasoning process of the input data, a detailed explanation of the detection result (such as correct, wrong or uncertain), etc.

[0216] Specifically, based on the training data in the sample data set, the initial large language model is pre-trained, so that the pre-trained large language model can output target detection results and corresponding target description data in a specific format based on the output data, instruction data and output options of the initial judgment module.

[0217] Optionally, based on the sample data set, the initial large language model is pre-trained using a second loss function to obtain a pre-trained large language model.

[0218] Optionally, the second loss function includes one or more of a cross entropy loss function generated based on text, a structural similarity loss, a semantic consistency loss function, and a combined loss function.

[0219] Among them, the cross entropy loss function based on text generation refers to the function used to measure the difference between the text generated by the model and the target text.

[0220] The text sequence output by the initial large language model is: (in, represents the rth word or token in the text sequence), the target text sequence output by the pre-trained large language model is: For example, the cross entropy loss function based on text generation can be expressed as follows:

[0221] ;

[0222] Where V represents the vocabulary size, Indicates whether the word at position r in the target text is the oth word in the vocabulary; if the word at position r in the target text is the oth word in the vocabulary, then is 1, otherwise is 0. It represents the probability that the word at position r in the text generated by the initial large language model is the o-th word in the vocabulary, z represents the number of words or word units in the text sequence, and s represents the number of words or word units in the target text sequence.

[0223] In the process of pre-training (or fine-tuning) the initial large language model to output a target text in a specific format, the initial large language model is guided to learn the correct target text generation mode by calculating the cross entropy loss function value between the text generated by the initial large language model and the target text in a specific format expected in the dataset.

[0224] Fine-tuning refers to further training on a task-specific dataset based on a pre-trained model to adjust the model parameters to better suit the target task. During fine-tuning, most of the layers of the pre-trained model are usually frozen, and only the newly added layers are trained or a small number of key layers are adjusted. This can not only retain the useful features learned by the pre-trained model, but also quickly adapt to the specific needs of the new task. In addition, choosing the right learning rate and training rounds is also the key to successful fine-tuning.

[0225] Taking the target text as "Since the shadow traces in the image data are inconsistent with the time information described in the text data, the input data is judged to be wrong data" as an example, the initial large language model can learn the semantics, grammar and logical order of the target text, so that the generated text gradually becomes similar to such target text in terms of semantics, grammar and logical order. When the probability distribution difference between the generated text output by the initial large language model and the target text is large, the constant in the cross entropy loss function can be increased to facilitate the initial large language model to learn the correct word options, sentence structure, phrase combination and other information in the target text.

[0226] Among them, the structural similarity loss function is used to measure the structural similarity between the generated text output by the initial large language model and the target text, and can determine the differences between the generated text and the target text in terms of the number of sentences, paragraph division, location of important information, etc.

[0227] The structural feature vector of the generated text is , the structural feature vector of the target text is For example, the structural similarity loss function can be expressed as follows:

[0228] ;

[0229] in, express Norm, used to measure the Euclidean distance between two structural feature vectors.

[0230] For example, the specific format in the target text includes: 1. Feature information such as image features and text features; 2. Detection results; 3. Description data of the detection results. The generated text output by the initial large language model includes: 1. Detection results; 2. Feature information such as image features and text features; 3. Description data of the detection results. At this time, the constant in the structural similarity loss can be increased to guide the initial large language model to learn the structure in the target text and improve the readability and logic of the generated text of the initial large language model.

[0231] Optionally, the semantic consistency loss function can be used to measure the semantic similarity between the generated text output by the initial large language model and the first data input into the initial large language model (including initial judgment module output data, instruction text data, output selection, etc.).

[0232] Optionally, through a pre-trained semantic vector model, such as a bidirectional encoder representation model (Bidirectional Encoder Representations from Transformers, BERT), the generated text is converted into a corresponding first semantic vector, the first data is converted into a corresponding second semantic vector, and the cosine similarity between the first semantic vector and the second semantic vector is calculated.

[0233] The first semantic vector is , the second semantic vector is As an example, the semantic consistency loss function is expressed as follows:

[0234] ;

[0235] Specifically, if the generated text output by the initial large language model is semantically inconsistent with the input first data (for example, the edge trace features of the image data in the input first data are questionable, while the output generated text includes the input data as correct data), the constant in the semantic consistency loss function can be increased so that the initial large language model learns to output generated text that is semantically consistent with the input first data, thereby improving the accuracy of the target detection results.

[0236] Optionally, the combined loss function refers to combining the above-mentioned several loss functions to comprehensively improve the matching degree between the generated text output by the initial large language model and the input first data, thereby improving the accuracy of the target detection result.

[0237] Fig. 9 A schematic diagram of a fusion data authenticity detection process provided in an embodiment of the present application.

[0238] See also Fig. 9 , extract the image data and text data from the input data, pre-process the image data, and obtain the corresponding input image data set. Input the input image data set into the image data extraction module in the pre-trained authenticity detection model, and use the image extraction module to extract the image feature vector of the image data. Input the text data into the text feature extraction module, and use the text feature extraction module to extract the text feature vector of the text data. The image extraction module inputs the image feature vector into the similarity calculation module, and the text feature extraction module inputs the text feature vector into the similarity calculation module. The similarity calculation module determines the similarity feature vector and inputs it into the adaptor, and uses the adaptor to extract the initial detection vector of the input data. Splice the initial detection vector with the instruction text data and the output data, and input the spliced ​​data into the pre-trained large language model to obtain the target detection result output by the pre-trained large language model for indicating the authenticity of the input data, as well as the target detection data corresponding to the target detection result.

[0239] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0240] Fig.10 This is a schematic diagram of the structure of the data authenticity detection device provided in the embodiment of the present application. Fig.10 As shown, an embodiment of the present application further provides a data authenticity detection device, comprising:

[0241] The acquisition module 401 is used to acquire input data and output options; wherein the input data includes image data and text data, and the output options include at least one detection result;

[0242] The detection module 402 is used to input the input data and output options into the pre-trained authenticity detection model to obtain the target detection result of the input data output by the pre-trained authenticity detection model, and the target description data corresponding to the target detection result; wherein the target detection result is used to indicate the authenticity of the input data.

[0243] In some embodiments, the pre-trained authenticity detection model includes a pre-trained judgment module and a pre-trained large language model, and the pre-trained judgment module is connected to the pre-trained large language model; the detection module 402 includes:

[0244] An extraction submodule, used for inputting input data into a pre-trained judgment module, and extracting an initial detection vector of the input data using the pre-trained judgment module;

[0245] The detection submodule is used to input the initial detection vector and output options into the pre-trained large language model to obtain the target detection results and target description data output by the pre-trained large language model.

[0246] In some embodiments, the pre-trained judgment module includes an image feature extraction module, a text feature extraction module, a similarity calculation module and an adaptor, the image feature extraction module and the text feature extraction module are both connected to the similarity calculation module; the similarity calculation module is connected to the adaptor; the extraction submodule includes:

[0247] An extraction unit, used for inputting the image data into the image feature extraction module, and using the image feature extraction module to extract multiple image feature vectors of the image data;

[0248] Inputting the text data into a text feature extraction module, and using the text feature extraction module to extract multiple text feature vectors of the text data;

[0249] A first determining unit, configured to input a plurality of image feature vectors and a plurality of text feature vectors into a similarity calculation module, and determine similarity feature vectors of the plurality of image feature vectors and the plurality of text feature vectors using the similarity calculation module;

[0250] The classification unit is used to input the similarity feature vector into the adaptor, and use the adaptor to perform classification calculation on the similarity feature vector to obtain an initial detection vector.

[0251] In some embodiments, the vector determination unit includes:

[0252] A first determination subunit is used to input a plurality of image feature vectors and a plurality of text feature vectors into a similarity calculation module, and determine an attention score matrix based on the plurality of text feature vectors and the plurality of image feature vectors using the similarity calculation module;

[0253] A second determination subunit is used to determine a first attention score between each text feature vector and any image feature vector, and a second attention score between each image feature vector and any text feature vector according to the attention score matrix;

[0254] A third determination subunit, configured to determine image context data corresponding to the plurality of text feature vectors according to each of the first attention scores;

[0255] a fourth determination subunit, configured to determine text context data corresponding to the plurality of image feature vectors according to each second attention score;

[0256] The fifth determining subunit is used to determine a similarity feature vector according to the image context data and the text context data.

[0257] In some embodiments, a classification unit comprises:

[0258] A first processing subunit is used to input the similarity feature vector into the adaptor, and use the adaptor to perform linear transformation processing on the similarity feature vector to obtain a first output vector;

[0259] A second processing subunit is used to perform feature enhancement processing on the first output vector to obtain a second output vector;

[0260] The third processing subunit is used to perform dimension adaptation processing on the second output vector to obtain an initial detection vector.

[0261] In some embodiments, the detection submodule comprises:

[0262] A first processing unit is used to convert instruction text data into a first vector and convert output options into a second vector; the instruction text data is used to indicate a detection target corresponding to the input data; the detection target includes a detection result and description data corresponding to the detection result;

[0263] A second determining unit, configured to determine a target input vector based on the first vector, the second vector and the initial detection vector;

[0264] The detection unit is used to input the target input vector into the pre-trained large language model to obtain the target detection result and description data output by the pre-trained large language model.

[0265] In some embodiments, the extraction submodule includes:

[0266] A segmentation unit, used for segmenting the image data into a plurality of sub-image data;

[0267] The set determination unit is used to determine the information entropy of each sub-image data, and determine the input image data set according to the information entropy of each sub-image data.

[0268] In some embodiments, the set determination unit includes:

[0269] The information entropy determination subunit is further used to determine the information entropy of each sub-image data and collect the position information of each sub-image data;

[0270] The stitching subunit is used to stitch the information entropy and corresponding position information of each sub-image data to obtain an input image data set; wherein the input image data set includes multiple input image data arranged in ascending or descending order based on information entropy; each input image data includes at least one sub-image data, information entropy and position information of the sub-image data.

[0271] In some embodiments, the information entropy determination subunit is specifically used to:

[0272] Acquire a pixel value set of each sub-image data; wherein the pixel value set includes each pixel value in the sub-image data;

[0273] Based on the pixel value set of each sub-image data, the probability of each pixel value in the sub-image data is determined, and according to the probability of each pixel value in the sub-image data, the information entropy of the sub-image data is determined.

[0274] In some embodiments, the device further comprises:

[0275] A first acquisition module is used to acquire image comparison sample data; wherein the image comparison sample data includes at least one first image sample data pair and at least one second image sample data pair; the first image sample data pair includes two first target image sample data whose information entropy is less than or equal to a first threshold; the second image sample data pair includes two second target image sample data whose information entropy is greater than or equal to a second threshold; the second threshold is greater than the first threshold;

[0276] The first pre-training module is used to pre-train the initial judgment module based on the image comparison sample data to obtain a pre-trained judgment module.

[0277] In some embodiments, the first pre-training module includes:

[0278] A pre-training unit is used to pre-train an initial judgment module based on image comparison sample data using a loss function to obtain a pre-trained judgment module; the loss function is determined based on a first similarity metric and a second similarity metric, the first similarity metric is used to indicate a similarity metric between two first target image sample data in each first image sample data pair, and the second similarity metric is used to indicate a similarity metric between two second target image sample data in each second image sample data pair.

[0279] In some embodiments, a device comprises:

[0280] A second acquisition module is used to acquire a sample data set; wherein the sample data set includes a plurality of training data, each of which includes instruction data, output options, and output data of the initial judgment module during the pre-training of the initial judgment module; the instruction data is used to indicate the format of the output result of the initial large language model;

[0281] The second pre-training module is used to pre-train the initial large language model based on the sample data set to obtain a pre-trained large language model.

[0282] For the description of the features in the embodiment corresponding to the data authenticity detection device, reference can be made to the relevant description of the embodiment corresponding to the data authenticity detection method, which will not be repeated here.

[0283] Fig.11 This is a schematic diagram of the structure of the electronic device provided in this application. Fig.11 As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus.

[0284] In the specific implementation process, at least one processor 501 executes the computer execution instructions stored in the memory 502, so that the at least one processor 501 executes the above-mentioned data authenticity detection method embodiment.

[0285] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0286] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0287] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0288] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0289] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data authenticity detection method embodiments when running.

[0290] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0291] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above data authenticity detection method embodiments are implemented.

[0292] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned data authenticity detection method embodiments are implemented.

[0293] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0294] The above is a detailed introduction to a data authenticity detection method, electronic device, storage medium and program product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data authenticity detection method, characterized in that: include: Acquire input data and output options; wherein the input data includes image data and text data, and the output options include at least one detection result; The input data and the output options are input into a pre-trained authenticity detection model to obtain a target detection result of the input data output by the pre-trained authenticity detection model, and target description data corresponding to the target detection result; wherein the target detection result is used to indicate the authenticity of the input data.

2. The data authenticity detection method according to claim 1, characterized in that: The pre-trained authenticity detection model includes a pre-trained judgment module and a pre-trained large language model, and the pre-trained judgment module is connected to the pre-trained large language model; the input data and the output options are input into the pre-trained authenticity detection model to obtain the target detection result of the input data output by the pre-trained authenticity detection model, and the target description data corresponding to the target detection result, including: Inputting the input data into the pre-trained judgment module, and extracting an initial detection vector of the input data using the pre-trained judgment module; The initial detection vector and the output options are input into the pre-trained large language model to obtain the target detection result and the target description data output by the pre-trained large language model.

3. The data authenticity detection method according to claim 2, characterized in that: The pre-trained judgment module includes an image feature extraction module, a text feature extraction module, a similarity calculation module and an adaptor, wherein the image feature extraction module and the text feature extraction module are both connected to the similarity calculation module; the similarity calculation module is connected to the adaptor; the input data is input into the pre-trained judgment module, and the pre-trained judgment module is used to extract an initial detection vector of the input data, including: Inputting the image data into the image feature extraction module, and using the image feature extraction module to extract multiple image feature vectors of the image data; Inputting the text data into the text feature extraction module, and using the text feature extraction module to extract multiple text feature vectors of the text data; Inputting the plurality of image feature vectors and the plurality of text feature vectors into the similarity calculation module, and using the similarity calculation module to determine similarity feature vectors of the plurality of image feature vectors and the plurality of text feature vectors; The similarity feature vector is input into the adaptive device, and the adaptive device is used to perform classification calculation on the similarity feature vector to obtain the initial detection vector.

4. The data authenticity detection method according to claim 3, characterized in that: The step of inputting the plurality of image feature vectors and the plurality of text feature vectors into the similarity calculation module, and using the similarity calculation module to determine similarity feature vectors of the plurality of image feature vectors and the plurality of text feature vectors comprises: Inputting the plurality of image feature vectors and the plurality of text feature vectors into the similarity calculation module, and using the similarity calculation module to determine an attention score matrix based on the plurality of text feature vectors and the plurality of image feature vectors; Determine, according to the attention score matrix, a first attention score between each of the text feature vectors and any one of the image feature vectors, and a second attention score between each of the image feature vectors and any one of the text feature vectors; Determining image context data corresponding to the plurality of text feature vectors according to each of the first attention scores; Determining text context data corresponding to the plurality of image feature vectors according to each of the second attention scores; The similarity feature vector is determined according to the image context data and the text context data.

5. The data authenticity detection method according to claim 3, characterized in that: The step of inputting the similarity feature vector into the adaptor and using the adaptor to perform classification calculation on the similarity feature vector to obtain the initial detection vector includes: Inputting the similarity feature vector into the adaptor, and using the adaptor to perform linear transformation processing on the similarity feature vector to obtain a first output vector; Performing feature enhancement processing on the first output vector to obtain a second output vector; Perform dimension adaptation processing on the second output vector to obtain the initial detection vector.

6. The data authenticity detection method according to claim 2, characterized in that: The step of inputting the initial detection vector and the output option into the pre-trained large language model to obtain the target detection result and the target description data output by the pre-trained large language model includes: Converting instruction text data into a first vector, and converting the output option into a second vector; the instruction text data is used to indicate a detection target corresponding to the input data; the detection target includes a detection result and description data corresponding to the detection result; Determine a target input vector based on the first vector, the second vector and the initial detection vector; The target input vector is input into the pre-trained large language model to obtain the target detection result and the description data output by the pre-trained large language model.

7. The data authenticity detection method according to claim 3, characterized in that: Before the image data is input into the image feature extraction module and the image feature extraction module is used to extract a plurality of image feature vectors of the image data, the method includes: dividing the image data into a plurality of sub-image data; The information entropy of each sub-image data is determined, and the input image data set is determined according to the information entropy of each sub-image data.

8. The data authenticity detection method according to claim 7, characterized in that: The determining the information entropy of each sub-image data, and determining the input image data set according to the information entropy of each sub-image data, comprises: Determining the information entropy of each sub-image data, and collecting the position information of each sub-image data; The information entropy of each of the sub-image data and the corresponding position information are spliced ​​to obtain the input image data set; wherein the input image data set includes a plurality of input image data arranged in ascending or descending order based on information entropy; each of the input image data includes at least one sub-image data, the information entropy and position information of the sub-image data.

9. The data authenticity detection method according to claim 8, characterized in that: The determining the information entropy of each sub-image data comprises: Acquire a pixel value set of each of the sub-image data; wherein the pixel value set includes each pixel value in the sub-image data; Based on the pixel value sets of each sub-image data, the probability of each pixel value in the sub-image data is determined, and according to the probability of each pixel value in the sub-image data, the information entropy of the sub-image data is determined.

10. The data authenticity detection method according to claim 2, characterized in that: The method further comprises: Collect image comparison sample data; wherein the image comparison sample data includes at least one first image sample data pair and at least one second image sample data pair; the first image sample data pair includes two first target image sample data whose information entropy is less than or equal to a first threshold; the second image sample data pair includes two second target image sample data whose information entropy is greater than or equal to a second threshold; the second threshold is greater than the first threshold; The initial judgment module is pre-trained based on the image comparison sample data to obtain the pre-trained judgment module.

11. The data authenticity detection method according to claim 10, characterized in that: The pre-training of the initial judgment module based on the image comparison sample data to obtain the pre-trained judgment module includes: Based on the image comparison sample data, an initial judgment module is pre-trained using a loss function to obtain the pre-trained judgment module; the loss function is determined based on a first similarity metric and a second similarity metric, the first similarity metric is used to indicate a similarity metric between two first target image sample data in each pair of the first image sample data, and the second similarity metric is used to indicate a similarity metric between two second target image sample data in each pair of the second image sample data.

12. The data authenticity detection method according to claim 10, characterized in that: After the initial judgment module is pre-trained based on the image comparison sample data to obtain the pre-trained judgment module, the method includes: Collecting a sample data set; wherein the sample data set includes a plurality of training data, each of the training data includes instruction data, the output options, and output data of the initial judgment module during the pre-training of the initial judgment module; the instruction data is used to indicate the format of the output result of the initial large language model; Based on the sample data set, the initial large language model is pre-trained to obtain the pre-trained large language model.

13. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data authenticity detection method as claimed in any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data authenticity detection method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data authenticity detection method according to any one of claims 1 to 12 are implemented.

Images